Your browser does not support JavaScript! Skip to main content
Free 30-day trial Customer portal Contact
 
Rapita Systems
 

Industry leading verification tools & services

Rapita Verification Suite (RVS)

  RapiTest - Unit/system testing   RapiCover - Structural coverage analysis   RapiTime - Timing analysis (inc. WCET)   RapiTask - Scheduling visualization   RapiCoverZero - Zero footprint coverage analysis   RapiTimeZero - Zero footprint timing analysis   RapiTaskZero - Zero footprint scheduling analysis

Multicore verification

  CAST-32A Compliance   Multicore Timing Solution   RapiDaemons

Services

  V & V Services   Qualification   Training   Tool Integration  Support

Industries

  Aerospace (DO-178C)   Automotive (ISO 26262)   Space

Other

  RTBx   Mx-Suite   Software licensing   Product life cycle policy

Latest from Rapita HQ

Latest news

Rapita hosts SCADE webinar with ANSYS
FACE Virtual Technical Interchange Meeting
RVS 3.14 Launched
Propelling the next generation of scientists
View News

Latest from the Rapita blog

Software verification on the Solar Orbiter
Metrowerks CodeTest - How and why to upgrade
Leveraging FACE Conformance Artifacts to Support Airworthiness
Assured Multicore Partitioning for FACE Systems
View Blog

Latest discovery pages

matlab_simulink MATLAB Simulink MCDC coverage and WCET analysis
code_coverage_ada Code coverage for Ada, C and C++
amc-20-193 AMC 20-193
embedded_software_testing Embedded Software Testing Tools
View Discovery pages

Upcoming events

Future Airborne Capability Environment (FACE) Exihibition
2021-09-14
Safe Use of Multi-Core Processors Seminar
2021-11-10
View Events

Technical resources for industry professionals

Latest White papers

Multicore Timing Analysis for DO-178C
Seven Roadblocks to 100% Structural Coverage (and how to avoid them)
Eight top code coverage questions in embedded avionics systems
View White papers

Latest Videos

Efficient testing with RVS and ANSYS® SCADE® Test™
Enabling cost-effective modular avionics with FACE
Streamlined software verification thumbnail
Streamlined software verification with RVS 3.14
Qualification guidance thumbnail
Clear qualification guidance with RVS qualification kits
View Videos

Latest Case studies

Cobham Aerospace Connectivity: RapiCover continues to deliver on the most challenging targets
DO-178B Level A Embraer FCS
Validation of COTS Ada Compiler for Safety-Critical Applications
View Case studies

Other Downloads

 Webinars

 Brochures

 Product briefs

 Technical notes

 Research projects

Discover Rapita

Who we are

The company menu

  • About us
  • Customers
  • Distributors
  • Locations
  • Partners
  • Research projects
  • Contact us

Careers

Careers menu

  • Current opportunities & application process
  • Working at Rapita

US office

+1 248-957-9801
info@rapitasystems.com
41131 Vincenti Ct.
Novi, MI, 48375
USA

UK office

+44 (0)1904 413945
info@rapitasystems.com
Atlas House
York, YO10 3JB
UK

Back to Top

What are the overheads of code coverage? Using "Doom" as an example

Breadcrumb

  1. Home
  2. Blog
  3. What are the overheads of code coverage? Using "Doom" as an example
2015-05-01

One question that we are frequently asked is what instrumentation overhead RapiCover introduces. Achieving low overhead instrumentation is recognized by our customers as a key strength of RapiCover. To measure structural code coverage of embedded software, or for that matter any software, code coverage tools like RapiCover use instrumentation (additional code that records what code has been executed).

Raspberry Pi Raspberry Pi

In practice, this is a question that we tend to answer as "it depends", which, although true, isn't a particularly helpful answer. Instrumentation overhead depends on three things:

  • The "shape" and size of the software to be measured. The complexity of the software will have an impact on how much instrumentation is required (for example, when measuring decision coverage, linear code requires less instrumentation than code featuring lots of if-statements).
  • The type of coverage required. In the world of high-integrity aerospace and automotive systems, this is driven by the software integrity level/development assurance level of the software. Closely related to this is the density of instrumentation needed to measure the required type of coverage. Some types of coverage (for example, function coverage) require very little instrumentation, whereas others (such as MC/DC) require more.
  • The overhead of each instrumentation point. This is affected by the approach that is taken for implementing the instrumentation code.

As you might expect, the first of these factors is entirely within the realm of the application developer. In addressing the second factor, we have implemented a number of optimizations within RapiCover which minimize the instrumentation density. How we address the third factor is a matter of some interest. RapiCover, like our timing analysis tool RapiTime, provides an "open interface" instrumentation library. We provide relatively lightweight, "out-of-the-box" generic instrumentation code. This gives a good starting position for our users – enough to prove the coverage process. Once the coverage process works, the instrumentation can be optimized.

To illustrate the benefits of a well-optimized instrumentation library, we are using the classic game "Doom", compiled for the Raspberry Pi – an ARM-based mini-computer typical of many embedded systems. We set the game up to be controlled by a demo file (a recording of a player's moves, acting as a test vector). We removed any parts of the code that rely on real-world timing, so that the game runs as quickly as possible. The result is a version of the game that requires no user input, instead playing through the demo recording as quickly as possible. The demo ("Doom Done Quick") is a play-through of the game, visiting every level. The following video shows the demo being played back at normal speed.

When played in real time, the play-through takes 1181 seconds. On the Raspberry Pi, running as quickly as possible, the play-through takes only 220 seconds. We added statement and decision coverage to the example code. With our generic instrumentation, the play-through takes 329 seconds – 49% slower than the uninstrumented version (220 seconds). Replacing the generic version with a highly-optimized, target-specific instrumentation library, we find that the play-through time is even less. With further optimization, we were able to reduce the overhead to 3.2% (227 seconds). If the overheads introduced by code coverage are a concern (and in embedded systems, this is often the case), the per-instrumentation point cost is likely to have the single largest impact. As the Doom example shows, the ability to optimize instrumentation points can be exploited to achieve impressively low overheads.

White papers

Related blog posts

False positive and false negative in software testing

.
2019-05-22

1000 conditions per decision ought to be enough for anybody

.
2019-01-28

Code coverage without instrumentation

.
2018-10-18

Merging coverage data from multiple test runs

.
2017-01-31

Pagination

  • Current page 1
  • Page 2
  • Page 3
  • Page 4
  • Page 5
  • Page 6
  • Next page Next ›
  • Last page Last »
  • Solutions
    • Rapita Verification Suite
    • RapiTest
    • RapiCover
    • RapiTime
    • RapiTask

    • CAST-32A Compliance Package
    • Verification and Validation Services
    • Qualification
    • Training
    • Integration
  • Latest
  • Latest menu

    • News
    • Blog
    • Events
    • Videos
  • Downloads
  • Downloads menu

    • Brochures
    • Webinars
    • White Papers
    • Case Studies
    • Product briefs
    • Technical notes
    • Software licensing
  • Company
  • Company menu

    • About Rapita
    • Customers
    • Distributors
    • Industries
    • Locations
    • Partners
    • Research projects
    • Contact
  • Discover
    • AMC 20-193
    • What is CAST-32A?
    • Multicore Timing Analysis
    • MC/DC Coverage
    • Code coverage for Ada, C & C++
    • Embedded Software Testing Tools
    • Aerospace Software Testing
    • Automotive Software Testing
    • Certifying eVTOL
    • DO-178C Testing
    • WCET Tools
    • Worst Case Execution Time
    • Timing analysis (WCET) & Code coverage for Matlab Simulink

All materials © Rapita Systems Ltd. 2021 - All rights reserved | Privacy information Subscribe to our newsletter