Previewing the Avionics Europe presentation on timing test effectiveness by Rapita Systems

DO-178B/DO-330 Tool Qualification: Test Effectiveness for WCET Analysis Tools (a presentation for Avionics Europe, Munich, Wednesday 20th February 2013)

Confidence in software tools rests on the effectiveness of tool verification – essentially, asking the right questions. To determine the right questions for WCET tools, the full presentation will include our WCET tool test effectiveness framework and explain how it influences our tool testing. By way of a sneak preview, this blog post introduces some of the key issues which will be explored in greater detail during Avionics Europe.

For a product test to be effective, it must have the ability to discriminate correct and incorrect behaviour, and it must be representative. If it cannot discriminate, then it may claim that the product operates correctly when in fact there is some incorrect operation. If it is not representative, then the claims that it makes about the test environment do not translate into claims about the product in use.

For many software products, a representative test environment is reasonably easy to set up (and there are many, many testing frameworks and testing tools that make this job easier). The issue is in discriminating behaviours - the state space of the software cannot be explored exhaustively, so some thought needs to go into choosing the behaviours to examine.

For software tools that work in diverse environments, the representative test environment is harder to generate. It is likely that multiple test environments are needed, and each one may support a number of configurations. This is a challenge for RVS, our suite of on-target verification tools for embedded systems, as RVS is highly integrated with software build systems and target facilities, and the biggest challenge is our on-target execution time analysis tool, RapiTime.

To understand more about our own needs when providing qualification evidence about RapiTime, we performed a systematic (HAZOP-style) analysis of the information flows between the tool user, the tools and the software running on the target to plan tests that were likely to be effective. Examples of the results obtained from the analysis process include:

• discrimination issues that we had already made explicit:
  • The code running on target needs to include a diverse range of function calls, branches and loops;
  • The parts of the code running on target during the test need to take a known amount of time.
• discrimination issues that we only made explicit through this process:
  • The timer that provides the timestamps on target must be independent from the timer that controls the amount of time that the code runs.
• representativeness issues that we only made explicit through this process:
  • The test must run on hardware where execution times are difficult to control, as well as on hardware where execution times are easy to control.

We'll describe the process and some of the results in more detail in our presentation at Avionics Europe on Wednesday 20th February at 14:45.