Leaders in Measurement-based Worst Case Execution Time (WCET) tools

What is the difference between RapiTime's WCET and end-to-end testing?

RapiTime works by observing the behaviour of a set of tests. From these tests it derives a lot of information about

  • the timing behaviour of small sections of code
  • loop iterations, and some other control flow
  • test coverage

Based on this information and the source code, RapiTime calculates which code is on the longest path. This path is not necessarily a path that has been tested (although all parts of this path must have been tested at some time during the testing).

As a very simple example, consider the code:

if (x)
{
  a();
}
 
if (y)
{
  b();
}

And we perform two test cases:

  1. x = 1; y = 0;
  2. x = 0; y = 1;

It is clear that the worst case path (x=1; y=1) has not been tested, however all parts of this path have been tested. RapiTime will deduce that the worst case path is when both a() and b() are executed, even though they were never executed together in one test case.

From this (partial) testing, you can have knowledge of

  • the predicted worst case path
  • the predicted execution time for this path (based on the maximum times for a() and b() during the partial testing)
  • which parts of the code have never been executed (therefore how to improve the testing)

The principal demonstrated above is applied throughout the whole application automatically. Thus the worst-case execution time (WCET) reported is the upper bound of the combination of all test parts, not just end-to-end test runs (although these are also reported for information).

If one of the test cases has not been tested (at all) then RapiTime will report this by indicating which parts of the code have never been exercised during testing.

And therefore we see the difference: with end to end testing, to measure the worst case execution time, you need to find and then test the worst case path; RapiTime is able to deduce the worst case path and estimate an upper bound on its execution time.

After performing this testing and analysis, it may be useful to try to focus testing effort on the predicted worst case path by trying to replicate the hypothetical test case that would cause a() and b() to both be executed. RapiTime highlights code on the worst-case path, assisting in the generation test cases to exercise the worst-case path. Those additional tests can be merged into the first set of test results (if required) to give more accuracy to the WCET calculation.

It is interesting to note the difference between the computed worst case execution time and the maximum observed execution time in any one test. If these two values have a big difference then it can indicate that the testing can be improved to test a longer path in one test case.