Measuring Execution Times

In previous blog posts we defined the concept of execution time [see Explaining execution time and The difference between execution times and response times]. In brief the execution time is the time spent actually running a thread, and not time spent executing other threads.

Now it is time to consider how to actually measure execution times.

The traditional approach for measuring how long a software component takes to execute is to place instrumentation points at the start and end of sections of code to be measured. These instrumentation points are used to record the elapsed time, either by toggling an output port which can be monitored via an oscilloscope or logic analyzer, or by reading an on-chip timer and recording the resulting timestamps in memory.

Because this approach doesn’t take account of context switches that take place within the partition or between partitions, it measures response time rather than execution time.

Measuring response times in this way is extremely imprecise because the timings are affected by two, unrelated, sources of variability:

• Different execution times within the thread of execution being measured;
• The arrival of context switches (either inter- or intra-partition).

When combined, these two sources of timing variability can lead to significant, and difficult to predict variations in timing measurements, which are not all attributable to the thread being measured.

A better alternative is to take care to measure just the execution time of the thread, thus eliminating one of the sources of variability. This means either:

1. Run the thread in a way that ensures that context switches cannot occur (e.g. with interrupts disabled); or
2. Identify and instrument sources of context switches. To eliminate context switch overheads, the instrumentation must be placed as close to the start and end of the context switch as possible.

The first case is the easiest to manage – all measurements made correspond directly with the execution time of the thread. In the second case, you need to exclude periods spent executing other threads/partitions. This is illustrated below:

Based on the above instrumentation points, the execution time can be derived by excluding the time spent context switched (58344 – 54999) from the elapsed measurement time (62192 – 54234).

There are, of course, situations where neither of these approaches will work, for example on Windows applications. We will address these in future blog posts.

Once you've been able to eliminate the variability that arises from context switches, you are left with variability that arises from different execution times within the thread. See next week's blog for a discussion about how you can deal with this.