- Register for DO-178 Virtual Training
Multicore Timing Solution
Solving the challenges of multicore timing analysis
Our unique solution to multicore timing analysis produces execution time evidence for multicore systems.
By following a V-model process, our engineers investigate multicore systems and produce evidence about multicore timing behavior.
Our industry-leading tooling, including our unique RapiDaemon technology (which generates interference during tests), reduces analysis effort through automation.
Our approach has been designed to support projects within the CAST-32A and ISO 26262 context.
Working with us
- We recognize that each test project is different, and work with you to meet your needs.
- We run testing activities on-site, at our headquarters in the UK, and at Rapita Systems, Inc. in Novi, Michigan. We can support projects with UK / US EYES ONLY requirements.
- We can answer multicore timing questions and produce evidence for you, or implement a method and provide training so you can do so yourself.
RapiDaemons are specialized microbenchmark programs that generate contention on hardware resources such as buses, caches and GPUs.
They support multicore timing analysis by generating contention while multicore timing tests are run, allowing interference effects to be considered while performing the analysis.
Each RapiDaemon applies contention to a specific hardware resource on a specific hardware architecture, either matching a desired level of contention or maximizing contention on the resource.
RapiDaemons are built on the Barcelona Supercomputing Center's microbenchmark technology (MuBT).
Our CAST-32A Compliance packages provide a complete solution for certifying multicore aerospace projects in accordance with DO-178C and CAST-32A guidance.
It includes a range of products and services to address all CAST-32A objectives.
When developing safety-critical applications to DO-178C (CAST-32A) guidelines or ISO 26262 standards, there are special requirements for using multicore processors. Evidence must be produced to demonstrate that software operates within timing deadlines.
The goal of multicore timing analysis is to produce execution time evidence for these complex systems. In multicore processors, multiple cores compete for the same shared resources, resulting in potential interference channels that can affect execution time. Accounting for this interference and producing robust execution time evidence is a challenge addressed by Rapita’s Multicore Timing Solution.
By following a V-model process, our engineers investigate multicore systems and produce evidence about multicore timing behavior. Our approach has been designed to support projects within the DO 178C (CAST32A) and ISO 26262 context.
You can see an example workflow of Rapita’s Multicore Timing Solution in our Multicore White Paper.
Our CAST-32A Compliance packages comprise three components: a process, tool automation, and services.
Our multicore timing analysis process is a V-model process that we developed in line with DO-178 and CAST-32A. It follows a requirements-based testing approach that focuses on identifying and quantifying interference channels on multicore platforms. We help address CAST-32A objectives not related to multicore timing through engineering services.
The tools we have developed let us apply tests to multicore hardware (RapiTest) and collect timing data (RapiTime) and other metrics such as scheduling metrics (RapiTask) from them. We use RapiDaemons to create a configurable degree of traffic on shared hardware resources during tests, so we can analyze the impact of this on the application’s timing behavior.
Our CAST-32A Compliance services include tool integration, porting RapiDaemons, performing timing analysis, identifying interference channels, and others depending on customer needs.
No, but we are currently working on projects that will go through certification with the FAA and EASA.
Rapita is a recognized leader in multicore timing analysis, with multiple professional publications on the subject:
- Steven H. VanderLeest and Samuel R. Thompson, “Measuring the Impact of Interference Channels on Multicore Avionics” to appear in the Proceedings of the 39th Digital Avionics Systems Conference (DASC), San Antonio, TX, Oct 2020.
- VanderLeest, S.H. and Evripidou, C., “An Approach to Verification of Interference Concerns for Multicore Systems (CAST-32A),” best papers of 2020 SAE Aerotech, to appear in SAE International Journal of Advances and Current Practices in Mobility.
- VanderLeest, S.H. and Evripidou, C., “An Approach to Verification of Interference Concerns for Multicore Systems (CAST-32A),” SAE Technical Paper 2020-01-0016, 2020, doi:10.4271/2020-01-0016.
- Steven H. VanderLeest, Jesse Millwood, Christopher Guikema, “A Framework for Analyzing Shared Resource Interference in a Multicore System,” Proceedings of the 37th Digital Avionics Systems Conference (DASC), London, Sep 2018.
Our multicore team technical leads have also served in the following professional organizations:
- Vice-chair for the Enterprise Architecture (EA-25) subcommittee on airworthiness for the Future Airborne Capability Environment (FACE) consortium, 2020.
- Chair for the “Multicore Assurance” session of the 39th Digital Avionics Systems Conference, 2020.
- Chair for the “Integrated Modular Avionics” track of the 37th Digital Avionics Systems Conference, 2018.
Rapita Systems are uniquely positioned to offer the combination of expertise and tools required to effectively perform multicore timing analysis.
Whilst the challenge of certifying multicore systems for safety-critical applications is a relatively new one for the industry as a whole, we have been researching this area for over a decade. Rapita are working with key industry stakeholders, including major chip-manufacturers like NXP, to support them in refining the evidence required to satisfy certification authorities.
Rapita have extensive experience in providing software verification solutions for some of the best-known aerospace and automotive companies in the world. For example, BAE Systems used RapiTime (one of the tools in our Multicore Timing Solution) to identify worst-case execution time optimizations for the Mission Control Computer on their Hawk Trainer jet.
See more of our Case Studies.
In the CAST-32A position paper published by the FAA, an interference channel is defined as "a platform property that may cause interference between independent applications". This definition can be applied to a range of ‘platform properties’, including thermal factors etc.
Of these interference channels, interference caused by the sharing of certain resources in multicore systems is one of the most significant in terms of execution times. Interference based on shared resources may occur in multicore systems when multiple cores simultaneously compete for use of shared resources such as buses, caches and main memory.
Rapita’s Multicore Timing Solution analyzes the effects of this type of interference channel.
A very simple example of a shared resource interference channel is as follows:
In this simplified example, tasks running independently on the two cores may need to access main memory simultaneously via the memory controller. These accesses can interfere with each other, potentially degrading system performance.
There is no standard list that fits all platforms. Some interference channels can be more common than others, such as the ones related to the memory hierarchy (i.e. caches and main memory). The identification of interference channels (for which we provide a service) is an important activity that identifies the interference channels whose impact on the system’s timing behavior must be assessed.
How long does it take to comprehensively analyze the interference channels present in multicore hardware?
This depends on the platform, project needs, and whether we have already performed analysis on a similar hardware platform previously. Our solution includes an initial pilot phase in which we study the system and estimate the amount of time needed for subsequent phases. Typical projects run for between 2 and 12 months, depending on the scope of the analysis and complexity of the system.
We can analyze almost all hardware architectures. Our engineers work with you to determine the optimal strategy for integrating our RVS tools with your target, including hardware characterization and design considerations to best fit the hardware you're using.
To work with an architecture that is new to us, we first identify which metrics we can collect from the hardware, then adapt RapiDaemons for the architecture and implement a strategy to collect data from it.
We've worked with the following boards, CPUs and RTOSs:
|Ultrascale ZCU102||ARM A53||Deos|
|NXP QorIQ T2080||PowerPC e6500||VxWorks 653 3.0|
|NXP QorIQ T2080||PowerPC e6500||PikeOS 5.x|
|TI Keystone K2L||ARM A15||PikeOS|
|NVIDIA Xavier SoC||Carmel (ARMv8 variant by NVIDIA)||QNX|
|AURIX||Tricore TC377TX||Bare metal|
We also have upcoming projects with the following combinations of boards, CPUs and RTOSs:
|Ultrascale ZCU102||ARM A53 and R5||Helix/VxWorks7|
|NXP Layerscape LS1048A||ARM A53||Deos|
Some of the multicore systems that we’ve worked with are listed in our FAQ “Which hardware architectures can you analyze?”.
If we have already worked on a similar multicore platform to yours, it may take less time to perform hardware analysis for your platform.
Yes. We have already run projects analyzing the Nvidia Xavier AGX (CUDA) and we have ongoing projects analyzing AMD’s Embedded Radeon E9171 GPU featuring the CoreAVI Vulkan SC driver.
As per CAST-32A, each interference channel on the system must be analyzed to ensure that its effects on timing behavior have been mitigated. Some channels may be eliminated by architectural analysis or because they are deactivated for your system. For all channels that have not been eliminated, CAST-32A requires that multicore timing analysis be done to ensure that software operates within its timing deadlines. For each channel, one must provide analysis test reports and a summary to show that the set of tests is complete – Rapita provides this service to show completeness. Note that CAST-32A includes objectives other than testing, such as planning objectives. We provide templates to help you complete these.
Rapita have been providing execution time analysis services and tooling since 2004.
RapiTime, part of the Rapita Verification Suite (RVS), is the timing analysis component of our Multicore Timing Solution. Our customers have qualified RapiTime on several DO178C DAL A projects where it has been successfully used to generate certification evidence by some of the most well-known aerospace companies in the world. See our Case Studies.
As well as providing a mature tool chain, we support the customer in ensuring that their test data is good enough, so that the timing information they generate from the target is reliable.
Our RapiDaemons are configured and tested (see the FAQ: ‘configuring and porting’) to ensure that they behave as expected on each specific customer platform.
We also assess available observability channels as part of a processor analysis. This primarily applies to the use of performance counters, where we assess their accuracy and usefulness for obtaining meaningful insights into the system under observation.
RTOS vendors may provide partitioning mechanisms for their multicore processors, but these do not guarantee the complete elimination of interference. Instead, they are designed to provide an upper limit on the interference, sometimes at the expense of average-case performance.
In aerospace, these partitioning mechanisms may be referred to as ‘robust partitioning’. CAST-32A (the FAA’s position paper on multicore processors in avionics) identifies allowances for some of the objectives if you have robust partitioning in place – but it is still necessary to verify that the partitioning is indeed as robust as claimed.
From a certification standpoint, regardless of the methodology behind the RTOS vendor’s approach to eliminating interference, the effectiveness of the technology needs to be verified.
It is possible for companies to perform multicore timing analysis internally, but it is a highly complex undertaking which is very costly in terms of budget and effort. Anecdotally, one of our customers reported that it took them five years and a budget in the millions of dollars to analyze one specific platform.
Our Multicore Timing Solution is typically delivered as a turn-key solution, from initial system analysis and configuration all the way through to providing evidence for certification.
Some customers prefer to outsource only parts of the process to Rapita. For example, it is possible for a customer to purchase RapiDaemons under license and use them to gather and analyze their own data.
We’re completely flexible, and we understand that different customers have different needs. As such, you can purchase any component of our multicore timing analysis solution separately if that’s what you need. This includes, but is not limited to:
- Tool licenses (RapiTest, RapiTime, RapiTask)
- Services to integrate automation tools to work with your multicore system
- RTBx hardware to collect trace data from your multicore system
- Generic libraries of RapiDaemons
- Services to port RapiDaemons to your multicore system
- Services to perform multicore timing analysis
Yes: our approach can be used to get an in-depth understanding of how sensitive software can be to other software. For example:
- Task 1 executes acceptably in isolation and with most other tasks, but if it executes simultaneously with Task 127, its function X takes 10 times as long to return.
- This intelligence can feed into system integration activities to ensure that function X can never execute at the same time as Task 127.
The information from this type of analysis can also provide insights into potential improvements to the implementation of the two tasks. Sensitive tasks are not always the guilty party: other tasks can be overly aggressive and cause delays in the rest of the system.
For safety reasons, WCET will always be somewhat pessimistic. However, techniques that work well for single-core systems risk generating a WCET that is unreasonably large when applied to multicore systems, because the effects of contention can become disproportionate. The objective, therefore, is to calculate a value that is plausible and useful, without being optimistic. Optimism in relation to WCET is inherently unsafe.
It is not enough to identify how sensitive an application’s tasks are to different types and levels of interference; it is also necessary to understand what degree of interference a task may suffer in reality. It is possible to lessen the pessimism in WCET analysis by viewing the processor under observation through this paradigm.
The degree to which we can reduce pessimism is dependent on how effectively we can analyze the system. Factors influencing this include:
- The overhead of the tracing mechanism (which affects depth of instrumentation)
- The availability and reliability of performance counters
- The availability of information regarding other tasks executing on the system
- The quality of tests that exercise the code
Cache partitioning is all about predictability, not performance. Your code might execute faster on average without cache partitioning, but it’s probably not as predictable and can be quite sensitive to whatever executes in parallel.
Cache partitioning aims to remove all the sensitivity to other tasks sharing the caches, thus making your task more predictable – but potentially at the expense of overall performance. In critical systems, predictability is of far greater importance than performance.
Rapita’s Multicore Timing Solution can be used to exercise cache partitioning mechanisms by analyzing any shared – and usually undocumented – structures internal to the caches.
To analyze how a specific task is affected by contention on a specific resource, we need to be able to synchronize the execution of the task with the execution of RapiDaemons (the applications that generate contention on the resource).
Usually it is highly desirable to have RTOS/HV support for enabling user-level access to performance counters. Additionally, context switch information is very valuable when performing timing analysis.
Yes. Our solution makes it easy to specify the core on which you run your tests, and the level of resource contention to apply from each other core in the system.
We can also analyze systems that use non-synchronized clocks such as those often present in AMP platforms, by using the RTBx to timestamp data.
The maximum number of metrics we can collect depends on the performance monitoring unit(s) (or equivalent) on the hardware. An ARM A53, for example, lets us collect at least 30 metrics, but only access 6 in a single test. By running tests multiple times, however, we could collect all 30 metrics.
Developing a one-button tool solution for multicore timing analysis would be impossible. This is because interference, which can have a huge impact on a task’s execution time, must be taken into account when analyzing multicore timing behavior.
Analyzing interference effects is a difficult challenge that cannot be automatically solved through a software-only solution. Using approaches developed for timing analysis of single-core systems would result in a high level of pessimism, as it would assume that the highest level of interference possible is feasible, while this is almost never the case.
It is possible to collect a range of metrics by instrumenting your source code with the Rapita Verification Suite (RVS), including a range of execution time metrics:
- RapiTime: high-water mark and maximum execution times
- RapiTask: scheduling metrics such as periodicity, separation, fragmentation and core migration
It is also possible to collect information on events in your hardware using performance counters. The information we can collect depends on the performance monitoring unit(s) (or equivalent) of your system, but typically includes events such as L2 cache accesses, bus accesses, memory accesses and instructions executed. We can also collect information about operating system activity such as task switching and interrupt handling via event tracing or hooks.
Yes, we formally test and assess the accuracy of performance counters to ensure the validity of results we collect for the software under analysis.
A RapiDaemon is an application designed to create contention in a predictable and controlled manner on a specific shared resource in a multicore system. The effect of this contention can then be measured to identify interference channels and impacts on execution times.
Rapita have developed and optimized a set of standard RapiDaemons that target shared resources common to most multicore architectures, such as main memory. Some projects will require the creation of custom RapiDaemons for a specific architecture.
RapiDaemons are delivered as object files that are compatible with your multicore platform. To configure these for use on your platform, these need to be linked together with your software’s object files. After this initial work, RapiDaemons can be automatically initialized using RapiTest and RapiTime.
When we deliver RapiDaemons, we include documentation describing how to configure and initialize them.
Before timing analysis can begin, it is essential to confirm that existing RapiDaemons operate as intended on a specific board. If they do not, it may be necessary to customize them to make them behave as they are designed to (a similar intention to a target integration, where we get the instrumented application software running on a specific target).
'Porting’ and ‘configuration’ of RapiDaemons are roughly equivalent terms in this context. For RapiDaemons to work as intended, they need to go through a configuration phase where their internal parameters are tuned as required. They’re then tested to ensure compatibility with the platform in question.
Each type of RapiDaemon may be implemented for a different instruction-set architecture and platform. While the main logic behind their behavior remains the same, they must be ported to execute correctly on each new platform.
After linking RapiDaemons object files with your software (see “How do I use RapiDaemons?”), RapiDaemons are typically initialized through either RapiTest or RapiTime, which automate the initialization, making it easy to use RapiDaemons. You can, however, use RapiDaemons without using RVS tools.
Some of the multicore systems that we’ve worked with are listed in our FAQ “Which hardware architectures can you analyze?”. Standard RapiDaemons are available for many of the systems that we’ve analyzed. For more information, contact us.
We deliver RapiDaemons themselves as binary applications that are ported for compatibility with your multicore system. We also include documentation describing the purpose of each RapiDaemon in the library and how to use them. We recommend that you engage our multicore services to analyze your system and design interference channel testing, which uses RapiDaemons. We can equip you with the tools and tests to continue from there, or you can continue using our services to do it for you.
We include support and maintenance for RapiDaemons, giving you access to our support team and to new RapiDaemons that we develop.
Certification packages are available for RapiDaemons, including validation evidence for your use of RapiDaemons in projects complying with safety standards such as DO-178C.
Yes. We develop and test RapiDaemons against appropriate requirements, e.g. RapiDaemon M should access resource R when run N times.
RapiDaemons are licensed for a number of calendar years.
There are 3 licensing options for RapiDaemons, which suit projects using fewer or more multicore environments. For this purpose, a multicore environment includes the board, the CPU, and any external devices and configuration options being used, with any change to any of these yielding a new unique environment for licensing purposes.
Single use licenses give you access to RapiDaemons for a single multicore environment or project.
Re-useable licenses give you access to RapiDaemons for up to 3 multicore environments or projects.
Product line licenses give you access to RapiDaemons for up to 10 multicore environments or projects.
Ada-Europe & DASIA 2019
17 Jun 2019
Multicore processor testing and validation
03 May 2019
Highlights from Embedded World 2019
06 Mar 2019
Highlights from HIS 2018
13 Nov 2018