FACE Technical Interchange Meeting - September 2021
Bell selects Rapita for Invictus 360 FARA Project
RVS 3.15 Launched
Current opportunities at Rapita
Introducing the RapiTest Editor
Software verification on the Solar Orbiter
Metrowerks CodeTest - How and why to upgrade
Leveraging FACE Conformance Artifacts to Support Airworthiness
DO-178C
MATLAB Simulink MCDC coverage and WCET analysis
Code coverage for Ada, C and C++
AMC 20-193
Aerospace TechWeek 2021
Safe Use of Multi-Core Processors Seminar
What is MC/DC?
Modified Condition/Decision Coverage (MC/DC) is a code coverage criterion commonly used in software testing. For example, DO-178C software development guidance in the aerospace industry requires MC/DC for the most critical software level (DAL A).
Code coverage is a way of measuring the effectiveness of your test cases. The higher the percentage of code that has been covered by testing, the less likely it is to contain bugs when compared to code that has a lower coverage score. There are three other types of code coverage that are worth considering in relation to MC/DC: Statement coverage, Decision coverage and Multiple condition coverage.
In addition to the criteria required by statement and decision coverage, MC/DC requires that 'Each condition in a decision has been shown to independently affect that decision's outcome'.
The MC/DC Criterion
“Each condition in a decision has been shown to independently affect that decision's outcome. A condition is shown to independently affect a decision's outcome by varying just that condition while holding fixed all other possible conditions".
Different types of code coverage
There are many different ways to approach code coverage; the ones we are concerned with are outlined in the table below:
|
Coverage criteria |
Statement coverage |
Decision coverage |
MC/DC |
Multiple condition coverage |
|
Every statement in the program has been invoked at least once |
✔ |
|
|
|
|
Every point of entry and exit in the program has been invoked at least once |
|
✔ |
✔ |
✔ |
|
Every control statement (i.e., branchpoint) in the program has taken all possible outcomes (i.e., branches) at least once |
|
✔ |
✔ |
✔ |
|
Every non-constant Boolean expression in the program has evaluated to both a True and False result |
|
✔ |
✔ |
✔ |
|
Every non-constant condition in a Boolean expression in the program has evaluated to both a True and False result |
|
|
✔ |
✔ |
|
Every non-constant condition in a Boolean expression in the program has been shown to independently affect that expression’s outcome |
|
|
✔ |
✔ |
|
Every combination of condition outcomes within a decision has been invoked at least once |
|
|
|
✔ |
Credit: Nasa - https://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/20010057789.pdf
MC/DC in action: The Cup of Coffee Example
Imagine making a cup of coffee. To make a warm and tasty cup of coffee, we'd need ALL of the following: a kettle, a cup and coffee. If any of the components were missing, we wouldn't be able to make our coffee. Or, to express this another way:
if( kettle && cup && coffee ) {
return cup_of_coffee;
}
else {
return none;
}
Or to illustrate it visually:
Tests 4 & 8 demonstrate that ‘kettle’ can independently affect the outcome.
Tests 6 & 8 demonstrate that ‘mug’ can independently affect the outcome.
Tests 7 & 8 demonstrate that ‘coffee’ can independently affect the outcome.
Please note, decision coverage and statement coverage are also covered in the selected tests.
The tests required for MC/DC analysis in this case are tests 4, 6, 7 & 8. Tests 1, 2, 3 and 5 are not required as we can satisfy the MC/DC criterion without them. These redundant tests need not be included in the coverage report.
MC/DC vs. MCC
- Multiple condition coverage (MCC) testing is characterized as number of tests = 2C. In our example we have three conditions (kettle, cup and cofffee) therefore tests = 23 = 8.
- MC/DC requires significantly fewer tests (C + 1). In our example we have three conditions, therefore 3 + 1 = 4.
Naturally, when there are more conditions to test, the amount of testing increases. In the above case, Multiple condition coverage (MCC) analysis would require twice as many tests as MC/DC analysis. In a real-world setting, most aerospace projects would include some decisions with 16 conditions or more.
Let’s look at what would be required to assess coverage for a decision with 16 conditions.
- Multiple condition coverage (MCC) tests = 216 = 65,536.
- MC/DC: tests = 16 + 1 = 17.
- Using MC/DC reduces the number of tests needed by 65,519.
As demonstrated above, when more complexity is added to a decision, the number of extra test cases required by MCC grows exponentially. This means that, when using MC/DC, we require a much smaller number of test cases in comparison to multiple condition coverage (MCC), while still maintaining a high error-detection probability.
Most coverage analysis tools support what they deem to be a "reasonable" number of conditions per decision (20 conditions per decision is common). RapiCover, however, is more powerful than most coverage analysis tools and supports up to 1000 conditions per decision.
Why MC/DC?
Aerospace and automotive guidance prioritises safety above all else in the software development lifecycle. With that in mind, truly ‘exhaustive testing’, as encapsulated by MCC, would be the safest and most rigorous approach in a perfect world. MC/DC represents a compromise that finds a balance between rigor and effort; positioning itself in between DC and MCC. MC/DC requires a much smaller number of test cases in comparison to multiple condition coverage (MCC), while retaining a high error-detection probability.
When is MC/DC required?
DAL A aerospace projectsAny software that is critical to provide (or prevent failure of) continued safe flight and landing of an aircraft is defined as being Design Assurance Level A software. Any aerospace software that is being created at this level of assurance must measure and report MC/DC.
ASIL D automotive projectsISO 26262 prescribes MC/DC for ASIL D (the highest criticality level) software.
