Going above and beyond the quality standards

We take the quality and testing of our software very seriously – as seriously as you take yours. We take it so seriously that we meet many criteria that aren't required (per DO-330) for qualification of our tools, based on their Tool Qualification Level, TQL-5. We do this because we think it’s good practice, and because it gives us confidence in the quality of our products, which we consider to be just as important to us as it is to our customers. In this blog, we share a few examples of how we ensure this quality.

Requirements-based engineering

A key element of our quality approach is that we take requirements-based development further than is required by DO-330 or ISO 26262. Rather than using only a single set of Tool Operational Requirements for our RVS tools, we add a second level of detail to these requirements to better ensure that we examine the full scope of tool inputs. This is essential for a tool that deals with the complexities of different programming language standards.

Within that context, we write the developer-TOR, the internal tool requirements, the tool test plans and the tool component tests, and we manage full upwards and downwards traceability of all of these items. We ensure that all items are reviewed independently and against each other, hence complying with items in DO-330's “ Table T-6: Testing of outputs of integration process ”. This approach means that we have baselines both for the development and qualification of our tools. This is an example of how we respect the DO-330 criterion “ T8-2 Baselines and traceability are established ”, for which compliance is not required at TQL5.

This is a very laborious process, but it gives us confidence in understanding exactly how our tools behave, giving credence to the claim that we take quality very seriously.

Testing and automation

We’ve developed our own test harness and infrastructure that lets us fully automate running our tests. We run our full testing suite (a close equivalent to running system acceptance tests) multiple times daily as part of our in-house continuous build system. This reduces the risk of human error and lets us run tests often and detect any defects early in development.

We run state-of-the art testing – not only of nominal cases, but also of robustness cases, range cases etc. Our full qualification kit test suite includes over 180,000 lines of code. Of course, it isn’t just about the quantity of test code, but about testing the right components of our tools, and we have this covered – for example, we test nested decisions up to a nesting depth of 80 to ensure that our tools work with rare (but possible) corner cases. This gives us visibility and confidence in the completeness of our test suite.

Configuration management process

We use version control and issue-tracking (bug) management systems. This respects the DO-330 criterion “ T8-3 Problem reporting, change control, change review, and configuration status accounting are established ”, which we meet even though it‘s not needed at TQL-5. Whenever we fix a bug in our software, we write a test demonstrating that the bug is no longer present. We then add this to our growing suite of regression tests to ensure that we never reintroduce past errors in our software.

We maintain our tool life cycle environment under revision control, and we tag it automatically as part of the tool release process. This complies with another DO-330 criterion that’s optional for us – “ T8-5 Tool life cycle environment control is established ” – and helps us provide support to customers using previous versions of our tools. We also tag and securely store all of our tool releases for future use or reference. Finally, we can (and sometimes do) re-run the release process and compare the built artifacts with the ones we stored previously to ensure that we are able to rebuild superseded releases.

Plans and standards

We have internal processes that we apply to tool releases and non-conformance scenarios, and we perform periodic internal audits. We also have internal coding standards, requirements standards, and testing standards that we follow and audit against. This meets the DO-330 criterion “ T9-1 Assurance is obtained that tool plans and standards are developed and reviewed for consistency ”.

Assurance issues

There is no software without bugs. We do everything we can to find defects in our tools as early as possible and fix them. If we find defects that affect our assurance case, we let you know about them instead of hiding them, as is well explained in our post on our Assurance Issue process.

Conclusion

At Rapita, we don’t just do what we must to develop and qualify our tools, we do everything we think is important to deliver quality tools that we’re confident about, so that you can rely on them for your software verification.

Whatever your tool qualification needs are, we believe that we can meet them. We currently offer tool qualification support for the use of RVS tools in DO-178B, DO-178C and ISO 26262 projects. For more information, visit our qualification page or contact us.