Automating test generation with AUTOSAC

As anyone working with safety-critical software knows, testing is a costly process. DO-178 and ISO 26262 guidelines set standards for software quality assurance that require significant effort to meet. In the avionics software industry, it has been estimated that over 50% of the cost of overall development is spent on testing!

How much of this cost could be avoided if test requirements were written in a computer-parseable language so that tests can be generated automatically from them?

In January 2016 we partnered with Altran UK, Rolls-Royce and the Universty of Oxford to answer just that. In the 15-month NATEP-funded AUTOSAC project, we aimed to AUTOmate the generation of tests from Spark Ada Contracts.

During the project, we used SPARK pre- and post-conditions to define requirements. These replace typical natural-language requirements while, critically, remaining separate from the code itself. Here's an example requirement for a function that calculates the square root of a postive integer:

function Integer_Sqrt (P : Positive) return Positive
   with Pre  => P > 0,
        Post => (Integer_Sqrt'Result + 1) * (Integer_Sqrt'Result + 1) > M
                  and then (Integer_Sqrt'Result * Integer_Sqrt'Result) <= M;

We used requirements like this example to generate tests for real-world Ada code that can replace tests written manually. Using RapiTestFramework, test generation was integrated into the workflow so tests could be easily generated, executed and reviewed. Also, by using the code coverage measurement and reporting capabilities of RapiCover, test results were reported clearly, allowing project partners to quickly identify coverage holes and potential problems in the requirements and implementation of the code.

Using the process in a real-world case study, we demonstrated that it produced robust tests and reduced the time and effort needed to test requirements, while still being independent of the code under test. You can find more information on the project in the final report.

Looking ahead, we will continue to work with Rolls-Royce, Altran, and the University of Oxford to develop automated test-generation solutions that reduce the time and cost of safety-critical software development. This development will take place in future research projects such as SECT-AIR, case-studies, and through direct commercial involvement.

If you would like to receive updates on RapiTestFramework, make sure you are signed up to our mailing list.