Testing Safety-Critical Software with AdaTEST

A full service, well supported and highly controllable tool for all your formal testing needs.

The increased adoption of embedded Linux within the general consumer electronics market gives rise to new areas of application development for embedded Linux outside the usual realm of PDAs and mobile phones. Industries such as avionics, railway signaling, process control and medicine are all users of embedded systems. Common to them all is a need for safety-critical software. Safety-critical software is a class of systems whose failure may cause injury or death to human beings. In addition to real-time requirements, including proper control over timing and scheduling, such systems have absolute demands regarding correctness of behavior. Please refer to Kevin Dankwardt's excellent article "Real Time and Linux" for more on real-time systems.

Strict formal methods are applied in developing safety-critical software. Counted among these methods are various forms of testing. Testing is performed to eliminate possible bugs and to ensure correctness of behavior. The requirements for developing safety-critical systems are so strict that even tools used in the development process must comply with minimum requirements for formal methodology.

One such testing tool AdaTEST, from the British company IPL. AdaTEST is, of course, a tool for testing Ada software. It has been audited and found qualified for use on projects complying with the RTCA's DO-178B, an international safety standard for the avionics industry. AdaTEST therefore can be used for developing safety-critical systems. However, a pertinent question arises: AdaTEST is designed for testing software written in Ada; with the power of C at hand, why bother with programming Ada for the Linux platform?

Linux Programming with Ada

Ada and Linux aren't a necessarily obvious combination. As several free and/or commercial real-time Linux implementations already are available on the market, the infrastructure for developing safety-critical Linux systems is in place. Unlike general purpose languages, say C and Java, hard real-time requirements are inherent in the Ada core language's tasking model. The task is an Ada-language construct equivalent to the operating system's thread. Due to its strong typing, we cane be confident that Ada programs contain few surprises--a perfect match for developing safety-critical software. Ada has therefore become a de facto standard for industries like avionics and railway signaling.

As for embedded platforms, Ada was originally developed by the US Department of Defense for use in embedded system applications. It is therefore a perfect match for the future's embedded, safety-critical Linux solutions.

But how does Ada mix with Linux? In fact, it mixes quite well. The GNU Ada tool chain (GNAT) is an Ada front-end to gcc, tying Ada closely with the operating system. With standard facilities to import C functions, Ada allows for metal-near programming by importing any C functions, including system calls if need be.

AdaTEST

Despite its commercial license, AdaTEST comes with out-of-the-box support for GNAT, which makes it interesting for developing Linux software.

AdaTEST provides facilities for dynamic testing, coverage analysis and static analysis. Dynamic testing is what most of us know by the general term "testing". Its purpose is to make sure the software does what it should. Coverage analysis produces metrics to evaluate whether the tests are sufficiently thorough. A static analysis assesses the software's complexity and use of language constructs. Although important parts of the AdaTEST suite, dynamic and static analysis are outside the scope of this article.

AdaTEST consists of a test harness and a library. The harness provides facilities to run, verify the results of and document dynamic tests. It consists of a set of library directives that are accessed from the test script. The test script is the basis for all your testing; it is simply an Ada procedure that exercises the software being tested.

To make sure the software does what it is supposed to do, the output is verified. Verification is handled with a CHECK function. The CHECK call compares an actual output value with an expected value, and it returns a true or false response, depending on the result. AdaTEST ships with CHECK functions for all of the types defined in Ada. AdaTEST also comes with CHECKs to compare memory blocks and check for external events, as well as a set of generic CHECK functions for instantiation to verify your own types.

The test harness allows you to compile the test script into an executable. Once the executable is run, a test report is written to an ASCII file. Events classed as unexpected are marked with >>, followed by an appropriate error message. A typical example of an unexpected event is a CHECK that returns false. The report ends with a test summary that prints the number of passed CHECKs, the number of failed CHECKs, the number of unexpected errors and all possible script errors (i.e., syntax errors in the test script). At the very end of the report, an overall test result is recorded. The test script fails if one or more unexpected events have occurred.

______________________

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Re: Testing Safety-Critical Software with AdaTEST

Anonymous's picture

We used AdaTEST on a DO-178B project too. The tool was fantastic, and the support from IPL, the best I've ever encountered from a tool vendor.

I take on-board the last comment - about testing the code against the code - but that's really not the way to run a safety-critical project. We tested against our design specifications and reaped the benefits of doing so.

Re: Testing Safety-Critical Software with AdaTEST

Anonymous's picture

Before considering a product like AdaTest, or IPL's C equivalent Cantata, make sure your project isn't just using them so a box can be ticked. I worked on an ATC project where they were both used, consumed many man hours, and achieved nothing because the engineers were coding the tests from looking at the source code under test. In effect, they were testing the compiler, not the project code. They should have been writing test scripts based on some requirements and design documentation.

That ATC project is still hitting the headlines! Go by boat!

Webinar
One Click, Universal Protection: Implementing Centralized Security Policies on Linux Systems

As Linux continues to play an ever increasing role in corporate data centers and institutions, ensuring the integrity and protection of these systems must be a priority. With 60% of the world's websites and an increasing share of organization's mission-critical workloads running on Linux, failing to stop malware and other advanced threats on Linux can increasingly impact an organization's reputation and bottom line.

Learn More

Sponsored by Bit9

Webinar
Linux Backup and Recovery Webinar

Most companies incorporate backup procedures for critical data, which can be restored quickly if a loss occurs. However, fewer companies are prepared for catastrophic system failures, in which they lose all data, the entire operating system, applications, settings, patches and more, reducing their system(s) to “bare metal.” After all, before data can be restored to a system, there must be a system to restore it to.

In this one hour webinar, learn how to enhance your existing backup strategies for better disaster recovery preparedness using Storix System Backup Administrator (SBAdmin), a highly flexible bare-metal recovery solution for UNIX and Linux systems.

Learn More

Sponsored by Storix