Structural Coverage Target: The proof of a 100% structural source code coverage is required by almost all functional safety standards (IEC61508, ISO26262, DO 178C, etc.). In the individual SIL / ASIL levels, only the type of source code coverage is differentiated. Essentially, the Statement Coverage (low SIL / ASIL Level), the Branch Coverage and the MC / DC Coverage (high SIL / ASIL Level) are required. For good reasons, however, e.g. no path coverage required. These would mean that you would check all the combinations of paths that are possible in a software. This would be an extremely high multiple of test cases compared to MC / DC coverage.

In the following blog, I discuss the question of the advantages and disadvantages of determining the coverage on the final hardware as opposed to a coverage analysis on a simulator provided by the compiler manufacturer.

Spontaneously you may come to the conclusion that a coverage measurement on the final hardware is the best method for safety reasons. Some of the arguments that are repeatedly put forward are: Dead code is found under real conditions. The coverage is relied upon taking into account the real HW / SW interfaces. Formal unit tests are not required any more.
For a meaningful answer, it has to be clarified whether hardware is “only” the microcontroller and allocated memory. Alternatively the focus of the test could also be the entire hardware including I / Os,  FPGAs etc.

Structural Source Code Coverage – The common practice:

The measurement of code coverage  by executing the source code on the microcontroller has been a common practice in aviation for decades. From these many years it can be concluded that there are almost no mistakes, which are found additionally. A test on a simulator shows exactly the same errors in most cases. Here there is no recognizable advantage of the test on the target. It should also be noted that in this procedure virtually all software is not instrumented at once, but only relatively small parts. Because of the available tools on the market, however, it is not a big effort to determine the complete coverage.

Structural Source Code Coverage measurement on the real hardware:

A “real” test on the hardware does usually include the complete hardware (e.g. Microcontroller I/Os, FPGAs, etc.). However, with the conventional methods of coverage measurement, the problem arises that the timing behavior of the software as well as the memory requirement will change massively if the software is instrumented. The instrumentation is again necessary to be able to determine the coverage at all. This means, in practice quite often, that the software is no longer or only very limited to run on the real hardware.

However, in order to carry out tests in this way, it is often the case that only parts of the software are instrumented, tests are carried out accordingly, and the total coverage is then added up. The problem here is, however, that no commercial tool support is available for this. This results in a completely unacceptable effort to determine the overall coverage. With very time-critical systems, any instrumentation can lead to a meaningless test being no longer possible.

Solution:

The obvious solution to the above problem is a technology that no longer requires the source code to be modified, but which can still measure code coverage. Currently, there are developments from some tool vendors ongoing, which may result in such a solution. For selected hardware architectures, some solutions are already close to be finished.
The aerospace industry has developed an alternative approach which also achieves all safety goals with respect to the structural coverage:

1. All tests (unit, integration and system tests) are compiled with exactly the same compiler and compiler settings as used in the operational code.
2. Simulators and emulators are accepted for execution of coverage tests.
3. The tests are written 100% against requirements. The structural coverage is only determined in the background. Whenever a gap occurs, it is analysed. There are, in principle, only three possible gaps: incomplete requirements, incomplete test specifications, unused source code. With this approach, an 100% source code coverage is achieved, which is covered by 100% requirements-based tests.

Related HEICON Blog posts

• Structural Source Code Coverage – Cost without benefit?
• Structural source code coverage and Requirements – Is there any dependency?
• Requirement completeness using data- and control flow analysis

The HEICON team will be pleased to support you with our services if you have individual questions regarding your project. Please send a mail to: info[at]heicon-ulm.de.