The measurement of structural source code coverage is nowadays defined as an important procedure in many functional safety standards. The non-intrusive measurement of structural coverage offers completely new possibilities in the future. For a long time, it was industry-wide consensus that structural coverage should and could only be determined in so-called white-box tests. In many textbooks, the measurement of structural source code coverage is even promoted as an independent test method. The following blog post shows why things are likely to change in the future.

Introduction of structural source code coverage:

First a short introduction to structural source code coverage.

This is a measure of the degree to which source code is covered by testing. There are different measures of coverage. Usually you start with the coverage of the executable source code lines. This means that once each line of code has been executed, you have 100% coverage.

The famous MC/DC Coverage proves that each parameter within a condition has an effect on the result of the decision. The minimum number of test cases required is the number of variables in a decision + 1.

The following figure illustrates test cases to achieve MC/DC coverage:

Conclusions from practical work:

Across industries, it has been shown that the simple measurement of structural coverage is not a quality statement about the product. This also applies if 100% structural coverage is measured.

What is the reason for this?

Tests that only aim to measure structural coverage find hardly any errors. Good tests always include the observation of the data flow by software. Practice shows more and more clearly that requirements that define functional expectations of the software are needed to find errors.

The measurement of structural coverage represents a meaningful quality statement about the software, only if requirements are taken into account when creating the tests. It is generally accepted that it is practically impossible to create complete requirements and tests for a system. Structural coverage is a very efficient corrective for this particular methodological weakness. A low structural coverage can have only one of the following 3 causes:

1. the requirements are not complete
2. the tests are not complete
3. the uncovered source code is not needed at all

Therefore, it makes no sense to treat structural source coverage measurement as an independent test method. The measurement only makes sense if the corresponding tests are based on requirements.

Conclusion:

In future pure White-Box-Tests (=Unit-Tests which only measure the structural Source Code Coverage) are not needed any more for any formal prove.

This means that for most industries, the pressure will increase to measure coverage at higher test levels. In embedded systems, this is usually the level at which the hardware and software are tested -The integration test. Only for this test level there are usually functional requirements are available.

However, up to now there was usually no technology at all to determine the structural source code coverage with a reasonable effort at this level.

This will change in the near future. The technology and tool development for the non-intrusive measurement of structural source code coverage is making great progress and will replace the classic source code instrumentation in the next few years according to my prognosis. Structural source code coverage can thus take on a completely new role in safety-relevant software development.

Are you ready for a functional safety workshop, to analyse improvement potentials in your development process, then send a mail to: info[at]heicon-ulm.de or call +49 (0) 7353 981 781.