Testing of platforms are challenges! Do you develop platforms in embedded systems in order to offer more customer solutions without having to make a completely new development for each customer project?
Or do you have used an embedded operating system in order to develop your application as independent as possible from the hardware?
For the development of the systems you have probably benefited from this policy, but what was about the testing? Was it clear and easy to test the platform? Was it easy to achieve complete test coverage? Were you able to quickly and efficiently create good requirements? No? Then you are right here!

Challenge – Functionality

In my experience, the testing of platforms is one of the biggest challenges for testing at all. Particularly in safety-critical projects a high test coverage has to be achieved. Likewise then comprehensible requirements for the platform must be in place.
Why is testing of such a system a particular problem? An important point is that tests are fairly simple, if a clearly defined functionality can be tested. Usually, that’s not the case when you testing platforms. As the name implies, the platform is only a basis on which a specific functionality is to be implemented.

Challenge – Interfaces

A platform is then usually good if many different functions can be developed. This means that the platform itself has a lot of possibilities. There are only few restrictions and the interfaces are as variable and flexible as possible. At the same time the platforms provides an abstraction layer for the hardware, so that the application to be developed has not to care about the interaction with the hardware. For operating systems, quite complex mechanisms are demanded, to fulfill the functional safety aspects. Platform developments generally run parallel to customer projects, so a classic requirements engineering is very difficult.
Each one of these points increases the testing effort! Combining the aspects creates a high complexity for testing such systems:

Interfaces are very variable and flexible 

That means for testing that various combinations must be tested. Extreme values are to be used in the test, which brings the test system to the brink of its possibilities.

Number of interfaces

The combinations of values to be stimulated as well as the possibilities of the reactions of the platform can quickly go towards “infinite”. Platforms do have typically a lot of software interfaces as well as physical interfaces (such as fieldbuses, analog or digital signals).

Platform abstracts the hardware

This means that many mechanisms are incorporated in the platform, but they are difficult to be seen from the outside. It is very difficult and sometimes impossible to design requirements. A prototype of hardware has to be available before the requirements can be completed. As a result, in practice, this means that many aspects are not described in Requirements and they are thus also not tested.

Complex mechanisms in oder to meet functional safety demands

The test of such mechanisms – especially if Robustness test cases are called – makes very high demands on test environments. Often many aspects can not be tested due to lack of skills of test environments. One example is the task scheduling. It must, for example, to be ensured that the non-safety-critical tasks do not affect the safety-critical tasks. That means it must be implemented in the operating system, a mechanism to exit non-safety-critical tasks in case of failure guaranteed, so that the critical tasks definitely gets it scheduled execution time. If we are in the range of milliseconds tasks here, then the test system for certain test cases may need to be twice as fast or even faster, to make the stimulation of data or the result interrogation correct.

Parallel Requirement Engineering (system/customer requirements and platform requirements

This quickly leads to the fact that customer requirements and platform requirements are in contradiction to each other. The customer requirement can not fully be implement on the existing platform. Likewise, the platform development is often faced with the problem, not even to know potential customers requirements. Then solutions are implemented which often end up inevitably in contradictions with customer requirements.

Solution proposal

What could be the solution of these problems?
Requirements Engineering: In hardly any other area of the development of embedded systems, the prototype phase is so fundamentally important as here. Due to the strong dependency on the hardware and the associated high complexity and many interdependencies of various parameters, it is usually not possible to establish the requirements completely and without errors, and just implement the system.
A more realistic way is to write first a few rough, and more general requirements and then to build a prototype of the system. The prototype phase clarifies usually in a very efficient manner the issues. Only then it is possible to specify the requirements in the necessary level of detail and accuracy. Unfortunately, the results of the prototype phase are very often not recorded in the requirements. Therefore writing good tests is a very difficult task.

Interfaces: There should always be specified as many limits as possible on one interfaces. This simplifies the testing considerably. According to the experience, it is also true that no software works with all combinations of extreme parameters. Usually there are physical or software coding boundaries which are often not documented. If these boundaries are documented, then for the user / integrator and tester the work will be considerably simplified. The number of interfaces can not really be limited at platforms usually. Here comes the quality of the tester to fruition. He has meaningful combinations of interfaces for the test to choose and he must make reasonable equivalence classes. The better this is done, the better the test coverage with a minimum number of test cases.
Quality of the test environment: Particularly when testing very safety-critical systems the test environment requires sufficient accuracy and real-time capability. If the limits are reached of what is possible, you can often get good results by using a sensible combination of test and analysis. In such individual cases, you should contact the assessor for the functional safety or to the certifying authority to obtain an agreement on the chosen test approach. Experience shows that an intelligent verification strategy does often significantly limits the cost for a test environment.

Conclusion

As described above, the testing of platforms is often very complex and makes high demands. Nevertheless, it is also possible to achieve high quality results with a well prepared approach. In this area, however, great experience is a strong guarantee of success.

Related HEICON Blog posts

• Implicit Testing – A good idea (Part 1)?
• Implicit Testing – A good idea (Part 2)?
• Comparison and evaluation of different test design techniques
• Risk-based testing: Method for identifying the right test cases
• Psychology of Testers
• Management aspects of testing
• Structural Source Code Coverage – Cost without benefit?
• Structural source code coverage and Requirements – Is there any dependency?

Are you ready for a status workshop, to analyse improvement potentials in your test engineering, then send a mail to: info[at]heicon-ulm.de or call +49 (0) 7353 981 781.