Fault Injection Test in ISO 26262 – Do you really need it?
Fault Injection Test: The ISO 26262 defines the fault injection test as a test method for the system integration and unit test level (ISO 26262-4 [System] Tables 5, 8, 10, 13, 15, 18; ISO 26262-5 [Hardware] Table 11; ISO 26262-6 [software] tables 10, 13).
This method has certainly a large part in the implementation of a possible error-free and therefore safe system. My focus in this blog is on an efficient implementation of this test method in practice. I will compare practices in the aerospace with those in the automotive industry.
Die ISO 26262-4 [System] beschreibt den Fault Injection Test wie folgt:
The test uses special means to introduce faults into the item. This can be done within the item via a special test interface or specially prepared elements or communication devices. The method is often used to improve the test coverage of the safety requirements, because during normal operation safety mechanisms are not invoked.
The ISO 26262-4 [System] defines the fault injection test as follows: Fault Injection Test includes injection of arbitrary faults in order to test safety mechanisms (e.g. corrupting software or hardware components, corrupting values of variables, by introducing code mutations, or by corrupting values of CPU registers).
The following points are of particular importance for the subsequent considerations:
1) Requirements based testing and fault injection tests are treated as two (different?) test
2) The ISO 26262 assumes that you need the fault injection tests to increase the test coverage of Safety Requirements.
3) Within the fault injection tests arbitrary errors are introduced into the system to proof the safety mechanisms that have been encoded in the software to examine.
Now let’s take a quick look in the aerospace industry. How does this industry deal with the issue Fault Injection Test?
In the aerospace the standards DO-178C [Software], DO-254 [Hardware] and ARP 4754 [System] represent the comparable content of ISO 26262.
The term Fault Injection Test is not used in the aerospace standards. Instead the terms “Normal Range Test Cases” and “Robustness Test Cases” are representing all aspects of the testing. The DO-178C combines these two terms with the requirements-based testing. The following two questions are central in every safety-critical aerospace project:
1) Are the Requirements verified with Normal Range Test Cases?
2) Is the Source Code robust against High-Level Requirements?
In the aerospace industry, the requirements play a very important role. It is required that the entire source code is covered by appropriate requirements. This includes the code for the behavior of the system in case of failure. In relation to safety aspects, the correct functioning of this source code part has to be proven.
The requirements will be tested on normal range and Robustness Test Cases. Whereby the Robustness Test Cases on the Requirements aimed itself to question and possibly to find gaps in the requirements. The importance of the robustness testing is indisputable, but it is also important to recognize that the determination of whether an appropriate test “Passed” or “Failed”, is not possible any more in a lot of cases. In not a few cases, namely lack the expected system behaviour. Ideally, testers and system specialists together discuss the problematic cases. The result of the discussion is then always a determination of the correct system behaviour. A documentation of the results, is the definition / modification of requirements.
If you follow this procedure, this means that the relevant Robustness Test Cases also verify Requirements.
If one follows the logic in the aerospace industry, then fault injection tests contain both Normal range tests and partly Robustness tests. In any case, there is a corresponding requirement for each test.
If we take now the aerospace view and interpret the ISO 26262, with respect to the Fault Injection Test then some exciting result appear:
- A clear test result, including test results of fault injection tests, you can get only if the expected result is known. Requirements define this expected result. If you follow this logic, the following question arises: Where is the difference between a fault injection test and a requirements-based test in the ISO 26262? The fault injection testing is then “just” a subset of requirements-based testing.
- If fault injection tests are not available, there is definitely a lack of test coverage of Safety Requirements. Each Safety Requirement needs to be 100% verified. From this point of view, fault injection tests are not anything special.
- From the perspective of aerospace also “arbitrary” errors result in a system always in a defined system behavior. In this definition “arbitrary” equated with “a mistake from a defined (= known) list of errors”.
The fault injection test at the end simply represents a subset of all requirements-based tests. Nevertheless it is an important subset! Experienced Tester are needed to get good Fault Injection Test and equally good test environments are to be provided for this kind of tests.
In the automotive industry, the practical application of requirements engineering is a discipline that has been rolled out only in recent years. In many current projects which do have an ASIL classification, the number of requirements is still not very high. The ISO 26262, for sure, also had this pragmatic aspect in view, when the fault injection test was defined. The fewer requirements there are the more sense makes the definition of such a test method.
Another aspect is that fault injection testing can be used to validate the safety mechanisms in a early project phase. The arguments in the aerospace industry are strongly influenced by the need to proof the compliance to the certification authority. Often the early stages of projects tend to come a bit too short.
In summary, it can be said that fault injection tests represent an important test method. Best advantage of this method can be made, if requirements are available for the expected system or software behaviour.
Are you ready for a functional safety workshop, to identify improvement potentials in your development process? Send a mail to: info[at]heicon-ulm.de or call +49 (0) 7353 981 781.
DO-178C has clarified that robustness must be specified in requirements, and all requirements are to be verified. Responses to abnormal input conditions must be documented as expected behavior. Specification of robust behavior in response to abnormal conditions can be tested, and the completion of this objective is measured as completion of requirements verification. Of course the question then is, are our requirements complete. That is a more difficult validation question and not verification.
Fault injection just seems a little ‘random’ to me. How do you know when you are done? And when you think you are done, how close are you from actually being done?
If you can accomplish fault injection testing cheaply then by all means do so as a rough measure of quality, but don’t make any claims base on the outcomes of such tests.
You seem to be focusing on software testing, where what you say is correct.
For hardware, faults are something completely different: they are either degradation over time (e.g., think of fuses burnout, at a very small scale), or “glitches” in functionality due to external interference (e.g., cosmic radiation). You need to prove these do not cause serious hazards, and for that fault simulation is irreplaceable.
Feel free to contact us for more information: firstname.lastname@example.org http://www.optima-da.com
I’m not against Fault Injection Tests. The opposite is true – they are very important. However, my argument is, that they become even mor powerful, if you define an expected behaviour, i.e. Requirements. Your examples are very interesting, as it is really a challange to “implement” this faults. However, it should be also clearly be defined what reaction you expected if a fuse burnout happens. If cosmic radiation is a topic, its getting really difficult to implement “glitches” of functionality in a controlled and repeateable manner. So testing may be here not a realy option.
Thanks for throwing some light based on your experiences in Avionics industry. I however have some concerns with the way you look at ISO26262 (from an aerospace point of view)!
By stating that Requirements Based Testing subsumes Fault Injection Testing, there is an assumption that all requirements are specified (i.e. a comprehensive set of requirements exist, including expected behaviours in case of faults). This is tough to achieve in practice. We have seen that emergent properties like Safety pose very interesting validation challenges (and its after validation we get to know if the requirements are correct in the first place). My agrument here is that fault injection is helpful in validating safety requirements. If done efficiently during development (using a representative fault model), fault injection could eventually help in forming precise requirements and observing corner cases!
Now to the next interpretation!
You say “A clear test result, including test results of fault injection tests, you can get only if the expected result is known. Requirements define this expected result.”
To me, an activated fault usually creates an unexpected result (normal requirements specify only expected results) and it’s only safety requirements which try to understand (through FTA/FMEA analysis) and mitigate unexpected results. So to test the mitigation, unexpected conditions have to be created! An interesting question that popped up as I thought about it is whether Fault Injection only validates already existing safety requirements or does it end up finding an unexpected behaviour for which no requirement exists in the first place – in other words, can Fault Injection campaigns also uncover vulnerabilities not thought about during design phase? Going by ‘WYSIATI – What you see, is all there is’ principle, I’m very doubtful about it!
Lastly, for viewers reading this, especially seeing the title of the blog, they may fall under the impression that Fault Injection Tests are not needed! But everything has its place under the Sun. I think that you too meant fault injection would be meaningful in the light of an expected behaviour (which to me nothing but a safety requirement). So I think we both agree in principle except for the choice of words 😉
To summarize, I would say that until requirements get precise, so precise so as to also include faulty behaviour and reactions (this surely takes time), fault injection testing in ISO will be there to stay. After all, studying system behaviour under faults is so vital to the analysis of dependable and fault tolerant systems these days that there are dedicated research communities still working on representativeness, fidelity, etc of Fault Injection. Last thing biting here is a sufficient ‘END OF TEST’ criteria – as mentioned in one of the above comments.
I would be pleased to hear back your thoughts on my comments.
thank you very much for your comment.
As you said… we both agree except in the choise of words.
So the question comes up, why did I use this kind of wording?
The answer is, I want to highlight a problem which I see in automotive projects these days.
It is the project situation in a late phase. All the safety analysis, FMEA, FTA etc. are done. The software and hardware is “almost” development. Now Software Testing, HW/SW Integration Testing and System Testing shall be established. In this project situation the software/system reactions on failure cases should be defined., as the software/hardware developer where able to implement a failure reaction. The only place where they (should) take there information from are the functional requirements and the system/hardware/software architecture. So if the verification process starts, the failure rejections have to be defined.
From my point of view, you are talking more about the situation, when the system is defined, i.e. the early project phases. There is no question for me. In this situations when you prototpying etc. it is essential to also perform fault injection tests, based on fault models etc. The results of this tests need to be “just” doucumented” (in requirements), and then the situation I’m concerned would not appear.
Finally, we all know the world is not black and white. There is always some grey. In the aerospace, this grey is covered by robustness testing. And part of this robustness testing is fault injection testing. And as you mentioned, there are situations where we don’t know the expected test results. However, after the test is performed, we will know what we expect from the system and what not, as this is the motivation to perform such test. So if the add this information (by writing requirements), we have defined the expected behaviour.