Structural Coverage Analysis (SCA)
In this article, we will discuss about Structural coverage analysis (SCA). This is a very important topic, specially when you are doing software verification/validation for avionics software.
What is Coverage?
Coverage refers to the adequacy of the verification activities of the airborne software. It is a measure, not a method or a test. Therefore, coverage is expressed as the percentage(%) of the activity. Coverage is most frequently applied to the testing activities. Coverage is a very crucial measure, specially for the project manager. It act as an additional exit criteria and helps the management to decide when to stop the test activities. For example, 100% requirement coverage and 100% MC/DC coverage could be one of the exit criteria for a Level-A software testing activities.
When we have the high code coverage, that means that most of the code structure got exercised during the testing phase. It also indicates that there are very low chances of finding bugs in the software as compared to the software which has low code coverage.
Here is the code coverage formula:
There are two types of test coverage –
1) Software requirement coverage
2) Software structure coverage (SCA)
Requirement Coverage Analysis
Requirements coverage analysis determines how well the requirements-based testing verified the implementation of the software requirements and establishes traceability between the software requirements and test cases. (section 184.108.40.206 of DO 178B).
The requirement coverage ensures –
- Did we get a test for each requirement?
- The requirements are being tested and brings visibility to the results of the tests.
- Establish the traceability between requirements and test cases.
Requirement Coverage – How to achieve?
The requirement coverage analysis activity requires the traceability between the software requirements and the test cases. The requirement coverage can be achieved:
1) By analyzing the trace data
2) Analysis to confirm that the requirement bases test (RBT) cases satisfies normal and robustness testing.
3) Adding extra test cases based on the gap found during the requirement coverage analysis.
Structural Coverage Analysis
Structural coverage analysis determines how much of the code structure was executed by the requirements-based tests (DO-178B, section 220.127.116.11), and establishes traceability between the code structure and the test cases (DO-178B, section 6.2).
Structural Coverage – How to achieve?
Sometimes, the requirement bases test cases are not enough to exercise certain code structure. before starting the structural coverage analysis, we need to establish the traceability between the code structure and the test cases. Here are the steps to achieve the structural coverage:
1) Analyze the SCA information from the requirement based testing (RBT).
2) Identify the SCA gaps and add additional test cases if required
3) Update software code structure if required
4) Regenerate SCA information by exercising (RBT + additional test cases) on the updated instrumented software code.
5) Provide resolution to SCA
Why isn’t the requirement coverage good enough?
There are several reasons, why the requirement coverage alone is not good enough. The primary reason is that the requirement coverage alone can not confirm that the software code-base does not contain unintended or extra functionality. The requirement coverage is not enough to say that the implemented software code structure has been adequately tested.
Why Structural Coverage Analysis (SCA)?
Structural Coverage Analysis (SCA) is mandatory for any airborne software as per DO-178B/C.
- Required as per DO-178B/C.
- Provide the evidence that the software code structure was verified to the degree required for the particular software level.
- Helps to identify the missing requirement based test cases.
- Helps to identify missing or incomplete requirements.
- Provide the evidence that the software code-base does not contain any unintended functionality.
- Reveals the code structure that was not exercised during the requirement based testing (RBT).
- Very helpful for finding out the dead code/deactivated code.
Types of Structural Coverage
There are different types of structural coverage:
- Statement coverage
- Decision coverage
- Condition coverage
- Condition/Decision coverage
- Modified Condition/Decision coverage (MC/DC)
How to deal with (SCA) gaps?
There are could be several reasons as to why there are gaps in SCA:
1) Inadequate test case
2) Requirement/Source code error
3) Coverage Tool error
Here are the ways, you can deal with it:
1) Code Fix: Dead code exists in the software code base. These dead code need to be removed. Code was developed that has no associated requirement and the code is not needed.
2) Defensive code: Code that is intentionally inserted for robust behavior. This is often produced as a result of good programming practice. This is un-testable.
3) Manual Analysis: Deactivated code is verified by inspection or analysis instead of test. Therefore, manual analysis is required.
What is code instrumentation?
The code instrumentation process adds extra code to the given application to be able to produce the structural coverage data. The instrumentation code (extra code added by the coverage tool) collects the data during the test execution and provide the coverage data in human readable format. It is the job of the coverage tool to instrument the actual code.
If the program never reaches to certain code blocks during the test execution, the instrument code does not collect any data at that point.
There are several well-known coverage tool that are available:
1) LDRA ( Liverpool Data Research Associates)
3) RTRT (Rational Test Real Time)
In this article, I have briefly explained structural coverage analysis (SCA). I hope this article would be helpful for your better understanding. If you have any question/concerns, please feel free to comment below.