In this article, we will discuss Structural coverage analysis (SCA). This is a very important topic, especially when you are doing software verification/validation for avionics software.

What is Coverage?

Coverage refers to the adequacy of the verification activities of the airborne software. It is a measure, not a method or a test. Therefore, coverage is expressed as the percentage(%) of the activity. Coverage is most frequently applied to testing activities. Coverage is a very crucial measure, especially for the project manager. It acts as an additional exit criterion and helps the management to decide when to stop the test activities. For example, 100% requirement coverage and 100% MC/DC coverage could be one of the exit criteria for Level-A software testing activities.

When we have high code coverage, that means that most of the code structure got exercised during the testing phase. It also indicates that there are very low chances of finding bugs in the software as compared to software which has low code coverage.

Here is the code coverage formula:

There are two types of test coverage –
1) Software requirement coverage
2) Software structure coverage (SCA)

Requirement Coverage Analysis

Requirements coverage analysis determines how well the requirements-based testing verified the implementation of the software requirements and establishes traceability between the software requirements and test cases. (section 6.4.1.1 of DO 178B).

The requirement coverage ensures –

1. Did we get a test for each requirement?
2. The requirements are being tested and bring visibility to the results of the tests.
3. Establish the traceability between requirements and test cases.

Requirement Coverage – How to achieve?

The requirement coverage analysis activity requires traceability between the software requirements and the test cases. The requirement coverage can be achieved:

1) By analyzing the trace data
2) Analysis to confirm that the requirement bases test (RBT) cases satisfy normal and robustness testing.
3) Adding extra test cases based on the gap found during the requirement coverage analysis.

Structural Coverage Analysis

Structural coverage analysis determines how much of the code structure was executed by the requirements-based tests (DO-178B, section 6.4.4.2), and establishes traceability between the code structure and the test cases (DO-178B, section 6.2).

Structural Coverage – How to achieve?

Sometimes, the requirement bases test cases are not enough to exercise certain code structures. before starting the structural coverage analysis, we need to establish the traceability between the code structure and the test cases. Here are the steps to achieve the structural coverage:

1) Analyze the SCA information from the requirement-based testing (RBT).
2) Identify the SCA gaps and add additional test cases if required
3) Update software code structure if required
4) Regenerate SCA information by exercising (RBT + additional test cases) on the updated instrumented software code.
5) Provide resolution to SCA

Why isn’t the requirement coverage good enough?

There are several reasons, why the requirement coverage alone is not good enough. The primary reason is that the requirement coverage alone can not confirm that the software code-base does not contain unintended or extra functionality. The requirement coverage is not enough to say that the implemented software code structure has been adequately tested.

Why Structural Coverage Analysis (SCA)?

Structural Coverage Analysis (SCA) is mandatory for any airborne software as per DO-178B/C.

• Required as per DO-178B/C.
• Provide evidence that the software code structure was verified to the degree required for the particular software level.
• Helps to identify the missing requirement-based test cases.
• Helps to identify missing or incomplete requirements.
• Provide evidence that the software code-base does not contain any unintended functionality.
• Reveals the code structure that was not exercised during the requirement-based testing (RBT).
• Very helpful for finding out the dead code/deactivated code.

Types of Structural Coverage

There are different types of structural coverage:

1. Statement coverage
2. Decision coverage
3. Condition coverage
4. Condition/Decision coverage
5. Modified Condition/Decision coverage (MC/DC)

How to deal with (SCA) gaps?

There could be several reasons as to why there are gaps in SCA:
1) Inadequate test case
2) Requirement/Source code error
3) Coverage Tool error

Here are the ways, you can deal with it:
1) Code Fix: Dead code exists in the software code base. These dead codes need to be removed. Code was developed that has no associated requirement and the code is not needed.
2) Defensive code: Code that is intentionally inserted for robust behavior. This is often produced as a result of good programming practice. This is un-testable.
3) Manual Analysis: Deactivated code is verified by inspection or analysis instead of test. Therefore, manual analysis is required.

What is code instrumentation?

The code instrumentation process adds extra code to the given application to be able to produce the structural coverage data. The instrumentation code (extra code added by the coverage tool) collects the data during the test execution and provides the coverage data in a human-readable format. It is the job of the coverage tool to instrument the actual code.
If the program never reaches certain code blocks during the test execution, the instrument code does not collect any data at that point.

Coverage Tool

There are several well-known coverage tools that are available:
1) LDRA ( Liverpool Data Research Associates)
2) VectorCast
3) RTRT (Rational Test Real Time)

Conclusion:

In this article, I have briefly explained structural coverage analysis (SCA). I hope this article would be helpful for your better understanding. If you have any questions/concerns, please feel free to comment below.