The publication addresses the topic of assessing the completeness of the test coverage of software code to ensure compliance with the criteria of the test process, which are given in ISO 26262 in Part 6. And are used for the automotive industry as an application level. The common case of cumulative coating collection, which is observed at the component levels, is taken into account. Attention is paid to the case when the component consists of modules that have been implemented by MATLAB Simulink. The publication contains some approaches to estimating the completeness of model coverage by collecting cumulative coverage.
for code coverage test. Problems and solutions describing alternative development methods, namely
Model Based Design, belong to Pelliccione, Patrizio [
But a good solution is to abandon the classical and certainly effective method of development, namely - writing program code, and adopt an alternative method - Model-Based-Design and its "righthand side" of the V-scheme of development—Model-Based-Test.
According to ISO 26262, the part of the software component (this also applies to the hardware implementation), the failure of which threatens the life and health of both the vehicle user and the environment, are classified on a scale of ASIL levels ranging from A to D, where D describes cases of the highest threat.
Along with other criteria indicating the safety of software implementation, there are recommendations for evaluating the completeness of test coverage by test scenarios, followed by an evaluation of each of these scenarios according to the expected behavior.
It is important to note those levels of software implementation that require coverage assessment. Thus, according to ISO 26262 - 6 (9.4.5) - MCDC coverage is recommended for ASIL A, B, C and strongly recommended for ASIL D.
From a Model-Based-Design perspective, the "Unit" abstraction must be implemented within a single file and must occupy a subsystem allocated to it.
The Simulink environment with Simulink Coverage, Simulink Report and Simulink Test allows you to get an estimate of the test coverage for a single module.
For the most part, according to recommendations from Mathworks, the Test Harness is based on the Simulink Test tool. Test Harness includes such basic blocks as Test Assestments, Test Sequences, and Unit Under Test, which perform the roles of test sequence verification, test sequence generator, and the object under test, respectively. As shown in Fig. 1.
Thus, one module is implemented in a subsystem, and both compliance and the percentage of test coverage can be evaluated for it by embedded methods.
If we consider the developed software from a higher degree of abstraction, then, by integrating with each other, software modules form software components.
Modules within components have gained additional entities - interfaces. These interfaces have properties, in particular—data type. Coverage of two or more modules by tests implies that intermediate interfaces are also covered by tests, thus reducing the risk of errors at the interface level.
An important task is to develop methods for evaluating the completeness of component coverage after the first level of integration.
Mathworks provides several tools for assessing coverage at the integration level. One of them is
based on the Design Verifier Toolset and consists in looking for Dead-Logic in the implementation.
Design Verifier does not actually evaluate coverage. It points to parts of the implementation that are
not involved in any of the code or model scenarios. Thus, this tool does not comply with
recommendation ISO 26262-6 (9.4.5).
4. Results
name
refModelcovIds
topModelcovId
}
1
"TestHarness_NAME"
[2 1]
[
Another way, supported by Mathworks, is to use Simulink Coverage Tools, available with MATLAB, certified for ISO 26262 and ISO 29119 and gives access to coverage metrics for each of the component subsystems.
For example, Mathworks recommends using the following code: cvdo = cvsim (mdl) cvsave (<path_to_file>, cvdo) cvhtml (<path_to_file>, cvdo) with the .CVT file labeling the section describing the coverage. The section looks as follows: CV_Database { modelcov { id end // alternative method of the testing end otherwise % Custom Tool disp (' /// NDA'); disp ([' In the TestSuite exists ' num2str(NDA) ' TestCases']); warning ('off','all'); SimOut = sim (['PATH TO FILE ' NameReq '.slx'],'ReturnWorkspaceOutputs','on'); warning ('on','all'); res = 'PASS'; % Default - The test is Done
for TestCaseNum = 1:HowManyTestCasesAreHere % Repeat for eatch TestCase % Find Fail testcase. ('-1' - The Test is FAIL; % '1' - The Test is PASS. if (min (eval (['NDA.REQ_' NameReq '_result.TestCase' num2str(TestCaseNum) '.Data'])) res = 'FAIL'; TestStatus(TestCaseNum) = -1;
TestCaseStatus = 'FAIL'; else
TestStatus(TestCaseNum) = 1;
TestCaseStatus = 'PASS'; end disp ([' Unit Test for TestCase #' num2str(TestCaseNum) ' is ' TestCaseStatus]); end disp ('- - - - - - - - - - - - - - - - - - - - - - - - - - - - -'); disp (' <strong>UNIT TEST RESULT:</strong>'); disp ([' for REQ-' num2str(NameReq)]); disp ([' ' TaskTicket]); disp ([' ' res]); TestSuiteStatus = [' REQ-' num2str(NameReq) '(' TaskTicket ') is ' res]; disp ('- - - - - - - - - - - - - - - - - - - - - - - - - - - - -'); res = TestSuiteStatus; %% COVERAGE CALCULATION % precondition coverage.complexity = -1; coverage.condition = -1; coverage.decision = -1; coverage.execution = -1; coverage.mcdc = -1; warning ('off','all'); % if option is activated then ... if (TestMode.coverage.condition == true || ...
TestMode.coverage.decision == true || ...
TestMode.coverage.mcdc == true) disp (' <strong>Coverage and Complexity:</strong>'); mdl = ['NDA_' NameReq]; mdl_subsys = [PathTo.DesignModel 'NDA_' NameUUT]; open_system(mdl); open_system(mdl_subsys); [coverage.tests, coverage.data] = cvload (cvtFileName); if TestMode.coverage.decision == true cov = decisioninfo (coverage.data{1}, ['DesignModel_' NameUUT]);
During operation and augmentation, the method described by Mathworks showed critical dependencies, specifically: 1. in a component, all interfaces must be shared by blocks; 2. if a model is connected as a linked model, then the linked model must have the same dimensionality as the model to which it is authenticated.
If these conditions are not met, the method is invalid.
Promising for further consideration is a method that is based on the creation of Simulink libraries, which would contain a component entirely. In this case, all the blocks included in the components would have common interfaces with each other, so they would be suitable for collecting cumulative coverage. However, this approach to estimating coverage is redundant due to the need to build and maintain a library component. It is also not the typical, recommended by Mathworks, solution, hence causing additional difficulties in the qualification and certification phase of the software product, methods and tools that were used to build the vehicle.
I thank my colleagues, namely Oleksandr Denysenko and Oleksii Chkalov, for their support in the process of working on this topic, without whom the work would have dragged on for a silly time.
Last but not least, I thank Igor Kozakevych and Artem Kharchenko for organizational support in producing the publication and Bogdan Poklad for editing and translation.