=Paper=
{{Paper
|id=Vol-2066/seerts2018paper02
|storemode=property
|title=Quality Indicators for Automotive Test Case Specifications
|pdfUrl=https://ceur-ws.org/Vol-2066/seerts2018paper02.pdf
|volume=Vol-2066
|authors=Katharina Juhnke,Matthias Tichy,Frank Houdek
|dblpUrl=https://dblp.org/rec/conf/se/JuhnkeTH18
}}
==Quality Indicators for Automotive Test Case Specifications==
https://ceur-ws.org/Vol-2066/seerts2018paper02.pdf
Quality Indicators for Automotive
Test Case Specifications
Katharina Juhnke Matthias Tichy Frank Houdek
Daimler AG Ulm University Daimler AG
Group Research & MBC Development Institute of Software Engineering Group Research & MBC Development
Ulm, Germany and Programming Languages Ulm, Germany
Email: katharina.juhnke@daimler.com Ulm, Germany Email: frank.houdek@daimler.com
Email: matthias.tichy@uni-ulm.de
Abstract—Testing is an important quality assurance activity As data source, a total of 816 test case specifications from
during development of automotive software. Automotive OEMs a total of 16 diverse projects have been collected from an
and suppliers use test case specifications to specify, mostly infor- automotive OEM. The included test case specification were
mal, test cases as well as supporting information like traces to re-
quirements. While the quality of the test case specifications has a either created in-house or by suppliers.
high influence on the quality of the subsequent testing, quality of In Section II, we give an overview about test case specifica-
informal automotive test case specifications have not been investi- tions including an example. Section IV contains a description
gated yet. In this paper, we present 7 potential quality indicators, of potential quality indicators resulting from the analysis
ranging from requirement coverage to contents of a test step. The
quality indicators have been identified in a case study of 816 cur- including quantitative data on selected test case specifications.
rent test case specifications specified by an OEM and suppliers. Thereafter, we conclude and present some outlook on future
work in Section V.
Index Terms—Automotive software testing, test case specifica-
tion, quality indicators
II. AUTOMOTIVE T EST C ASE S PECIFICATION
I. I NTRODUCTION
A lot of innovation in the automotive domain is nowadays Test case specifications are a central part of the test docu-
addressed by software and electronic systems. Solid testing mentation [2] in the automotive environment. They are used
processes are an integral part of the development process to document the test cases to be performed. A test case
to verify that the implemented software works as expected. specification contains a set of test cases that are necessary to
Standards like ISO 26262 [1] or Automotive SPICE [4] require adequately test a particular test object according to defined test
a consistent test documentation. An essential part of the test objectives. A test case is basically characterized by a unique
documentation is the test case specification, which is defined identifier, pre- and postconditions, inputs, expected results,
by the software testing standard ISO 29119 [2]. The test case priority for the test execution and traceability information (e.g.
specification contains a set of test cases derived from the test references to the associated requirements) [2]. In addition to
basis for a particular test object [3] these test case basic attributes, other domain or company
Failures and misinterpretation of the test case specification specific test case attributes are often specified such as status,
shall be avoided at any time. While techniques like mutation test objective, author, model series or test platforms. These
testing can be used to assess the quality of automated tests attributes are called test case meta data.
[8], automated approaches to assess the quality of informal Test case meta data and basic attributes are relevant for a test
test case specifications are scarce [5] or restricted to domain- case and can be defined in a test case specification template.
specific test languages, i.e. for Testing and Test Control An example of an automotive test case and the application
Notation (TTCN-3) [6], [7]. In order to identify potential areas of a test case specification template is shown in Figure 1.
of improvement for automotive test case specifications, the aim The test case of a wiper and wash system is described by its
of this paper is to examine how the quality can be evaluated basic attributes and automotive specific test case meta data
using a given test case specification template. (e.g. vehicle family, test platform). The test case attributes are
The analysis is based on data extracted from the test case represented by the columns and the rows are used to define
specifications suitable as indicators for quality assessment different object types such as text, headlines, test cases or test
of test case specifications. The analysis provides insights steps. Not all attributes are relevant for each object type, so
into the specification of automotive test cases and identifies some cells shall be empty (dark gray cells in Fig. 1). Different
whether there is sufficient formalization within these test case object types have a hierarchical relationship to each other. For
specifications to enable an automated evaluation of the quality. instance test steps must be assigned to a test case.
SEERTS 2018: Workshop on Software Engineering for Applied Embedded RealTime Systems @ SE18, Ulm, Germany 96
(a) Test case basic attributes
(b) Test case meta data attributes (dashed border)
Fig. 1. Example of an automotive test case definition using a test case specification template
III. DATA C OLLECTION the identified quality indicators are discussed on the basis of
the various criteria examined.
We collected data from a IBM Rational DOORS database
of an automotive OEM and identified a total of 2435 test case
A. Criterion 1: Size of the test case specification with respect
specifications. Thereafter, obsolete test case specifications have
to requirement specification.
been excluded reducing the number of test case specifications
to 972. Obsolete test case specifications are those not based The average number of test cases in a test case specification
on the current test case specification template or whose last is 511 (cf. number of test cases in Fig. 2). However, very
modification date is before 2015. Furthermore, duplicates, large test case specifications contain more than 2200, small
backups, or test case specifications marked as obsolete by ones can contain only 31 test cases (cf. Fig. 2). However,
name have not been included resulting in a further reduction to the size of the test case specification is not very meaningful
816 test case specifications from a total of 16 diverse projects. and does not make any statement about the correctness of the
Projects are related to vehicle domains such as powertrain, test case. Therefore, they must be considered in correlation
chassis or comfort systems. to the testable requirements specified in the corresponding re-
The identified test case specifications based on a test case quirement specification. For instance, a test case specifications
specification template as shown in Figure 1. The test cases with 856 test cases (cf. Fig. 2, TCS 12) can be classified
are derived from natural language requirements and there- as large. The requirement specification associated with test
fore the relevant test case attributes are also specified by case specification 12 contains 2462 testable requirements. It
using natural language. is recommended that one requirement shall be tested by at least
We analyzed the collected data quantitatively based on in- one test case. Therefore, it is obvious that there should be more
formation that can be determined programmatically. Therefore, test cases to verify 2462 related requirements. However, the
we used the DOORS Extensible Language (DXL) to extract relationship between the size of the test case specification and
quantitative data from the identified test case specifications. the number of testable requirements can only be an indicator
of the completeness of the test case specification. Reference
values for an appropriate size of a test case specification can
IV. P OTENTIAL Q UALITY I NDICATORS be determined by previous test case specifications of a system.
The results of the analysis are presented in detail based on a
representative project. The selected project includes a total of B. Criterion 2: Distribution of contained object types.
12 test case specifications from the powertrain domain that are Test case specifications contain a large number of test cases
representative for automotive test case specifications. Figures and test steps, as shown in Figure 2 by the green bars. Objects
2 - 5 show the analysis results for this project. In the following, of type base scenario indicate reusable preconditions that can
SEERTS 2018: Workshop on Software Engineering for Applied Embedded RealTime Systems @ SE18, Ulm, Germany 97
Fig. 2. Number of test cases (dark green) and other contained object types per test case specification
be referenced by several test cases. The analysis revealed that herent test procedure descriptions are avoided. It also supports
there is often only a very small number of base scenarios in the assignment of documented expected results to a clearly
the test case specifications or that base scenarios are not used defined set of inputs and actions. For very small test cases, the
at all. Instead, preconditions are often copied from test case template used for the analyzed test case specifications allows
to test case instead of reusable constructs being stored and the definition of actions and expected results without using
referenced in base scenarios. This is particularly noticeable test steps. The analysis revealed that test case specifications
in large test case specifications. By using base scenarios, exist when test steps are omitted completely (see TCS 3, Fig.
duplicates could be reduced and the reuse of established 3). In such cases, it can often be observed that the inputs and
constructs can be increased. Reduction of duplicates also expected results are overloaded. Several pairs of inputs and
minimizes the amount of time and effort needed to make expected results are combined in one test step. That means that
changes in the reused parts, since the changes can then be it is no longer possible to unambiguously correlate the inputs
made centrally. Test case specifications also contain undefined to the expected results. This is also the case when test case
objects to which no object type is assigned (e.g. TCS 11, Fig. specifications use both approaches. In most cases, a mixture of
2). A small number of test case specifications contain object both approaches is found within a test case specification (e.g.
types that are individual and not predefined by the template. 9 of 12, Fig. 3). This form is more error-prone and makes it
This is a violation of the template, which can have an effect difficult to understand.
on export to downstream tools.
C. Criterion 3: Size of test cases. E. Criterion 5: Type of linked object types.
The average size of a test case, measured in terms of The analysis of the linked object types revealed links
the number of test steps to be performed, is 3,83 test steps. between test cases and artifacts. The table in Figure 5 shows
However, test cases with up to 76 test steps (see TCS 8, Fig. 3) a linkage scheme with allowed linkages. In general, these are
could also be identified in the representative project. Large test external links (e.g. artifacts are requirements whose correct
cases are more error-prone in the later test execution and make implementation shall be verified, see TCS 8 in Fig. 4) or
debugging more difficult. For instance, it is time-consuming to internal links, i.e artifacts such as base scenarios which are
execute 76 test steps manually in order to reproduce a failed referenced by a test case, see TCS 11 in Fig. 4). In some cases
step. In addition, an excessive number of test steps can be no requirements have been linked (cf. TCS 5, 6 and 7 in Fig.
an indicator that several test cases have been combined into 4). In such cases, there is an insufficient requirement coverage.
one large test step. Hence, it makes sense to split them up in The link analysis can also be used as quality indicator to detect
several test cases. errors in the documented requirements, i.e. if the object type
is not set for a requirement (undefined links, e.g. TCS 6, 7 and
D. Criterion 4: Type of test case specification. 11 in Fig. 4) or the requirement specification is not based on a
Test cases usually have test steps that structure the test case standard template (unknown links, e.g. TCS 5 in Fig. 4). In the
flow. This has the advantage that very large, extensive and co- case of the considered test case specifications and the linked
SEERTS 2018: Workshop on Software Engineering for Applied Embedded RealTime Systems @ SE18, Ulm, Germany 98
Fig. 3. Number of test steps per test case and type of test case specification
Fig. 5. Detailed evaluation of all links from test case objects (rows) to target
Fig. 4. Number of links according to the object types of the targets for each objects (columns) according to the linkage scheme (table). The cells contain
test case specification the total number of all links summarized from all test case specifications.
requirement specifications, the object type requirement must because it seems rather unlikely that a single test case can
be set for requirements. Furthermore, non template compliant cover that many requirements. According to our experience,
and incorrect links can be detected (see black bars in Fig. more than 20 linked requirements can be considered critical.
4). This includes links to headings, information or process
requirements that cannot be considered as testable in the G. Criterion 7: Template conformity.
context of a requirements-based testing approach.
The analysis revealed several violations of the template
guidelines. These have a significant negative effect on the fur-
F. Criterion 6: Number of linked object types.
ther processing of the test case specification (e. g. automated
A test case is linked to an average of 1,68 requirements. This verification mechanisms are not applicable or the export to
also corresponds to the premise that each requirement should downstream tools fails). For example, customizing the chapter
be checked by at least one test case. However, there are also structure (adding or renaming chapters) means that test cases
enormous deviations where a test case with 121 requirements are not included in the export or the export fails. Furthermore,
is linked. Such a high number of linked requirements can be it can be an indicator for the quality of a test case speci-
seen as an indicator to check such a test case, either because fication if certain mandatory attributes have not been filled
it is too large and could be divided into several test cases or (e. g. inputs, expected results, test platform, model series).
SEERTS 2018: Workshop on Software Engineering for Applied Embedded RealTime Systems @ SE18, Ulm, Germany 99
V. C ONCLUSION requirements, adherence to a given linking scheme or the
Our investigations show that test case specifications are not implementation of the test objectives and test platforms defined
completely documented in accordance with the guidelines of in the test concept. These criteria can be used to gain a first
the used test case specification template. Therefore, the root impression of the quality of the test case specification. This
cause needs to be investigated further. Adequate formalization could be a condition for whether a more detailed examination
is required for an automated quality assessment of the test case is reasonable at the given time.
specifications. Due to the highly individualized attribute sets Our future work will focus on the qualitative analysis of test
of the test case specifications, a structural evaluation based case descriptions, which are mostly based on natural language.
on the template does not appear to be feasible for all test Mechanisms are required to support and accelerate reviews of
case specifications. It is also difficult to evaluate the content test case specifications.
of the test cases and compare them with the contents of the
requirements and the test concept programmatically, since the R EFERENCES
test cases are usually written in prose. In order to perform [1] ISO 26262, Road Vehicles - Functional Safety, 2011.
automated quality assessments, test cases must be formalized [2] ISO 29119, Software and Systems Engineering - Software Testing, 2013.
and template guidelines must be fulfilled. Furthermore, it [3] ISTQB, Glossary of Testing Terms, 2015.
[4] VDA QMC Working Group / Automotive SIG. “Automotive SPICe:
should be noted that contents from the test concept and the Process Reference Model Process Assessment Model”, 3rd ed, 2016.
underlying requirements play an important role in the quality [5] R. Lachmann and I. Schaefer, “Towards Efficient and Effective Testing in
assessment of a test case specification. Since this information Automotive Software Development”, GI-Jahrestagung, 2014, pp. 2181–
2192.
is not available in the same data format (e.g. requirements, [6] B. Zeiss, D. Vega, I. Schieferdecker, H. Neukirchen and J. Grabowski
test cases in DOORS and test concepts in Word) an automated “Applying the ISO 9126 Quality Model to Test Specifications - Exem-
evaluation of compliance with the guidelines is not possible. plified for TTCN-3 Test Specifications”, Software Engineering GI, vol.
15 (6), pp. 231–242, 2007.
The analysis shows that there exist criteria which can [7] H. Neukirchen, B. Zeiss and J. Grabowski “An Approach to Quality
be used as quality indicators for a first quality assessment Engineering of TTCN-3 Test Specifications”, International Journal on
with regard to a examination of the structure of a test case Software Tools for Technology Transfer, vol. 10 (4), pp. 309–326, 2008.
[8] Y. Jia and M. Harman, “An Analysis and Survey of the Development
specification. This includes in particular the test case size of Mutation Testing”, IEEE Transactions on Software Engineering, vol.
with respect to the requirement specification, number of linked 37 (5), pp. 649–678, 2011.
SEERTS 2018: Workshop on Software Engineering for Applied Embedded RealTime Systems @ SE18, Ulm, Germany 100