-Software Product Lines (SPLs) are an approach to improve reusability of software in a large number of products that share a common set of features. In SPLs, Feature Models (FMs) are frequently used to model commonalities and variabilities. However, according to the best of our knowledge, there are no approaches to automatically generate test cases on the basis of a stand-alone FM. We introduce a method, which fills this gap. In single system software testing, Classification Trees (CTs) are a proven approach for generating test cases derived from the original system specification. In this paper, we explore the relations and similarities between FMs and CTs and integrate both methods to a unified approach called Feature Model for Testing (FMT).
SPL-engineering is one of the most promising approaches
of the Software Engineering community to reduce the
development costs, for as well as to increase the quality of families of
similar software product instances [
As a matter of fact, we are, therefore, running into a situation
where each instance of a certain brand of car possesses a
unique configuration of the embedded software of all its ECUs.
Testing all these millions of instances of an automotive SPL in
the following traditional way is no longer feasible: create all
actually used instances of an SPL and then develop for each
instance a separate suite of integration test cases. Hence, the
automotive industry as well as engineers from other domains
are urgently looking for new methods how to systematically
generate sets of software product instances that represent
equivalence classes of instances with a sufficiently similar
behavior from a system integration testing point of view.
Furthermore, model-based and more traditional black and
white box testing approaches are adapted in such a way that
families of test models and derived test cases can be developed,
together with semi-automatic procedures that allow one to
select the appropriate test cases for a specific SPL instance.
Examining more closely the state-of-the-art of SPL
development and software testing approaches in the automotive
industry we see that various kinds of feature modeling concepts
and tools are used for the design of SPLs and the selection
of needed instances [
To overcome these problems we will first present in Section II
of this paper the fundamentals of SPL description by means of
FMs and the specification of parameter equivalence classes for
testing purposes by means of CTs. Furthermore, this section
introduces our paper’s running example, a subset of a case
study provided by the Adam Opel GmbH which is a subsidiary
of GM (General Motors) Corp. Afterwards, we explain in more
detail the state-of-the-art of systematic testing approaches of
SPLs and black box testing with CTs in a related work section.
Section IV then systematically compares the basic modeling
elements of FMs, similar to the developed FMs in FODA [
II. FUNDAMENTALS
The contribution of this paper is to generate variants and corresponding test cases on the basis of one representation of the SPL.
Our running example is a very simple subset of hardware and software components of the recently released Opel Insignia. We restrain ourselves to four sensors, two actors (engines), and one software component. The sensors are rain light sensor (RLS), turn indicator sensor (TIS), steering angle sensor (SAS), and the vehicle speed sensor (VSS). The RLS detects rain and the TIS indicates the driver’s choice to turn left or right. SAS senses the steering angle and the VSS senses the speed of the car. Two different types of engines serve as hardware features: a 1.6 liter and a 2.0 liter turbo engine. Additionally, we use one feature of the (Adaptive Forward Lighting) AFL+ technology. The bending light is a functionality which belongs to AFL+ and realizes an adaptive curve light. All parameters presented in this paper are provided by the Adam Opel GmbH. We use the running example to exemplify the differences and commonalties between FMs and CTs and to motivate our approach of integrating both to a Feature Model for Testing (FMT).
SPLs provide a high level reuse of software in a
specific problem domain [
Single features edge with filled circle edge with unfilled circle
Group of features
filled circles and connected edges
unfilled circles and connected edges
Cardinality
Formal description
1..1
0..1
1..n
n..m
every FODA extension because our approach is not limited to
a certain FM. We also aim e.g. to support the Orthogonal
Variability Model (OVM) proposed by Pohl et al. [
The FM in Fig. 1 shows an excerpt of the Opel Insignia FM. All dependencies and constraints within the FM have to be taken into account when deriving a product.
After extracting a variant from the FM, suitable test cases
need to be built. For this purpose the CT-method was
introduced by Grochtmann and Grimm [
In Fig. 2 three test cases for the CT of the running example are given. Each test case consists of the equivalence classes marked by the black dots in a horizontal line.
Miscellaneous approaches exist dedicated to SPL testing.
Although there are no real FM-based testing approaches, many
test methods partially use the FM of the SPL under test. A
summary of methods is given in [
Incremental Testing: In this approach, the first product
derived from the SPL is tested individually, for instance by using
the CT-methodology. With respect to the commonalities
between the different products, the following products are tested
using regression testing techniques [
Reusable assets: The goal is to create reusable test assets.
To ensure reusability, these assets are created during domain
engineering. For each product these assets are customized
during application engineering. Pohl et al. apply model based
testing techniques. The authors use activity diagrams which
are developed in domain engineering, based on requirements,
and customized during application engineering to derive test
cases. The so called ScenTED approach focuses on the reuse
of test cases [
Division of responsibility: In [
Furthermore, we are not aware of any SPL testing approach
that combines SPL variant selection strategies with
parameter value selection strategies as supported by CT. The tool
pure::variants [
CTs are a widely used technique for test case generation.
There are many publications related to this topic. A CT can
be used to test a single product instance derived from the
FM. The resulting product instance has specific actors and
sensors, which interact complying to the specification. A CT
splits the input domain of the sensors into relevant equivalence
classes. These classes can be used to test an actor that is
part of the product instance. Several approaches to improve
and extend CTs like the Classification Hierarchy Table [
All CT-based testing approaches we are aware of share the drawback that they deal with single product instances only and thus are only compatible with a product-by-product SPL testing approach.
As introduced in the preceding sections FMs and CTs have been used in software engineering for rather different purposes until now. An in-depth comparison of the modeling language constructs of FMs and CTs in the sequel reveals that both modeling approaches share a majority of their concepts. Therefore, this section is structured as follows: the first subsection starts with the in-depth comparison of FM and CT modeling constructs. Based on the results of this comparison, the following subsection then selects a minimal number of language constructs that correspond to a superset of the concepts of both FM and CT. Finally, the last subsection introduces a metamodel that captures the essential design of the new integrated FMT modeling language from an abstract syntax point of view.
In this section we develop language constructs which present a minimal list of abstract language concepts that is a superset of the concepts of FM and CT. Table II lists the constructs of FMs and CTs and the corresponding constructs for the unified approach: FMT. First, we have to define which kinds of nodes the FMT needs to support. CTs distinguish between compositions, classification, and equivalence classes. Composition and classification (line 1 in Table II) in CTs are nodes with child elements and equivalence classes (line 3 in Table II) are leafs in a CT. In FMs only compound features contain child elements and leafs are always features. To integrate CTs and FMs with regard to the different kinds of nodes, we need to examine the differences. Compositions are always mandatory and classifications are always optional nodes. As a consequence, we can use the compound feature known from FMs, and use cardinalities to state that the compound feature is either optional (0..1) or mandatory (1..1). Therefore, we adopt the compound feature for the FMT approach. The leafs of CTs and FMs differ obviously. On the 1. 2. 3. 4. 5. 6. 7. 8. 9. 10.
Feature Model compound feature Feature none mandatory feature (1..1)
optional feature (0..1) (1..n) features (n..m) features cross-treeconstraints feature attributes feature types
Classification Tree composition, classification
none equivalence
class composition classification equivalence class none only between equivalence classes none none one hand, an equivalence class represents a value or range of values of a parameter necessary for testing. On the other hand a feature in FMs can be any feature or property of an SPL. To realize an integration we need a leaf, which is capable to represent both information: a feature in general and a representation of values of parameters. Another difference, which we have to take into account for the integration, are the cardinalities. In FMs features and compound features may have four different types of cardinalities to describe commonalities and variabilities. In CTs three of them are present: composition (1..1), classification (0..1), and equivalence classes (1..n). We adopt the missing cardinality of FM to ensure that the FMT is as expressive as the original FM. Furthermore, all four types of cardinalities may be used on all levels of an FMT tree in contrast to the much more restrictive approach of CT.
In addition, we allow cross-tree-constraints between all nodes of the FMT (line 8 in Table II). Finally, we adopt the node attributes and node types from feature modeling (line 9 and 10 in Table II).
We now describe the integration of FM and CT by means of a metamodel depicted in Fig. 3. We developed this class diagram on the basis of Table II. Please note that the depicted class diagram is a small extract of the complete FMT metamodel that illustrates the concept of the unified approach. It does e.g. not contain any information concerning the test case generation. The characteristics of the nodes of the FMT approach are described using the following classes: Compound Feature, Feature, and Atomic Feature. The last one can either be a feature representing its property in form of a literal or an interval. This is realized using the two classes Literal and Interval. These classes may represent: a value or a range of values of test parameters as known from equivalence classes in CTs a value or a range of values of parameters needed for product instantiation purposes the labels of features known from FMs. A node in the FMT can only be a Compound Feature or a leaf node (Literal or Interval), because the classes Feature and Atomic Feature are abstract. Compound Feature, Literal, and Interval inherit properties from Feature. All nodes in the FMT may have a Type and an arbitrary number of Attributes. In a Type the information of the data type of a feature is stored if existent. This is important, for instance, to distinguish between parameters of an integer or real data type. Attributes can store any information of the node. To obtain sufficient information to properly plan the test effort it is for example important for vehicle OEMs to embed information about realizing a feature in hardware or software. Additionally, we can apply constraints between Features and, therefore, also between Compound Features and Atomic Features. According to FMs we allow Require and Exclude constraints between all nodes. Since we want to adopt the cardinalities, describing the relation of features or groups of features to their parent node, we model a relation between Features and Compound Features by means of the class Dependency, which contains the cardinality constraints as attributes (minimum and maximum cardinality).
Fig. 4 depicts an object diagram that shows how dependencies and cardinalities are used to distinguish between the four different categories of subfeatures of Table II. A Feature is always connected to its Compound Feature by a dependency. Therefore, Dependencies with different cardinalities can be placed beneath a Compound Feature.
In this section we briefly describe the application of our FMT approach by generating test cases for our running example. Additionally, we discuss a tool, which is under development in our research groups.
We demonstrate the ability to derive products and test cases on the basis of the FMT using our running example depicted in Fig. 5. We now derive two different products and present the handling of the parameters. The two products differ because they use different types of engines which results in different configurations. The 2.0 Turbo ECOTEC engine allows a vehicle maximum speed of 240km/h and, therefore, needs to be tested above 200km/h. The 1.6 ECOTEC engine is limited to 192km/h and does not require the test instance for vehicle speeds above 200km/h. Therefore, when testing the product containing the 1.6 ECOTEC engine, the equivalence class representing a speed range between 201 and 250 km/h has to be disregarded.
For both instances we derived some example test cases, which was done according to the well-known procedure described in section II-C. In the following we will describe the FMT in Fig. 5 which integrates the information of the FM of Fig. 1 and the CT of Fig. 2. Leaf nodes of the FMT, therefore, either represent basic features of the Opel Insignia SPL or equivalence classes of (runtime) parameter values. All nonleaf nodes of the FMT are inherited from the FM of Fig. 1, whereas leaf nodes are inherited from Fig. 1 and 2. The nodes of the new FMT have to be interpreted as follows: All nodes below the sensors node represent optional or mandatory Opel Insignia sensors together with their output parameter value definitions which are used as input parameters for control function test cases.
All nodes below the actors node represent available actors (actuators) for the Opel Insignia such as different types of engines. Input parameters that control the behavior of these actors have been omitted due to lack of space. These missing parameters are output parameters of to be tested control functions. Specifications of their values can be used as oracles during the execution of test cases. A node like bending light represents a group of control functions that shall be tested. Again due to lack of space we do assume that bending light consists of a single function only which consumes output values of a subset of all sensors and produces input values for a subset of all actors defined in Fig. 6.
Node attributes (which are not visualized in Fig. 5) are used to distinguish these different categories of nodes of FMTs as well as for other purposes like the definition of additional node selection constraints (cf. metamodel of Fig. 3). Furthermore, Fig. 5 shows that the optional bending light functionality still requires the optional rain light sensor (as well as all other mandatory sensors of our SPL). The fact that the bending light control function also requires input values from the three other mandatory sensors is not visualized in Fig. 5. A more realistic FMT splits the bending light functionality into a number of subfunctionalities such as curve light, rain light, city light, or highway light which have to be tested separately. As a consequence we have to distinguish between features (functions, parameters, sensors, actors) that are part of a regarded product instance, but irrelevant for a specific test case, and features that are directly involved in a specific test case. The test case specifications in the lower part of Fig. 5 use black and white shapes for these two different purposes.
The selection of a specific variant and associated test cases is specified in a style adopted from CTs (due to lack of space the FMT metamodel of Fig. 3 does not cover these elements). Different shapes on vertical lines are used to distinguish between the following three cases, when a certain feature or parameter is selected:
2) circle: selection at installation time (flash time) 3) triangle: selection at runtime
The first two options correspond to the selection of a certain variant in an FM, whereas the third option corresponds to the selection of test cases with parameter values in a CT.
Regarding the bottom part of Fig. 5 we can see that the type of engine is of course selected at design time. The bending light functionality can be added or removed by firmware updates as long as the optional rain light sensor is present. The fact that all presented variants with their associated test cases do contain the optional rain light sensor is implicitly represented by the fact that all test cases possess a parameter value definition for this type of sensor (Wet or Dry). Black triangles represent Wet and Dry values that are actually used in a specific test case, whereas white triangles represent the fact that the rain light sensor is present and computes output values that are not needed as input for the just regarded test cases. Finally, Fig. 5 shows that four of the six depicted test cases are related to the functionality of the bending light (black bending light circles). In general, the distinction between black and white shapes related to other nodes of the FMT reflect the information which features and parameter values are relevant for which test case. It consists of nodes describing the product line and nodes for the test cases. Test case values are depicted in brackets.
Fig. 6. Using FMT for SPL testing
In this subsection we describe in more detail how the FMT
approach is used for SPL testing purposes, i.e. we sketch
a methodology how to generate the bottom part of Fig. 5
automatically. For this purpose we describe the current state
of development of our MoSo-PoLiTe project, which realizes
the test methodology described in [
1) Adding dependencies derived from system architecture
and code as additional edges in the FMT.
2) Simplification of the FMT such that the resulting FMT
uses a minimal set of modeling concepts.
3) Integrated selection of variants and runtime parameter
values using the pairwise combination approach of [
Within this paper we have presented a new approach how to integrate SPL engineering with feature models (FMs) and black-box testing of system functions with CTs. Our motivation for this line of research is based on the fact that both FM- and CT-based methods are, e.g., well established in the automotive industry for embedded software system development purposes. On the one hand FMs are a suitable modeling method to describe commonalities and variabilities of an SPL. CTs, on the other hand, support the generation of test cases, using equivalence classes of parameter values of a regarded system function. We are not aware of any integrated approach that combines both methods for the generation of variants as test candidates and the associated test cases. Today, FM-based methods are first used to select one variant after the other; then for each of these variants a separate CT has to be defined which is then used to guide the related test case selection process. The integration of FM and CT in the form of the presented new FMT (Feature Model for Testing) method seems to be the perfect symbiosis of two very promising and widely used techniques. The key advantages are: (1) we use a single model for SPLs and test case generation, (2) approaches known from FMs and CTs can still be applied, and (3) generation algorithms of variants and test cases can be combined. We are currently working on different projects using FMT for SPL testing purposes including the BMBF project feasiPLe. Ongoing and future research activities will address the following problems: Checking the consistency of the FMT with respect to an additionally available specification of the behavior of the studied system. The FMT needs to incorporate the complete functionality described in the specification. This must be done before generating test cases.
In embedded systems, our main application area,
realtime constraints play an important role. Furthermore, test
cases often have to be executed in a specified order and
continuous parameter values reflecting physical properties
of a controlled system and its environment have to be
synthesized from sequences of selected discrete
parameter values using well-known interpolation methods.
Defining a measure of completeness for the generated test
scenarios with respect to an additionally available system
behavior specification is challenging, too [
The nodes of the FMT have to be extended to describe the cost of creating configurations and requirements priority which is essential for complex SPLs in the automotive sector.
We are currently applying our approach to several SPL scenarios. These are real world examples and FMTs generated randomly.
We apply the pairwise testing approach [
Finally, we have to address the problem that realistic complete FMT models cannot be displayed directly as depicted in this paper. Various methods have to be developed how to collapse/hide irrelevant substructures efficiently.
This work was partially funded by the feasiPLe project, “Feature-driven, aspect-oriented and model-driven Software Product Line Development”, Federal Ministry of Education and Research (BMBF), Germany.