=Paper=
{{Paper
|id=Vol-2273/QuASoQ-07
|storemode=property
|title=Prioritization in Automotive Software Testing: Systematic Literature Review
|pdfUrl=https://ceur-ws.org/Vol-2273/QuASoQ-07.pdf
|volume=Vol-2273
|authors=Ankush Dadwal,Hironori Washizaki,Yoshiaki Fukazawa,Takahiro Iida,Masashi Mizoguchi,Kentaro Yoshimura
|dblpUrl=https://dblp.org/rec/conf/apsec/DadwalWFIMY18
}}
==Prioritization in Automotive Software Testing: Systematic Literature Review==
https://ceur-ws.org/Vol-2273/QuASoQ-07.pdf
6th International Workshop on Quantitative Approaches to Software Quality (QuASoQ 2018)
Prioritization in Automotive Software Testing:
Systematic Literature Review
Ankush Dadwal, Hironori Washizaki, Yoshiaki Fukazawa Takahiro Iida, Masashi Mizoguchi, Kentaro Yoshimura
Department of Computer Science and Engineering Control Platform Research Department
Waseda University Center for Technology Innovation - Controls
Tokyo, Japan Hitachi, Ltd. Research & Development Group
ankush.dadwal@toki.waseda.jp takahiro.iida.ac@hitachi.com
Abstract—Automotive Software Testing is a vital part of the in order to group concepts around a topic. Through analysis
automotive systems development process. Not identifying the criteria, it allows the quality of research to be evaluated. Herein
critical safety issues and failures of such systems can have the system review aims to identify common techniques in
serious or even fatal consequences. As the number of embedded
systems and technologies increases, testing all components automotive testing and to define new challenges.
becomes more challenging. Although testing is expensive, it The paper is structured as follows. Section II describes
is important to reduce bugs in an early stage to maintain related works. The systematic literature review approach is
safety and to avoid recalls. Hence, the testing time should detailed in Section III. Section IV presents the results obtained
be reduced without impacting the reliability. Several studies from the systematic review. Section V addresses potential
and surveys have prioritized Automotive Software Testing to
increase its effectiveness. The main goals of this study are to threats to validity. Finally, Section VI lists the conclusions
identify: (i) the publication trends of prioritization in Automotive and the definitions for future work.
Software Testing, (ii) which methods are used to prioritize
Automotive Software Testing, (iii) the distribution of studies II. R ELATED W ORKS
based on the quality evaluation, and (iv) how existing research Automakers have experienced the impact of the evolution of
on prioritization helps optimize Automotive Software Testing.
Index Terms—Automotive Software Testing, Prioritizing,
technology on automotive testing. Today, testing all systems
Systematic Literature Review manually is not only cost-intensive and time-consuming
but nearly impossible. Automating the testing phase would
I. I NTRODUCTION significantly reduce the cost of software development [14].
Literature about the prioritization efforts in the automobile
Currently the automotive industry is undergoing a major industry is scarce. Herein we focus on known techniques and
transition. Automakers have been adding new functions and their applicability to the investigated domain.
systems to meet the market’s demand for an ever-growing In the past few decades, numerous studies [5], [6], [7],
amount of software-intensive functions. However, these new [8], [9], [10], [13], [15], [16], [17], [19], [20], [24], [27]
functions and systems have some negative aspects. One have demonstrated that vehicles are becoming increasingly
is that automakers must enhance their testing techniques more complex and more connected. For example, an empirical
because vehicle complexity is increasing. Testing typically study, which aimed to investigate the potential regarding
consumes more than half of all development costs [4]. While quality improvements and cost savings, employed data from
testing a single software system is difficult, testing without 13 industry case studies as part of a three-year large-scale
prioritization is even more challenging due to the exponential research project. This study identified major goals and
number of products and the number of features. Today, strategies associated with (integrated) model-based analysis
software determines more than 90% of the functionality in and testing as well as evaluated the improvements achieved
automotive systems and software components are no longer [25].
handwritten [6]. The only study we found that reviews the literature about
Test case prioritization is a method to prioritize and schedule the benefits and the limitations of Automated Software Testing
test cases. In this technique, test cases are run in the order of is presented by Mantyla et al, [3] (2012). Their review, which
priority to minimize time, cost, and effort during the software included 25 works, tries to close the gap by investigating
testing phase. Every organization has its own methods to academic and practitioner views on software testing regarding
prioritize test cases. The automotive safety standard ISO26262 the benefits and the limits of test automation. They found that
requires extensive testing with numerous test cases. To achieve while benefits often come from stronger sources of evidence
a high productivity, the availability of quality assurance (experiments and case studies), limitations are more frequently
systems must be high [7]. reported in experience reports. Second, they conducted a
Herein we use a systematic literature review to evaluate survey of the practitioners’ view. The results showed that the
relevant publications on prioritization in the automotive main benefits of test automation are reusability, repeatability,
industry. A systematic review aims to assess scientific papers and reduced burden in test executions. Of the respondents,
Copyright © 2018 for this paper by its authors. 52
6th International Workshop on Quantitative Approaches to Software Quality (QuASoQ 2018)
45% agreed that the available tools are a poor fit for their needs Inclusion Removal
Initial Impurity Merge and
and 80% disagreed with the vision that automated testing duplicate and during
Search Removal Exclusion data
would fully replace manual testing. removal
criteria extraction
53 11
III. M ETHODOLOGY IEEE
We started the systematic literature review by specifying
our scope and searching only documents in the domain
145 15 48 29 25
of automotive software testing that discuss issues related ACM
to prioritization in the field of testing. Topics that focus TOTAL
only on software testing without prioritizing the test cases
are excluded. In this research, we followed the guidelines 147 30
Scopus
suggested in papers [1], [2] . This method is divided into three
steps.
A. Research Questions Fig. 1. Paper selection process
This study strives to answer the following research
questions:
RQ1. What are the publication trends of prioritization in considered the object of our research (i.e., Prioritization in
Automotive Software Testing? Automotive Software Testing).
This research question should characterize the interest and
ongoing research on this topic. Additionally, it will identify 1) Initial Search: We performed a search in three of the
relevant venues where results are being published and the largest and most complete scientific databases and indexing
contributions over time. systems in software engineering: ACM Digital Library, IEEE
RQ2. What are the methods used for prioritization in Xplore, and Scopus. We searched these databases using a
Automotive Software Testing? search string that included the important keywords in our
This research question should elucidate the different methods four research questions. Further, we augmented the keywords
used for prioritization in Automotive Software Testing. The with their synonyms, producing the following search string:
goal here is to determine the main methods and tools used by
researchers. ((”automobile” OR ”automotive” OR ”car”)
RQ3. How are the studies distributed based on a quality AND
evaluation of prioritization in Automotive Software Testing? (”software” OR ”program” OR ”code”)
This research question should reveal the quality distribution AND
of the selected primary studies and evaluate them accordingly. (”prioritization” OR ”priority” OR ”case selection”)
RQ4. How does existing research on prioritization help with AND
the optimization of Automotive Software Testing? (test*))
This research question should classify existing and future
research on prioritization in Automotive Software Testing and For consistency, we executed the query on titles, abstracts,
assess current research gaps. This is the most important and and keywords of papers in all the data sources at any time
challenging question as it aims to compile problems that have and any subject area.
yet to solved.
B. Search and Selection Process 2) Impurity Removal: Due to the nature of the involved
data sources, the search results included some elements
The search and selection process is a multi-stage process
that were clearly not research papers such as abstracts,
(Fig. 1). This multi-stage process allows us to fully control the
international standards, textbooks, etc. In this stage, we
number and characteristics of the studies that are considered
manually removed these results.
during various stages.
As mentioned in [1] and [2], we used three of the
largest scientific databases and indexing systems in software 3) Merge and Duplicate Removal: Here we combined all
engineering: ACM Digital Library, IEEE Xplore, and Scopus. studies into a single dataset. Duplicated entries were matched
These were selected because they are common, effective by title, authors, year, and venue of publication.
in systematic literature reviews in software engineering,
and capable of exporting the search results. Further, these 4) Inclusion and Exclusion Criteria: We considered all
databases provide mechanisms to perform keyword searches. the selected studies and filtered them according to a set of
We did not specify a fixed time frame when conducting the well-defined selection criteria. The inclusion and exclusion
search. To cover as many significant studies as possible, criteria of our study are:
the systematic literature search query was very generic and Inclusion criteria:
Copyright © 2018 for this paper by its authors. 53
6th International Workshop on Quantitative Approaches to Software Quality (QuASoQ 2018)
• Studies focusing on software testing specific to the
automotive industry.
• Studies providing a solution for prioritizing Automotive
Software Testing.
• Studies in the field of software engineering.
• Studies written in English.
Exclusion criteria:
• Studies that focus on the automotive industry, but do not
explicitly deal with software testing.
• Studies where software testing is only used as an
example. Fig. 2. Primary studies distributed by type of publication over the years
• Studies not available as full-text.
• Studies not presented in English. TABLE I
• Studies that are duplicates of other studies. APPLIED RESEARCH STRATERGIES
Res. strategies #Studies Studies
5) Removal during Data Extraction: When reviewing the Solution Proposal 20 P4, P5, P6, P7, P8, P9, P10,
primary studies in detail to extract information, all the authors P11, P12, P16, P17, P18, P19,
P20, P21, P22, P23, P24, P26,
agreed that four studies were semantically beyond the scope P27
of this research. Consequently, they were excluded. Evaluation Research 15 P5, P8, P10, P11, P13, P15,
P16, P17, P18, P19, P20, P21,
C. Data extraction P22, P26, P28
Validation Research 14 P5, P8, P9, P10, P11, P12,
Relevant information was extracted to answer the research P16, P18, P20, P21, P22, P24,
questions from the primary studies. We used data extraction P26, P28
forms to make sure that this task was carried out in an Opinion Paper 4 P13, P14, P25, P27
Survey Paper 1 P25
accurate and consistent manner. The data was collected and
stored in a spreadsheet using MS Excel to list the relevant
information of each paper. This technique helps extract and is maturing. A small but constant number of publications were
view data in a tabular form. published until 2014. However, prioritization has become an
important and eye-catching aspect in terms of research since
The following information was collected from each paper: 2014. The interest in prioritization of automotive software
testing has rapidly increased in the last few years.
• Publication title
Studies published before 2015 refer to slightly different
• Publication year
perspectives on prioritization than more recent papers. The
• Publication venue
number of papers has drastically increased since 2014. . [25]
• Problems faced by the authors
[11] [4] [6] [10] used model-based testing to improve the
• Testing method used
prioritization by increasing the effectiveness. On the other
• Limitations in field
hand, [6] [10] showed potential improvements and proposed
• Detail of the proposed solution
new model-based methods.
• Results obtained
Many researchers provided solution proposals (20/25)
• Rating of quality issues
and evaluation research (15/25) (Table I), indicating that
• Verification and validation
today’s researchers focus on industry and practitioner-oriented
• Future work suggested by the authors
studies (e.g., industrial case study, action research). Another
• Conclusions
common research strategy is validation research (14/25),
• Answers to research questions
highlighting the fact that there is some level of evidence
IV. A NALYSIS R ESULTS (e.g., simulations, experiments, prototypes, etc.) supporting
This section presents the analysis and each sub-section the proposed solutions. However, Table I also shows that few
answers the previously presented research questions. We used studies employ surveys (1/25), suggesting that future studies
the R software environment and Microsoft Excel to perform should fill this gap.
basic statistical operations and draw charts.
B. Methods Used (RQ2)
A. Publication Trends (RQ1) Due to the requirement of connected services for vehicles,
Figure 2 presents the distribution of publications over time. an interesting method is model-based testing. Figure 3 depicts
The most common publication types are conference papers a histogram of the distribution of the most common techniques
(17/25) followed by workshop papers (5/25), journals (2/25), in the literature. The most common testing methods are
and symposiums (1/25). The high number of conference papers model-based testing (7/25), regression testing (6/25), and black
may indicate that prioritization of automotive software testing box testing (5/25), followed by hardware in the loop testing
Copyright © 2018 for this paper by its authors. 54
6th International Workshop on Quantitative Approaches to Software Quality (QuASoQ 2018)
TABLE III
RATING OF REVIEWED ARTICLES
Reference QI1 QI2 QI3 QI4 Total
4 Y Y Y Y 4
5 Y P P Y 3
6 Y P Y P 3
7 Y Y P Y 3.5
8 Y P P Y 3
9 Y Y P P 3
10 Y Y N P 2.5
11 P Y P Y 3
12 Y P Y P 3
13 Y Y Y Y 4
14 Y P Y Y 3.5
15 Y P Y P 3
16 Y Y P Y 3.5
17 Y Y Y Y 4
18 P P Y Y 3
Fig. 3. Testing Methods
19 Y Y Y Y 4
20 Y N P P 2
21 Y P P Y 3
(4/25), software testing (4/25), functional testing (3/25), and 22 Y Y P Y 3.5
other. 23 P N Y N 1.5
Approaches that use model-based testing are found in [4] [6] 24 Y Y Y Y 4
[9] [10] [11] [25] [26]. Techniques listed as “OTHER” refer to 25 Y Y Y P 3.5
the use of integration testing [10] [24], software product line 26 Y Y P Y 3.5
testing [4] [11], system testing [5] [8], abstract testing [17], 27 Y Y Y N 3
combinational testing [18], cyber-physical system testing [13], 28 Y P Y P 3
end-of-line testing [7], simulation testing [12], stateflow testing Average 3.2
[15], and statistical testing [20].
Different methods or combination of methods are used in than 90% of the functionality of automotive systems and
multiple studies (Table II). These papers frequently target up to 80% of the automotive software can be automatically
model-based testing (7/25), regression testing (6/25), and black generated from models [6]. Additionally, model-based testing
box testing (5/25). Model-based development is an efficient, is a common solution to test embedded systems in automotive
reliable, and cost-effective paradigm to design and implement engineering. Regression testing is undertaken every time
complex embedded systems. The software determines more a model is updated to verify quality assurance, which is
time-consuming as it reruns an entire test suite after every
minor change. Test case selection for regression testing after
TABLE II new releases is an important task to maintain the availability
APPLIED TESTING METHODS
[7]. Typically studies focus on black-box testing scenarios
Testing Methods #Studies Studies because the source code is often unavailable in the automotive
Model-Based Testing 7 P4, P6, P9, P10,
P11, P25, P26
domain such as an OEM-supplier scenario [19]. Hardware in
Regression Testing 6 P5, P6, P7, P9, the loop testing (4/25) and software testing (4/25) are the next
P11, P19 most used methods. The most common method of testing
Black-Box Testing 5 P5, P8, P9, P16,
P19
the software and the Electronic Control Units (ECU) is the
Hardware in the Loop Testing Testing 4 P8, P16, P27, use of Hardware-In-the-Loop (HIL) simulation [27]. Software
P28 testing presents an approach to automatically generate test
Software Testing 4 P14, P21, P23,
P27
cases for a software product. Functional testing (3/25) strives
Functional Testing 3 P16, P22, P28 to demonstrate the correct implementation of functional
Integration Testing 2 P10, P24 requirements and is one of the most important approaches to
Software Product Line Testing 2 P4, P11
Testing
gain confidence in the correct functional behavior of a system
System Testing 2 P5, P8 [28]. Integration testing (2/25), software product line testing
Abstract Testing 1 P17 (2/25), and system testing (2/25) are used as the time donation
Combinational Testing 1 P18
Cyber-Physical Systems Testing 1 P13
when the testing process is limited. A negative highlight of
End-of-Line Testing 1 P7 this systematic review is the fact that only one paper directly
Simulation Testing 1 P12 employs a simulation testing method [12]. If a simulation
Stateflow Testing 1 P15
Statistical Testing 1 P20
environment can imitate the key criteria of the real-world
Copyright © 2018 for this paper by its authors. 55
6th International Workshop on Quantitative Approaches to Software Quality (QuASoQ 2018)
TABLE IV
VALIDATION ,T ECHNIQUES /T OOLS , G APS , AND M AIN OUTCOMES OF STUDIES
Validation No. of Study Techniques/Tools Gaps Main Outcome(s)
Papers
Industrial 14 P5 Test case selection based a on Find better heuristic clustering approaches Regression effort can be minimized
Case (56%) Stochastic model
Study
P8 Test case selection based on a Change deployed libraries 82.3% reduction in tested functions
component and communication
model
P10 Taster tool, Proposed new Focus on the optimization of the New framework for the MBT
framework for MBT classification structure used within
the priority assignment procedure
P11 Dissimilarity-based TCP Investigate the fault detection capabilities Dissimilarity-based TCP issues are solved
of the approach
P15 Test selection algorithms Develop optimal guidelines to divide test Output-based algorithms consistently
oracle budget across the output-based outperform coverage-based algorithms in
selection algorithms revealing faults
P16 Evolutionary Testing Investigate how to configure the Solution provide complements systematic
Framework, modularHiL, evolutionary testing system to reduce testing in that it generates test cases
MESSINA the number of pre-tests for situations that would otherwise be
unforeseen by testers
P17 Migration from traditional to Extend the formalism to handle Abstract testing is comparable in
abstract testing non-functional requirements effectiveness
P18 Equivalence Class Partitioning Investigatee efficient test case generation Efficient reduction in the final effective
(ECP), Boundary Value and discover more feasible tools and number of test cases by 42 (88% reduction)
Analysis (BVA), Choice empirical studies to work
Relationship Framework (CRF)
P19 Test case combination Six approaches are presented to improve Machine learning approach in black-box
test efficiency testing
P20 Combination of test models in *NO GAPS MENTIONED* Higher coverage is achieved compared with
MATLAB/Simulink manually created test cases
P21 Automatically generate test Refine the functional coverage model Improvement actions are identified for test
cases case generation
P22 Proposed unified model Implement a large survey on software More than 90% of the requirements are
specifications in Johnson Controls represented by the model
company
P26 End-to-end test framework Investigate the nature of the test model and Automation of executable test script
the relevance to the generated test cases generation
P28 Search-based testing principles Further investigate safety requirements Promising approach to ensure safety
requirements
Technique 6 P4 Similarity-Based Product Include more Solution space information Improvement in the effectiveness of SPL
Comparison(24%) Prioritization w.r.t Deltas (Such as Source code) testing
P6 New model-based method for Implement performance evaluation with a Future regression testing can be sped up
test case prioritization large-scale case study
P7 Combined fault diagnosis and Evaluate using an end-of-line test system System can find test cases to increase the
test case selection at a real assembly line test coverage
P13 Weight-based search algorithms Study weight tuning, different fitness Results suggest that all the search
functions, and cost-effectiveness measures algorithms outperform Random Search
P23 Model slicing technique for *NO GAPS MENTIONED* Complexity of Simulink models can be
optimal test case generation reduced
P24 OUTFIT tool Evaluate it with other domains such as Potential defects can be effectively
medical and avionic systems identified
Statistical 1 P9 Dependences between the Study exhaustiveness of the path search Reduction in test-cases for regression
Evaluation (4%) components of embedded and correctness of path search testing
systems
Simulation 1 P12 Parallelly execute loosely Fully automate the process of segmentation Reduce the simulation testing time for both
(4%) coupled segments and instrumentation successful and failed runs
Others 3 P14 Three approaches to Focus on MC/DC test case generation from Approaches can be combined to support
(12%) automatically generate MC/DC formal specifications different kinds of decisions
test cases
P25 Survey paper Publish more detailed description of the Improvements that are possible with
applied evaluation approach MBAT technologies
P27 Study on requirements *NO GAPS MENTIONED* Synect provides easy test requirement
specifications and management of the test
results
Copyright © 2018 for this paper by its authors. 56
6th International Workshop on Quantitative Approaches to Software Quality (QuASoQ 2018)
environment, it should be used to provide early feedback TABLE VI
on the vehicle’s design. VALIDATION
Techniques #Studies Studies
C. Quality Evaluation (RQ3) Industrial Case Study 14 P5, P8, P10, P11, P15,
According to [29], a ranking was created to rate papers P16, P17, P18, P19, P20,
P21, P22, P26, P28
based on the relevance to the topic and the quality of the Technique Comparison 6 P4, P6, P7, P13, P23, P24
paper. Quality Issues (QI) are: Statistical Evaluation 1 P9
Simulation 1 P12
• QI1 Is the paper’s goal clear?
Others 3 P14, P25, P27
• QI2 Does the assessment approach match the goals?
• QI3 Can the method be replicated?
• QI4 Are results shown in detail?
studies are focus on industrial case studies (n = 14) and
technique comparisons (n = 6).
For each quality issue, articles were rated as: Yes (Y) when
Table VI lists the techniques used to validate the selected
the issue is addressed in the text, Partial (P) when the issue is
studies. The most common are industrial case studies
partially addressed in the text, and No (N) when the issue is
(14/25) followed by technique comparisons (6/25), statistical
not addressed in the text. These ratings were scored as Yes = 1
evaluations (1/25), simulations (1/25), and others (3/25).
point, Partial = 0.5, and No = 0. Table III shows the papers that
Technique comparisons include studies that propose and then
were analyzed in this SLR and their respective scores based
compare a new method to an old one. Others include three
on the Quality Issues discussed above.
studies, which [14] [27] talk about the advantages, limitations,
D. Existing Research (RQ4) and requirements of different approaches. [25] is a survey
paper from 13 industry case studies.
Here we discuss the recurring problems that are targeted
by primary studies, which methods described in RQ2 are V. T HREATS TO VALIDITY
validated, and the gaps mentioned in the research.
Recurring problems are time consumption (15/25), cost The analysis was conducted by a single person. Thus, one
(13/25), and complexity (14/25) followed by test case selection threat is that some information may be omitted. Moreover, the
(3/25) and quality improvement (3/25) (Table V). Because analysis is limited by the analytical skills of that single person.
the testing time is expensive, it should be reduced without
VI. C ONCLUSION
an uncontrolled reduction of reliability. The entire test suite
must be rerun each time the system is updated or modified. This paper overviews the Prioritization in Automotive
Consequently, each modification makes the testing process Software Testing. The results should help companies that
more time-consuming. Automotive systems are becoming are planning to incorporate prioritization into their strategies.
more complex due to a higher rate of integration and shared Researchers can also benefit because this study depicts the
usage. The high complexity results in numerous interfaces, limitations and gaps in current research. Additionally, the
and many signals must be processed inside the system [9]. trends in other embedded and non-embedded domains must
Testing activities can account for a considerable part of the be investigated as this should provide a more detailed picture
software production costs. However, only two studies discuss and lessons learned regarding prioritization in Automotive
improving efficiency [18] and safety [26] which is a negative Software Testing. Future work includes (i) a qualitative study
highlight. to better understand test execution, test case generation,
Table IV presents the studies within each category, test case selection, and test analysis and (ii) addressing the
Techniques/Tools, gaps, and the main study outcomes. Most identified research gaps.
R EFERENCES
TABLE V [1] B. Kitchenham and S. Charters, “Guidelines for performing systematic
TARGET PROBLEMS literature reviews in software engineering,” 2007.
[2] B. Kitchenham and P. Brereton, “A systematic review of systematic
Problems #Studies Studies
review process research in software engineering,” Inf. Softw. Technol.,
Time Consumption 15 P5, P6, P8, P9, P10, P12, vol. 55, no. 12, pp. 2049–2075, Dec. 2013. [Online]. Available:
P13, P14, P15, P16, P17, http://dx.doi.org/10.1016/j.infsof.2013.07.010
P19, P20, P24, P27 [3] D. M. Rafi, K. R. K. Moses, K. Petersen, and M. V. Mäntylä, “Benefits
Complexity 14 P5, P6, P7, P8, P9, P10, and limitations of automated software testing: Systematic literature
P13, P15, P16, P17, P19, review and practitioner survey,” in 2012 7th International Workshop on
P20, P24, P27 Automation of Software Test (AST), June 2012, pp. 36–42.
Cost 13 P5, P6, P8, P9, P10, P13,
[4] M. Al-Hajjaji, S. Lity, R. Lachmann, T. Thüm, I. Schaefer, and G. Saake,
P15, P16, P17, P19, P20,
“Delta-oriented product prioritization for similarity-based product-line
P24, P27
testing,” in 2017 IEEE/ACM 2nd International Workshop on Variability
Test Case Selection 3 P7, P11, P28
and Complexity in Software Design (VACE), May 2017, pp. 34–40.
Quality Improvement 3 P22,P25,P26
[5] I. Alagöz, T. Herpel, and R. German, “A selection method for black
Test Generation 2 P21,P23
box regression testing with a statistically defined quality level,” in 2017
Problem Space Information 1 P4
IEEE International Conference on Software Testing, Verification and
Improving Efficiency 1 P18
Validation (ICST), March 2017, pp. 114–125.
Safety 1 P26
Copyright © 2018 for this paper by its authors. 57
6th International Workshop on Quantitative Approaches to Software Quality (QuASoQ 2018)
[6] A. Morozov, K. Ding, T. Chen, and K. Janschek, “Test suite prioritization [21] R. Awedikian and B. Yannou, “Design of a validation test process of
for efficient regression testing of model-based automotive software,” an automotive software,” International Journal on Interactive Design
in 2017 International Conference on Software Analysis, Testing and and Manufacturing, vol. 4, no. 4, pp. 259–268, 2010, cited By :2.
Evolution (SATE), Nov 2017, pp. 20–29. [Online]. Available: www.scopus.com
[7] S. Abele and M. Weyrich, “A combined fault diagnosis and test case [22] R. Awedikian, B. Yannou, P. Lebreton, L. Bouclier, and M. Mekhilef,
selection assistant for automotive end-of-line test systems,” in 2016 IEEE “A simulated model of software specifications for automating functional
14th International Conference on Industrial Informatics (INDIN), July tests design,” in Proceedings DESIGN 2008, the 10th International
2016, pp. 1072–1077. Design Conference, 2008, pp. 561–570, cited By :1. [Online]. Available:
[8] S. Vöst and S. Wagner, “Trace-based test selection to support continuous www.scopus.com
integration in the automotive industry,” in 2016 IEEE/ACM International [23] Z. Jiang, X. Wu, Z. Dong, and M. Mu, “Optimal test case generation
Workshop on Continuous Software Evolution and Delivery (CSED), May for simulink models using slicing,” in Proceedings - 2017 IEEE
2016, pp. 34–40. International Conference on Software Quality, Reliability and Security
[9] P. Caliebe, T. Herpel, and R. German, “Dependency-based test case Companion, QRS-C 2017, 2017, pp. 363–369. [Online]. Available:
selection and prioritization in embedded systems,” in 2012 IEEE www.scopus.com
Fifth International Conference on Software Testing, Verification and [24] D. Holling, A. Hofbauer, A. Pretschner, and M. Gemmar, “Profiting
Validation, April 2012, pp. 731–735. from unit tests for integration testing,” in Proceedings - 2016
[10] L. Krejčı́ and J. Novák, “Model-based testing of automotive distributed IEEE International Conference on Software Testing, Verification and
systems with automated prioritization,” in 2017 9th IEEE International Validation, ICST 2016, 2016, pp. 353–363, cited By :1. [Online].
Conference on Intelligent Data Acquisition and Advanced Computing Available: www.scopus.com
Systems: Technology and Applications (IDAACS), vol. 2, Sept 2017, pp. [25] M. Klas, T. Bauer, A. Dereani, T. Soderqvist, and P. Helle,
668–673. “A large-scale technology evaluation study: Effects of model-based
[11] R. Lachmann, S. Lity, M. Al-Hajjaji, F. Fürchtegott, and I. Schaefer, analysis and testing,” in Proceedings - International Conference on
“Fine-grained test case prioritization for integration testing of Software Engineering, vol. 2, 2015, pp. 119–128, cited By :4. [Online].
delta-oriented software product lines,” in Proceedings of the 7th Available: www.scopus.com
International Workshop on Feature-Oriented Software Development, [26] J. Lasalle, F. Peureux, and J. Guillet, “Automatic test concretization to
ser. FOSD 2016. New York, NY, USA: ACM, 2016, pp. 1–10. supply end-to-end mbt for automotive mechatronic systems,” in 2011
[Online]. Available: http://doi.acm.org/10.1145/3001867.3001868 International Workshop on End-to-End Test Script Engineering, ETSE
[12] M. A. Al Mamun and J. Hansson, “Reducing simulation testing 2011 - Proceedings, 2011, pp. 16–23, cited By :4. [Online]. Available:
time by parallel execution of loosely coupled segments of a test www.scopus.com
scenario,” in Proceedings of International Workshop on Engineering [27] A. Bansal, M. Muli, and K. Patil, “Taming complexity while gaining
Simulations for Cyber-Physical Systems, ser. ES4CPS ’14. New efficiency: Requirements for the next generation of test automation
York, NY, USA: ACM, 2007, pp. 33:33–33:37. [Online]. Available: tools,” in AUTOTESTCON (Proceedings), 2013, pp. 123–128, cited By
http://doi.acm.org/10.1145/2559627.2559635 :1. [Online]. Available: www.scopus.com
[13] A. Arrieta, S. Wang, G. Sagardui, and L. Etxeberria, “Test [28] F. Lindlar and A. Windisch, “A search-based approach to functional
case prioritization of configurable cyber-physical systems with hardware-in-the-loop testing,” in Proceedings - 2nd International
weight-based search algorithms,” in Proceedings of the Genetic and Symposium on Search Based Software Engineering, SSBSE 2010, 2010,
Evolutionary Computation Conference 2016, ser. GECCO ’16. New pp. 111–119, cited By :7. [Online]. Available: www.scopus.com
York, NY, USA: ACM, 2016, pp. 1053–1060. [Online]. Available: [29] U. Kanewala and J. M. Bieman, “Testing scientific software:
http://doi.acm.org/10.1145/2908812.2908871 A systematic literature review,” Inf. Softw. Technol., vol. 56,
[14] S. Kangoye, A. Todoskoff, M. BARREAU, and P. GERMANICUS, no. 10, pp. 1219–1232, Oct. 2014. [Online]. Available:
“Mc/dc test case generation approaches for decisions,” in Proceedings of http://dx.doi.org/10.1016/j.infsof.2014.05.006
the ASWEC 2015 24th Australasian Software Engineering Conference,
ser. ASWEC ’ 15 Vol. II. New York, NY, USA: ACM, 2015, pp. 74–80.
[Online]. Available: http://doi.acm.org/10.1145/2811681.2811696
[15] R. Matinnejad, S. Nejati, L. C. Briand, and T. Bruckmann,
“Effective test suites for mixed discrete-continuous stateflow
controllers,” in Proceedings of the 2015 10th Joint Meeting on
Foundations of Software Engineering, ser. ESEC/FSE 2015. New
York, NY, USA: ACM, 2015, pp. 84–95. [Online]. Available:
http://doi.acm.org/10.1145/2786805.2786818
[16] P. M. Kruse, J. Wegener, and S. Wappler, “A highly configurable
test system for evolutionary black-box testing of embedded
systems,” in Proceedings of the 11th Annual Conference on
Genetic and Evolutionary Computation, ser. GECCO ’09. New
York, NY, USA: ACM, 2009, pp. 1545–1552. [Online]. Available:
http://doi.acm.org/10.1145/1569901.1570108
[17] F. Merz, C. Sinz, H. Post, T. Gorges, and T. Kropf, “Bridging the
gap between test cases and requirements by abstract testing,” Innov.
Syst. Softw. Eng., vol. 11, no. 4, pp. 233–242, Dec. 2015. [Online].
Available: http://dx.doi.org/10.1007/s11334-015-0245-7
[18] J. S. Eo, H. R. Choi, R. Gao, S. y. Lee, and W. E. Wong, “Case study
of requirements-based test case generation on an automotive domain,”
in 2015 IEEE International Conference on Software Quality, Reliability
and Security - Companion, Aug 2015, pp. 210–215.
[19] R. Lachmann and I. Schaefer, “Towards efficient and effective testing
in automotive software development,” in Lecture Notes in Informatics
(LNI), Proceedings - Series of the Gesellschaft fur Informatik (GI),
vol. P-232, 2014, pp. 2181–2192, cited By :2. [Online]. Available:
www.scopus.com
[20] S. Siegl, K. S. Hielscher, and R. German, “Modeling and statistical
testing of real time embedded automotive systems by combination
of test models and reference models in matlab/simulink,” in 2011
21st International Conference on Systems Engineering, Aug 2011, pp.
180–185.
Copyright © 2018 for this paper by its authors. 58