6th International Workshop on Quantitative Approaches to Software Quality (QuASoQ 2018) Prioritization in Automotive Software Testing: Systematic Literature Review Ankush Dadwal, Hironori Washizaki, Yoshiaki Fukazawa Takahiro Iida, Masashi Mizoguchi, Kentaro Yoshimura Department of Computer Science and Engineering Control Platform Research Department Waseda University Center for Technology Innovation - Controls Tokyo, Japan Hitachi, Ltd. Research & Development Group ankush.dadwal@toki.waseda.jp takahiro.iida.ac@hitachi.com Abstract—Automotive Software Testing is a vital part of the in order to group concepts around a topic. Through analysis automotive systems development process. Not identifying the criteria, it allows the quality of research to be evaluated. Herein critical safety issues and failures of such systems can have the system review aims to identify common techniques in serious or even fatal consequences. As the number of embedded systems and technologies increases, testing all components automotive testing and to define new challenges. becomes more challenging. Although testing is expensive, it The paper is structured as follows. Section II describes is important to reduce bugs in an early stage to maintain related works. The systematic literature review approach is safety and to avoid recalls. Hence, the testing time should detailed in Section III. Section IV presents the results obtained be reduced without impacting the reliability. Several studies from the systematic review. Section V addresses potential and surveys have prioritized Automotive Software Testing to increase its effectiveness. The main goals of this study are to threats to validity. Finally, Section VI lists the conclusions identify: (i) the publication trends of prioritization in Automotive and the definitions for future work. Software Testing, (ii) which methods are used to prioritize Automotive Software Testing, (iii) the distribution of studies II. R ELATED W ORKS based on the quality evaluation, and (iv) how existing research Automakers have experienced the impact of the evolution of on prioritization helps optimize Automotive Software Testing. Index Terms—Automotive Software Testing, Prioritizing, technology on automotive testing. Today, testing all systems Systematic Literature Review manually is not only cost-intensive and time-consuming but nearly impossible. Automating the testing phase would I. I NTRODUCTION significantly reduce the cost of software development [14]. Literature about the prioritization efforts in the automobile Currently the automotive industry is undergoing a major industry is scarce. Herein we focus on known techniques and transition. Automakers have been adding new functions and their applicability to the investigated domain. systems to meet the market’s demand for an ever-growing In the past few decades, numerous studies [5], [6], [7], amount of software-intensive functions. However, these new [8], [9], [10], [13], [15], [16], [17], [19], [20], [24], [27] functions and systems have some negative aspects. One have demonstrated that vehicles are becoming increasingly is that automakers must enhance their testing techniques more complex and more connected. For example, an empirical because vehicle complexity is increasing. Testing typically study, which aimed to investigate the potential regarding consumes more than half of all development costs [4]. While quality improvements and cost savings, employed data from testing a single software system is difficult, testing without 13 industry case studies as part of a three-year large-scale prioritization is even more challenging due to the exponential research project. This study identified major goals and number of products and the number of features. Today, strategies associated with (integrated) model-based analysis software determines more than 90% of the functionality in and testing as well as evaluated the improvements achieved automotive systems and software components are no longer [25]. handwritten [6]. The only study we found that reviews the literature about Test case prioritization is a method to prioritize and schedule the benefits and the limitations of Automated Software Testing test cases. In this technique, test cases are run in the order of is presented by Mantyla et al, [3] (2012). Their review, which priority to minimize time, cost, and effort during the software included 25 works, tries to close the gap by investigating testing phase. Every organization has its own methods to academic and practitioner views on software testing regarding prioritize test cases. The automotive safety standard ISO26262 the benefits and the limits of test automation. They found that requires extensive testing with numerous test cases. To achieve while benefits often come from stronger sources of evidence a high productivity, the availability of quality assurance (experiments and case studies), limitations are more frequently systems must be high [7]. reported in experience reports. Second, they conducted a Herein we use a systematic literature review to evaluate survey of the practitioners’ view. The results showed that the relevant publications on prioritization in the automotive main benefits of test automation are reusability, repeatability, industry. A systematic review aims to assess scientific papers and reduced burden in test executions. Of the respondents, Copyright © 2018 for this paper by its authors. 52 6th International Workshop on Quantitative Approaches to Software Quality (QuASoQ 2018) 45% agreed that the available tools are a poor fit for their needs Inclusion Removal Initial Impurity Merge and and 80% disagreed with the vision that automated testing duplicate and during Search Removal Exclusion data would fully replace manual testing. removal criteria extraction 53 11 III. M ETHODOLOGY IEEE We started the systematic literature review by specifying our scope and searching only documents in the domain 145 15 48 29 25 of automotive software testing that discuss issues related ACM to prioritization in the field of testing. Topics that focus TOTAL only on software testing without prioritizing the test cases are excluded. In this research, we followed the guidelines 147 30 Scopus suggested in papers [1], [2] . This method is divided into three steps. A. Research Questions Fig. 1. Paper selection process This study strives to answer the following research questions: RQ1. What are the publication trends of prioritization in considered the object of our research (i.e., Prioritization in Automotive Software Testing? Automotive Software Testing). This research question should characterize the interest and ongoing research on this topic. Additionally, it will identify 1) Initial Search: We performed a search in three of the relevant venues where results are being published and the largest and most complete scientific databases and indexing contributions over time. systems in software engineering: ACM Digital Library, IEEE RQ2. What are the methods used for prioritization in Xplore, and Scopus. We searched these databases using a Automotive Software Testing? search string that included the important keywords in our This research question should elucidate the different methods four research questions. Further, we augmented the keywords used for prioritization in Automotive Software Testing. The with their synonyms, producing the following search string: goal here is to determine the main methods and tools used by researchers. ((”automobile” OR ”automotive” OR ”car”) RQ3. How are the studies distributed based on a quality AND evaluation of prioritization in Automotive Software Testing? (”software” OR ”program” OR ”code”) This research question should reveal the quality distribution AND of the selected primary studies and evaluate them accordingly. (”prioritization” OR ”priority” OR ”case selection”) RQ4. How does existing research on prioritization help with AND the optimization of Automotive Software Testing? (test*)) This research question should classify existing and future research on prioritization in Automotive Software Testing and For consistency, we executed the query on titles, abstracts, assess current research gaps. This is the most important and and keywords of papers in all the data sources at any time challenging question as it aims to compile problems that have and any subject area. yet to solved. B. Search and Selection Process 2) Impurity Removal: Due to the nature of the involved data sources, the search results included some elements The search and selection process is a multi-stage process that were clearly not research papers such as abstracts, (Fig. 1). This multi-stage process allows us to fully control the international standards, textbooks, etc. In this stage, we number and characteristics of the studies that are considered manually removed these results. during various stages. As mentioned in [1] and [2], we used three of the largest scientific databases and indexing systems in software 3) Merge and Duplicate Removal: Here we combined all engineering: ACM Digital Library, IEEE Xplore, and Scopus. studies into a single dataset. Duplicated entries were matched These were selected because they are common, effective by title, authors, year, and venue of publication. in systematic literature reviews in software engineering, and capable of exporting the search results. Further, these 4) Inclusion and Exclusion Criteria: We considered all databases provide mechanisms to perform keyword searches. the selected studies and filtered them according to a set of We did not specify a fixed time frame when conducting the well-defined selection criteria. The inclusion and exclusion search. To cover as many significant studies as possible, criteria of our study are: the systematic literature search query was very generic and Inclusion criteria: Copyright © 2018 for this paper by its authors. 53 6th International Workshop on Quantitative Approaches to Software Quality (QuASoQ 2018) • Studies focusing on software testing specific to the automotive industry. • Studies providing a solution for prioritizing Automotive Software Testing. • Studies in the field of software engineering. • Studies written in English. Exclusion criteria: • Studies that focus on the automotive industry, but do not explicitly deal with software testing. • Studies where software testing is only used as an example. Fig. 2. Primary studies distributed by type of publication over the years • Studies not available as full-text. • Studies not presented in English. TABLE I • Studies that are duplicates of other studies. APPLIED RESEARCH STRATERGIES Res. strategies #Studies Studies 5) Removal during Data Extraction: When reviewing the Solution Proposal 20 P4, P5, P6, P7, P8, P9, P10, primary studies in detail to extract information, all the authors P11, P12, P16, P17, P18, P19, P20, P21, P22, P23, P24, P26, agreed that four studies were semantically beyond the scope P27 of this research. Consequently, they were excluded. Evaluation Research 15 P5, P8, P10, P11, P13, P15, P16, P17, P18, P19, P20, P21, C. Data extraction P22, P26, P28 Validation Research 14 P5, P8, P9, P10, P11, P12, Relevant information was extracted to answer the research P16, P18, P20, P21, P22, P24, questions from the primary studies. We used data extraction P26, P28 forms to make sure that this task was carried out in an Opinion Paper 4 P13, P14, P25, P27 Survey Paper 1 P25 accurate and consistent manner. The data was collected and stored in a spreadsheet using MS Excel to list the relevant information of each paper. This technique helps extract and is maturing. A small but constant number of publications were view data in a tabular form. published until 2014. However, prioritization has become an important and eye-catching aspect in terms of research since The following information was collected from each paper: 2014. The interest in prioritization of automotive software testing has rapidly increased in the last few years. • Publication title Studies published before 2015 refer to slightly different • Publication year perspectives on prioritization than more recent papers. The • Publication venue number of papers has drastically increased since 2014. . [25] • Problems faced by the authors [11] [4] [6] [10] used model-based testing to improve the • Testing method used prioritization by increasing the effectiveness. On the other • Limitations in field hand, [6] [10] showed potential improvements and proposed • Detail of the proposed solution new model-based methods. • Results obtained Many researchers provided solution proposals (20/25) • Rating of quality issues and evaluation research (15/25) (Table I), indicating that • Verification and validation today’s researchers focus on industry and practitioner-oriented • Future work suggested by the authors studies (e.g., industrial case study, action research). Another • Conclusions common research strategy is validation research (14/25), • Answers to research questions highlighting the fact that there is some level of evidence IV. A NALYSIS R ESULTS (e.g., simulations, experiments, prototypes, etc.) supporting This section presents the analysis and each sub-section the proposed solutions. However, Table I also shows that few answers the previously presented research questions. We used studies employ surveys (1/25), suggesting that future studies the R software environment and Microsoft Excel to perform should fill this gap. basic statistical operations and draw charts. B. Methods Used (RQ2) A. Publication Trends (RQ1) Due to the requirement of connected services for vehicles, Figure 2 presents the distribution of publications over time. an interesting method is model-based testing. Figure 3 depicts The most common publication types are conference papers a histogram of the distribution of the most common techniques (17/25) followed by workshop papers (5/25), journals (2/25), in the literature. The most common testing methods are and symposiums (1/25). The high number of conference papers model-based testing (7/25), regression testing (6/25), and black may indicate that prioritization of automotive software testing box testing (5/25), followed by hardware in the loop testing Copyright © 2018 for this paper by its authors. 54 6th International Workshop on Quantitative Approaches to Software Quality (QuASoQ 2018) TABLE III RATING OF REVIEWED ARTICLES Reference QI1 QI2 QI3 QI4 Total 4 Y Y Y Y 4 5 Y P P Y 3 6 Y P Y P 3 7 Y Y P Y 3.5 8 Y P P Y 3 9 Y Y P P 3 10 Y Y N P 2.5 11 P Y P Y 3 12 Y P Y P 3 13 Y Y Y Y 4 14 Y P Y Y 3.5 15 Y P Y P 3 16 Y Y P Y 3.5 17 Y Y Y Y 4 18 P P Y Y 3 Fig. 3. Testing Methods 19 Y Y Y Y 4 20 Y N P P 2 21 Y P P Y 3 (4/25), software testing (4/25), functional testing (3/25), and 22 Y Y P Y 3.5 other. 23 P N Y N 1.5 Approaches that use model-based testing are found in [4] [6] 24 Y Y Y Y 4 [9] [10] [11] [25] [26]. Techniques listed as “OTHER” refer to 25 Y Y Y P 3.5 the use of integration testing [10] [24], software product line 26 Y Y P Y 3.5 testing [4] [11], system testing [5] [8], abstract testing [17], 27 Y Y Y N 3 combinational testing [18], cyber-physical system testing [13], 28 Y P Y P 3 end-of-line testing [7], simulation testing [12], stateflow testing Average 3.2 [15], and statistical testing [20]. Different methods or combination of methods are used in than 90% of the functionality of automotive systems and multiple studies (Table II). These papers frequently target up to 80% of the automotive software can be automatically model-based testing (7/25), regression testing (6/25), and black generated from models [6]. Additionally, model-based testing box testing (5/25). Model-based development is an efficient, is a common solution to test embedded systems in automotive reliable, and cost-effective paradigm to design and implement engineering. Regression testing is undertaken every time complex embedded systems. The software determines more a model is updated to verify quality assurance, which is time-consuming as it reruns an entire test suite after every minor change. Test case selection for regression testing after TABLE II new releases is an important task to maintain the availability APPLIED TESTING METHODS [7]. Typically studies focus on black-box testing scenarios Testing Methods #Studies Studies because the source code is often unavailable in the automotive Model-Based Testing 7 P4, P6, P9, P10, P11, P25, P26 domain such as an OEM-supplier scenario [19]. Hardware in Regression Testing 6 P5, P6, P7, P9, the loop testing (4/25) and software testing (4/25) are the next P11, P19 most used methods. The most common method of testing Black-Box Testing 5 P5, P8, P9, P16, P19 the software and the Electronic Control Units (ECU) is the Hardware in the Loop Testing Testing 4 P8, P16, P27, use of Hardware-In-the-Loop (HIL) simulation [27]. Software P28 testing presents an approach to automatically generate test Software Testing 4 P14, P21, P23, P27 cases for a software product. Functional testing (3/25) strives Functional Testing 3 P16, P22, P28 to demonstrate the correct implementation of functional Integration Testing 2 P10, P24 requirements and is one of the most important approaches to Software Product Line Testing 2 P4, P11 Testing gain confidence in the correct functional behavior of a system System Testing 2 P5, P8 [28]. Integration testing (2/25), software product line testing Abstract Testing 1 P17 (2/25), and system testing (2/25) are used as the time donation Combinational Testing 1 P18 Cyber-Physical Systems Testing 1 P13 when the testing process is limited. A negative highlight of End-of-Line Testing 1 P7 this systematic review is the fact that only one paper directly Simulation Testing 1 P12 employs a simulation testing method [12]. If a simulation Stateflow Testing 1 P15 Statistical Testing 1 P20 environment can imitate the key criteria of the real-world Copyright © 2018 for this paper by its authors. 55 6th International Workshop on Quantitative Approaches to Software Quality (QuASoQ 2018) TABLE IV VALIDATION ,T ECHNIQUES /T OOLS , G APS , AND M AIN OUTCOMES OF STUDIES Validation No. of Study Techniques/Tools Gaps Main Outcome(s) Papers Industrial 14 P5 Test case selection based a on Find better heuristic clustering approaches Regression effort can be minimized Case (56%) Stochastic model Study P8 Test case selection based on a Change deployed libraries 82.3% reduction in tested functions component and communication model P10 Taster tool, Proposed new Focus on the optimization of the New framework for the MBT framework for MBT classification structure used within the priority assignment procedure P11 Dissimilarity-based TCP Investigate the fault detection capabilities Dissimilarity-based TCP issues are solved of the approach P15 Test selection algorithms Develop optimal guidelines to divide test Output-based algorithms consistently oracle budget across the output-based outperform coverage-based algorithms in selection algorithms revealing faults P16 Evolutionary Testing Investigate how to configure the Solution provide complements systematic Framework, modularHiL, evolutionary testing system to reduce testing in that it generates test cases MESSINA the number of pre-tests for situations that would otherwise be unforeseen by testers P17 Migration from traditional to Extend the formalism to handle Abstract testing is comparable in abstract testing non-functional requirements effectiveness P18 Equivalence Class Partitioning Investigatee efficient test case generation Efficient reduction in the final effective (ECP), Boundary Value and discover more feasible tools and number of test cases by 42 (88% reduction) Analysis (BVA), Choice empirical studies to work Relationship Framework (CRF) P19 Test case combination Six approaches are presented to improve Machine learning approach in black-box test efficiency testing P20 Combination of test models in *NO GAPS MENTIONED* Higher coverage is achieved compared with MATLAB/Simulink manually created test cases P21 Automatically generate test Refine the functional coverage model Improvement actions are identified for test cases case generation P22 Proposed unified model Implement a large survey on software More than 90% of the requirements are specifications in Johnson Controls represented by the model company P26 End-to-end test framework Investigate the nature of the test model and Automation of executable test script the relevance to the generated test cases generation P28 Search-based testing principles Further investigate safety requirements Promising approach to ensure safety requirements Technique 6 P4 Similarity-Based Product Include more Solution space information Improvement in the effectiveness of SPL Comparison(24%) Prioritization w.r.t Deltas (Such as Source code) testing P6 New model-based method for Implement performance evaluation with a Future regression testing can be sped up test case prioritization large-scale case study P7 Combined fault diagnosis and Evaluate using an end-of-line test system System can find test cases to increase the test case selection at a real assembly line test coverage P13 Weight-based search algorithms Study weight tuning, different fitness Results suggest that all the search functions, and cost-effectiveness measures algorithms outperform Random Search P23 Model slicing technique for *NO GAPS MENTIONED* Complexity of Simulink models can be optimal test case generation reduced P24 OUTFIT tool Evaluate it with other domains such as Potential defects can be effectively medical and avionic systems identified Statistical 1 P9 Dependences between the Study exhaustiveness of the path search Reduction in test-cases for regression Evaluation (4%) components of embedded and correctness of path search testing systems Simulation 1 P12 Parallelly execute loosely Fully automate the process of segmentation Reduce the simulation testing time for both (4%) coupled segments and instrumentation successful and failed runs Others 3 P14 Three approaches to Focus on MC/DC test case generation from Approaches can be combined to support (12%) automatically generate MC/DC formal specifications different kinds of decisions test cases P25 Survey paper Publish more detailed description of the Improvements that are possible with applied evaluation approach MBAT technologies P27 Study on requirements *NO GAPS MENTIONED* Synect provides easy test requirement specifications and management of the test results Copyright © 2018 for this paper by its authors. 56 6th International Workshop on Quantitative Approaches to Software Quality (QuASoQ 2018) environment, it should be used to provide early feedback TABLE VI on the vehicle’s design. VALIDATION Techniques #Studies Studies C. Quality Evaluation (RQ3) Industrial Case Study 14 P5, P8, P10, P11, P15, According to [29], a ranking was created to rate papers P16, P17, P18, P19, P20, P21, P22, P26, P28 based on the relevance to the topic and the quality of the Technique Comparison 6 P4, P6, P7, P13, P23, P24 paper. Quality Issues (QI) are: Statistical Evaluation 1 P9 Simulation 1 P12 • QI1 Is the paper’s goal clear? Others 3 P14, P25, P27 • QI2 Does the assessment approach match the goals? • QI3 Can the method be replicated? • QI4 Are results shown in detail? studies are focus on industrial case studies (n = 14) and technique comparisons (n = 6). For each quality issue, articles were rated as: Yes (Y) when Table VI lists the techniques used to validate the selected the issue is addressed in the text, Partial (P) when the issue is studies. The most common are industrial case studies partially addressed in the text, and No (N) when the issue is (14/25) followed by technique comparisons (6/25), statistical not addressed in the text. These ratings were scored as Yes = 1 evaluations (1/25), simulations (1/25), and others (3/25). point, Partial = 0.5, and No = 0. Table III shows the papers that Technique comparisons include studies that propose and then were analyzed in this SLR and their respective scores based compare a new method to an old one. Others include three on the Quality Issues discussed above. studies, which [14] [27] talk about the advantages, limitations, D. Existing Research (RQ4) and requirements of different approaches. [25] is a survey paper from 13 industry case studies. Here we discuss the recurring problems that are targeted by primary studies, which methods described in RQ2 are V. T HREATS TO VALIDITY validated, and the gaps mentioned in the research. Recurring problems are time consumption (15/25), cost The analysis was conducted by a single person. Thus, one (13/25), and complexity (14/25) followed by test case selection threat is that some information may be omitted. Moreover, the (3/25) and quality improvement (3/25) (Table V). Because analysis is limited by the analytical skills of that single person. the testing time is expensive, it should be reduced without VI. C ONCLUSION an uncontrolled reduction of reliability. The entire test suite must be rerun each time the system is updated or modified. This paper overviews the Prioritization in Automotive Consequently, each modification makes the testing process Software Testing. The results should help companies that more time-consuming. Automotive systems are becoming are planning to incorporate prioritization into their strategies. more complex due to a higher rate of integration and shared Researchers can also benefit because this study depicts the usage. The high complexity results in numerous interfaces, limitations and gaps in current research. Additionally, the and many signals must be processed inside the system [9]. trends in other embedded and non-embedded domains must Testing activities can account for a considerable part of the be investigated as this should provide a more detailed picture software production costs. However, only two studies discuss and lessons learned regarding prioritization in Automotive improving efficiency [18] and safety [26] which is a negative Software Testing. Future work includes (i) a qualitative study highlight. to better understand test execution, test case generation, Table IV presents the studies within each category, test case selection, and test analysis and (ii) addressing the Techniques/Tools, gaps, and the main study outcomes. Most identified research gaps. R EFERENCES TABLE V [1] B. Kitchenham and S. Charters, “Guidelines for performing systematic TARGET PROBLEMS literature reviews in software engineering,” 2007. [2] B. Kitchenham and P. Brereton, “A systematic review of systematic Problems #Studies Studies review process research in software engineering,” Inf. Softw. Technol., Time Consumption 15 P5, P6, P8, P9, P10, P12, vol. 55, no. 12, pp. 2049–2075, Dec. 2013. [Online]. Available: P13, P14, P15, P16, P17, http://dx.doi.org/10.1016/j.infsof.2013.07.010 P19, P20, P24, P27 [3] D. M. Rafi, K. R. K. Moses, K. Petersen, and M. V. Mäntylä, “Benefits Complexity 14 P5, P6, P7, P8, P9, P10, and limitations of automated software testing: Systematic literature P13, P15, P16, P17, P19, review and practitioner survey,” in 2012 7th International Workshop on P20, P24, P27 Automation of Software Test (AST), June 2012, pp. 36–42. Cost 13 P5, P6, P8, P9, P10, P13, [4] M. Al-Hajjaji, S. Lity, R. Lachmann, T. Thüm, I. Schaefer, and G. Saake, P15, P16, P17, P19, P20, “Delta-oriented product prioritization for similarity-based product-line P24, P27 testing,” in 2017 IEEE/ACM 2nd International Workshop on Variability Test Case Selection 3 P7, P11, P28 and Complexity in Software Design (VACE), May 2017, pp. 34–40. Quality Improvement 3 P22,P25,P26 [5] I. Alagöz, T. Herpel, and R. German, “A selection method for black Test Generation 2 P21,P23 box regression testing with a statistically defined quality level,” in 2017 Problem Space Information 1 P4 IEEE International Conference on Software Testing, Verification and Improving Efficiency 1 P18 Validation (ICST), March 2017, pp. 114–125. Safety 1 P26 Copyright © 2018 for this paper by its authors. 57 6th International Workshop on Quantitative Approaches to Software Quality (QuASoQ 2018) [6] A. Morozov, K. Ding, T. Chen, and K. Janschek, “Test suite prioritization [21] R. Awedikian and B. Yannou, “Design of a validation test process of for efficient regression testing of model-based automotive software,” an automotive software,” International Journal on Interactive Design in 2017 International Conference on Software Analysis, Testing and and Manufacturing, vol. 4, no. 4, pp. 259–268, 2010, cited By :2. Evolution (SATE), Nov 2017, pp. 20–29. [Online]. Available: www.scopus.com [7] S. Abele and M. Weyrich, “A combined fault diagnosis and test case [22] R. Awedikian, B. Yannou, P. Lebreton, L. Bouclier, and M. Mekhilef, selection assistant for automotive end-of-line test systems,” in 2016 IEEE “A simulated model of software specifications for automating functional 14th International Conference on Industrial Informatics (INDIN), July tests design,” in Proceedings DESIGN 2008, the 10th International 2016, pp. 1072–1077. Design Conference, 2008, pp. 561–570, cited By :1. [Online]. Available: [8] S. Vöst and S. Wagner, “Trace-based test selection to support continuous www.scopus.com integration in the automotive industry,” in 2016 IEEE/ACM International [23] Z. Jiang, X. Wu, Z. Dong, and M. Mu, “Optimal test case generation Workshop on Continuous Software Evolution and Delivery (CSED), May for simulink models using slicing,” in Proceedings - 2017 IEEE 2016, pp. 34–40. International Conference on Software Quality, Reliability and Security [9] P. Caliebe, T. Herpel, and R. German, “Dependency-based test case Companion, QRS-C 2017, 2017, pp. 363–369. [Online]. Available: selection and prioritization in embedded systems,” in 2012 IEEE www.scopus.com Fifth International Conference on Software Testing, Verification and [24] D. Holling, A. Hofbauer, A. Pretschner, and M. Gemmar, “Profiting Validation, April 2012, pp. 731–735. from unit tests for integration testing,” in Proceedings - 2016 [10] L. Krejčı́ and J. Novák, “Model-based testing of automotive distributed IEEE International Conference on Software Testing, Verification and systems with automated prioritization,” in 2017 9th IEEE International Validation, ICST 2016, 2016, pp. 353–363, cited By :1. [Online]. Conference on Intelligent Data Acquisition and Advanced Computing Available: www.scopus.com Systems: Technology and Applications (IDAACS), vol. 2, Sept 2017, pp. [25] M. Klas, T. Bauer, A. Dereani, T. Soderqvist, and P. Helle, 668–673. “A large-scale technology evaluation study: Effects of model-based [11] R. Lachmann, S. Lity, M. Al-Hajjaji, F. Fürchtegott, and I. Schaefer, analysis and testing,” in Proceedings - International Conference on “Fine-grained test case prioritization for integration testing of Software Engineering, vol. 2, 2015, pp. 119–128, cited By :4. [Online]. delta-oriented software product lines,” in Proceedings of the 7th Available: www.scopus.com International Workshop on Feature-Oriented Software Development, [26] J. Lasalle, F. Peureux, and J. Guillet, “Automatic test concretization to ser. FOSD 2016. New York, NY, USA: ACM, 2016, pp. 1–10. supply end-to-end mbt for automotive mechatronic systems,” in 2011 [Online]. Available: http://doi.acm.org/10.1145/3001867.3001868 International Workshop on End-to-End Test Script Engineering, ETSE [12] M. A. Al Mamun and J. Hansson, “Reducing simulation testing 2011 - Proceedings, 2011, pp. 16–23, cited By :4. [Online]. Available: time by parallel execution of loosely coupled segments of a test www.scopus.com scenario,” in Proceedings of International Workshop on Engineering [27] A. Bansal, M. Muli, and K. Patil, “Taming complexity while gaining Simulations for Cyber-Physical Systems, ser. ES4CPS ’14. New efficiency: Requirements for the next generation of test automation York, NY, USA: ACM, 2007, pp. 33:33–33:37. [Online]. Available: tools,” in AUTOTESTCON (Proceedings), 2013, pp. 123–128, cited By http://doi.acm.org/10.1145/2559627.2559635 :1. [Online]. Available: www.scopus.com [13] A. Arrieta, S. Wang, G. Sagardui, and L. Etxeberria, “Test [28] F. Lindlar and A. Windisch, “A search-based approach to functional case prioritization of configurable cyber-physical systems with hardware-in-the-loop testing,” in Proceedings - 2nd International weight-based search algorithms,” in Proceedings of the Genetic and Symposium on Search Based Software Engineering, SSBSE 2010, 2010, Evolutionary Computation Conference 2016, ser. GECCO ’16. New pp. 111–119, cited By :7. [Online]. Available: www.scopus.com York, NY, USA: ACM, 2016, pp. 1053–1060. [Online]. Available: [29] U. Kanewala and J. M. Bieman, “Testing scientific software: http://doi.acm.org/10.1145/2908812.2908871 A systematic literature review,” Inf. Softw. Technol., vol. 56, [14] S. Kangoye, A. Todoskoff, M. BARREAU, and P. GERMANICUS, no. 10, pp. 1219–1232, Oct. 2014. [Online]. Available: “Mc/dc test case generation approaches for decisions,” in Proceedings of http://dx.doi.org/10.1016/j.infsof.2014.05.006 the ASWEC 2015 24th Australasian Software Engineering Conference, ser. ASWEC ’ 15 Vol. II. New York, NY, USA: ACM, 2015, pp. 74–80. [Online]. Available: http://doi.acm.org/10.1145/2811681.2811696 [15] R. Matinnejad, S. Nejati, L. C. Briand, and T. Bruckmann, “Effective test suites for mixed discrete-continuous stateflow controllers,” in Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, ser. ESEC/FSE 2015. New York, NY, USA: ACM, 2015, pp. 84–95. [Online]. Available: http://doi.acm.org/10.1145/2786805.2786818 [16] P. M. Kruse, J. Wegener, and S. Wappler, “A highly configurable test system for evolutionary black-box testing of embedded systems,” in Proceedings of the 11th Annual Conference on Genetic and Evolutionary Computation, ser. GECCO ’09. New York, NY, USA: ACM, 2009, pp. 1545–1552. [Online]. Available: http://doi.acm.org/10.1145/1569901.1570108 [17] F. Merz, C. Sinz, H. Post, T. Gorges, and T. Kropf, “Bridging the gap between test cases and requirements by abstract testing,” Innov. Syst. Softw. Eng., vol. 11, no. 4, pp. 233–242, Dec. 2015. [Online]. Available: http://dx.doi.org/10.1007/s11334-015-0245-7 [18] J. S. Eo, H. R. Choi, R. Gao, S. y. Lee, and W. E. Wong, “Case study of requirements-based test case generation on an automotive domain,” in 2015 IEEE International Conference on Software Quality, Reliability and Security - Companion, Aug 2015, pp. 210–215. [19] R. Lachmann and I. Schaefer, “Towards efficient and effective testing in automotive software development,” in Lecture Notes in Informatics (LNI), Proceedings - Series of the Gesellschaft fur Informatik (GI), vol. P-232, 2014, pp. 2181–2192, cited By :2. [Online]. Available: www.scopus.com [20] S. Siegl, K. S. Hielscher, and R. German, “Modeling and statistical testing of real time embedded automotive systems by combination of test models and reference models in matlab/simulink,” in 2011 21st International Conference on Systems Engineering, Aug 2011, pp. 180–185. Copyright © 2018 for this paper by its authors. 58