Jianing Sun∗, Jiahui Wang, Yuyan Zhu, Xingyu Li, Ying Xie and Jiaxin Chen Chongqing University, 400044 Chongqing, China Abstract This paper presents a novel method for generating automated test scripts for Domain-Specific Languages (DSLs) in software testing, particularly for the automotive industry. It emphasizes the growing importance of software testing in ensuring product quality amid IT advancements. The paper reviews software testing's evolution, modern processes, and the role of Large Language Models (LLMs). It highlights DSLs' significance and uses the automotive sector to show how LLMs can automate test script generation. Tests indicate that in cases with a small sample size, the effectiveness of prompt engineering is superior to model fine-tuning. The proposed method thus relies on well-designed prompts to direct LLMs to produce accurate scripts. The generation system's overview is discussed, along with an evaluation of the scripts' quality using metrics like Levenshtein Distance. Results indicate that LLMs boost test automation, defect detection, and software reliability. Future work will optimize these tools for higher testing automation levels. Keywords software testing, domain-specific languages, large language models, Levenshtein Distance 1 1. Introduction describing testing as executing a program to uncover errors [6]. Software testing is a key component in ensuring the By 1983, IEEE had standardized software testing, quality and reliability of software products. In the defining it as a process -manual or automated- to verify rapidly developing information technology era, software system requirements [7]. The 1990s brought agile has become an indispensable part of our daily life and methodologies, integrating testing and development and work. With the increasing complexity and encouraging tester involvement from the earliest diversification of software functions, the importance of development stages [8]. In the 21st century, testing has software testing has become increasingly prominent. advanced, with a focus on exploratory testing that Software testing is a series of processes designed to highlights the tester initiative. The era of AI and big data check that a software product meets specified has intensified scrutiny of software testing. Despite still requirements and ensures its quality. It not only helps leveraging 20th-century methods, the field anticipates developers to find and fix defects, but also greatly future innovations, potentially revolutionizing testing enhances system security, especially in fields with high practices [9]. software safety requirements such as automotive and aviation [1]. 1.2. Modern Approaches 1.1. A Brief History of Software Testing The modern software testing process is crucial for ensuring software quality and functionality. It starts The origins of software testing date back to the 1950s, with requirement analysis, followed by developing a test focusing initially on debugging to identify and rectify plan, designing test cases, and preparing test data software faults [2][3][4]. As software complexity grew, (Figure 1). The test environment is set up, tests are the need for independent testing organizations became executed and recorded, and defects are tracked. apparent. In 1957, Charles Baker first defined program Regression and performance testing are conducted, testing, in his review of the book Digital Computer Programming by Dan McCracken, separating it from debugging. Bill Hetzel formalized software testing as a concept at the University of North Carolina in 1972, establishing it to ensure a program performs as intended Figure 1: Modern software testing process. [5]. Glenford J. Myers further refined this in 1979, QuASoQ 2024：12th International Workshop on Quantitative Approaches 0009-0001-4943-6038 (J. Sun); 0009-0000-0107-0666 (J. Wang); 0009- to Software Quality，3rd December 2024，Chongqing, China, 0008-1670-4607 (Y. Zhu); 0009-0009-4526-007X (X. Li); 0009-0003-8939- ∗ Corresponding author. 7832 (Y. Xie); 0009-0002-4492-4734 (J. Chen) © 2024 Copyright for this paper by its authors. Use permitted under j.sun@cqu.edu.cn (J. Sun) Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings 52 along with security and system testing. Acceptance aid in document summarization, provide travel advice, testing confirms business requirements are met. Test and improve user engagement. Tools like GitHub reports summarize results, and evaluations identify Copilot demonstrate their advantage by assisting in process improvements. coding tasks [10]. Techniques like automated testing, Continuous Also, LLMs boost software testing by automating Integration (CI), and Continuous Delivery (CD) enhance tasks, detecting defects, and ensuring reliability. They testing efficiency. Agile testing fosters collaboration improve fuzz and unit testing, creating test cases, and between testers and developers. Performance, security, suggesting fixes. Research shows their significant and mobile application testing ensure software benefits in expanding test coverage and error detection. reliability across different aspects. Cloud testing Future efforts will focus on optimizing testing tools and leverages cloud resources for extensive testing. techniques. 2. Domain-Specific Software Testing 2.1. Domain-Specific Languages Domain-specific languages (DSLs) are specialized languages designed for particular domains or tasks, offering simplified syntax for ease of use by domain experts [11]. They can be integrated with general- Figure 2: CI/CD is a software development practice purpose languages (GPL) like Java and C++, enhancing in which code changes are automatically integrated, development efficiency through tool support such as built, and tested, with successful builds being deployed analyzers and compilers. DSLs are crucial in various to production. industries, for example, HTML in web development and SQL in databases. They automate tasks like API AI-based testing leverages machine learning to documentation and legal document generation, automate software testing processes, enhancing improving efficiency and reducing errors. DSLs also efficiency and accuracy. It encompasses exploratory facilitate team collaboration by allowing non-technical testing to identify issues without fixed test cases, members to express requirements in a natural language- ensuring broader coverage. Model-driven testing and like format. testability design optimize test case generation and For software development, DSLs boost efficiency by software sustainability. Additionally, managing test data simplifying complex representations and promoting and implementing strategies such as Test Left Shift and code reusability. They accelerate prototyping and Test Right Shift further refine the development cycle. iteration, integrating seamlessly into existing tools and These dynamic approaches adapt to different workflows. Testing DSL-developed programs requires a methodologies, ensuring consistent software quality detailed plan with automated test scripts for regression throughout the testing process. testing. Test cases must be readable, and test data should reflect the domain specifics to ensure comprehensive 1.3. Large Language Models coverage and identify defects. Maintainability of test cases and DSL is essential for ongoing development Large Language Models (LLMs) are cutting-edge AI success. specialized in natural language understanding and generation. Trained on extensive datasets and 2.2. Software Testing using DSL from the employing neural networks like Transformers, they capture linguistic subtleties and perform a variety of Automobile Industry language tasks such as categorization, analysis, This section addresses the critical need for rigorous translation, and Q&A systems. They discern nuances, testing in passenger car product development, ensuring generate realistic text, and continuously adapt to quality and performance meet standards. linguistic evolution, raising concerns over data privacy Traditionally, automotive testing relies on manual, and ethics. labor-intensive translation of requirements into test LLMs have significantly impacted sectors like smart cases and scripts, causing significant strain on resources. offices, travel, e-commerce, and government by To streamline this process and integrate Continuous enhancing efficiency and personalization. In software Testing/Continuous Delivery development, LLMs are revolutionizing the field. They 53 Table 1 A segment of the test data, describing the test cases, pre-conditions, and the desired test script DSL to be generated. Test case Pre-condition Test script Forward gear is Configuration.Gears.Drive.is_activated() Signals.Check(signals=[Gears_GearsStatus], activated when the values=[Gears_GearsStatus_shift], waiting_time=100) car is moving Gear.Shift('drive') forward Reverse gear is Configuration.Gears.Reverse.is_activated() Signals.Check(signals=[Gears_GearsStatus], activated when the values=[Gears_GearsStatus_shift], waiting_time=100) car is being driven Gear.Shift('reverse') reversely At driving P gear is Configuration.Gears.Park.is_deactivated() current_speed = car.get_speed() deactivated current_gear = car.get_gear_status() Self.assertNotEqual(current_gear, 'park') car.stop() car.shift_gear('park') current_gear = car.get_gear_status() (CT/CD) pipelines, the industry is moving towards limited data, prompt engineering emerges as the slightly automated test development. superior approach. The automotive sector is in pursuit of an AI-driven Consequently, we have intentionally opted to solution to streamline the automation of test script employ the finesse of prompt engineering for the generation for its proprietary DSLs, which are integral automated crafting of test scripts. This strategic choice to the testing of a spectrum of automotive systems. The is rooted in its proven ability to deliver optimized prevailing manual methodology is marred by outcomes, even within the confines of our data scarcity. inefficiencies, susceptibility to errors, and variability in By leveraging the finesse of prompt engineering, we aim code quality, alongside insufficient test coverage. By to transcend the limitations imposed by scant data harnessing the capabilities of large language models, an availability, thereby enhancing the overall performance AI-powered tool has the potential to orchestrate this and reliability of our test script generation process. process, amplifying efficiency, curbing errors, and An integral element of our approach is the selection upholding uniformity in code excellence, thereby of the foundational Large Language Model. To this end, conquering existing challenges and invigorating the we have undertaken a model selection process, software development lifecycle. meticulously assessing ChatGLM3, Llama3, and Qwen2. Following an exhaustive comparison, we determined 2.3. Sample Data that Llama3's generative capabilities align more closely with our requirements. Hence, we have chosen Llama3 A total of 51 data samples (Table 1), each representing to serve as the underlying LLM for this study. a true mapping from a test case to a test script in a particular DSL format. For privacy protection purposes, all information, 3.1. Prompt to Make Precise Test Script program code and data in this paper have been Generation anonymized. Through a meticulous process of refinement, we've perfected our prompt for generating test scripts, as 3. Approach shown in the example. This fine-tuning ensures our AI model produces outputs that are both accurate and meet Broadly speaking, two dominant strategies have our objectives. emerged for augmenting the knowledge base: the art of Our prompt is divided into four key components prompt engineering, which is particularly effective for (Table 2): First, an exhaustive list of potential samples, modest datasets, and the process of model fine-tuning, excluding the current focus, provides a comprehensive which is best suited for addressing more substantial training context. Second, we concentrate on the specific volumes of data. Considering the current data landscape, test purpose to create targeted, efficient test scripts. characterized by a dearth of samples and inherent Third, we provide clear instructions in natural English uncertainties, a comprehensive evaluation was for the LLM to follow, ensuring a seamless and accurate undertaken to compare the merits of both prompt generation process. Lastly, we impose constraints to engineering and LLM fine-tuning. This analysis has optimize the generation process, enabling our LLM demonstrated that, under the present circumstances of 54 model to autonomously produce precise and relevant reflecting deeper learning and improved autonomy in test scripts without excessive input. script generation, ultimately advancing AI in software testing. Table 2 Pseudo code of prompt design. Prompt for test script generation (pseudo code) dataFrame = All sample mappings except the one which is being generated testCase = one test case which is being processed instruction = “Above is a list of test cases and corresponding test scripts, assembled in Json format. Please generate test script for the following test case:” condition = “Please export generated test script only, Figure 3: Generation System Overview. no leading text, no leading new lines.” prompt = dataFrame + CRLF + instruction + testCase 4. Result and Evaluation + condition This structured approach not only boosts script 4.1. Evaluation Metric accuracy but also enhances the efficiency of our testing This paper employs the Levenshtein Distance [12] to process, bringing us closer to our goal of fully automated evaluate the textual accuracy of our language model, AI-driven test script generation. providing an objective measure of how closely generated text matches the ground truth. This edit 3.2. Test Script Generation System distance metric, devised by Vladimir Levenshtein, Overview quantifies the minimum number of single-character The test script generation system, illustrated in Figure edits required to transform one string into another, 3, converts input test cases into executable scripts in the offering insights into model performance. It plays a partner's DSL language, verifying product functionality. crucial role in fields like Natural Language Processing, It uses outlined methodologies, and scripts are evaluated where it assesses text similarity, and Bioinformatics, by experts for accuracy and reliability, with corrections where it indicates genetic relatedness. Despite its higher made as needed. computational demands for longer strings, our use of Validated scripts are executed and stored, dynamic programming makes it an efficient tool for our informing future prompts and enhancing script analysis. The Levenshtein Distance aids in refining our generation over time. This cycle of evaluation and model, ensuring that the text generation is both accurate learning improves script quality and reduces manual and reliable. creation, aiming for an automated, self-improving The formal definition of Levenshtein Distance system that streamlines software testing. As data between two arbitrary strings 𝑎 and 𝑏 with length of storage grows, prompts become more complex, |𝑎| and |𝑏| respectively is given by |𝑎|, if |𝑏| = 0, ⎧ |𝑏|, if |𝑎| = 0, ⎪ ⎪ 𝐥𝐞𝐯 tail(𝑎), tail(𝑏) , if head(𝑎) = head(𝑏), 𝐥𝐞𝐯(𝑎, 𝑏) = 𝐥𝐞𝐯(tail(𝑎), 𝑏) ⎨ ⎪ 1 + min 𝐥𝐞𝐯 𝑎, tail(𝑏) , otherwise. ⎪ ⎩ 𝐥𝐞𝐯 tail(𝑎), tail(𝑏) 4.2. Result and Discussion where tail(𝑥) of any string 𝑥 of length 𝑛 is a In our comprehensive analysis, we have utilized the substring of 𝑥 without the first character, i.e. tail(𝑥) = Levenshtein Distance alongside the test script tail(𝑥 𝑥 ⋯ 𝑥 ) = 𝑥 𝑥 ⋯ 𝑥 and head(𝑥) of any generation methodologies previously discussed to assess string 𝑥 of length 𝑛 is a substring of 𝑥 without the last the output across all 51 data samples. It is crucial to character, i.e. head(𝑥) = head(𝑥 𝑥 ⋯ 𝑥 ) = highlight the exceptional stability achieved with the 𝑥 𝑥 ⋯𝑥 . prompts we've designed, particularly when employing 55 Llama3 as our LLM. The consistency of Llama3 is our continuous efforts to enhance its capabilities. noteworthy; for a given data sample, or in other words, Moreover, this stability ensures that our test script with the same prompt, the model reliably produces generation process is not only efficient but also identical results in each test scenario. This uniformity is dependable, providing our partners and users with a tool a testament to the robustness of our prompt engineering that they can trust to deliver consistent results. and the model's ability to deliver reliable outcomes. This The test results for all 51 samples are displayed in level of consistency is not only a significant advantage the horizontal bar chart in Figure 4, offering a clear in the context of test script generation but also a key visual representation of our system's performance. The factor in ensuring the reproducibility of our red bars in the chart signify the text lengths of the experiments. It allows us to confidently attribute any ground truth test scripts, serving as a benchmark for variations in the output to changes in the input data or comparison. It represents the ideal output, against which to the model's fine-tuning, rather than to the inherent the effectiveness of our system is measured. The pink instability of the model itself. By achieving such a high bars, on the other hand, denote the lengths of the test degree of stability, we pave the way for more accurate scripts generated by our system. This provides insight and meaningful evaluations of our model's performance, into the output of our AI-driven script generation which in turn, informs process, highlighting the efficiency and effectiveness with which our system translates prompts into executable test scripts. Most importantly, the blue bars in the chart represent the Levenshtein Distances for each sample, a critical metric that quantifies the difference between the generated test scripts and the ground truth. This distance is calculated based on the minimum number of single-character edits required to transform the generated test scripts into the ground truth test scripts. In this context, a shorter blue bar indicates a higher degree of similarity, suggesting that the generated script closely mirrors the ground truth, which is the goal of our system. As observed from Error! Reference source not found., it is evident that the system currently exhibits a noticeable margin of error. This finding is further accentuated and clarified in the subsequent statistical box plot in Figure 5, which provides a more detailed visualization of the distribution of errors across our dataset. It is apparent that our dataset, comprising a mere 51 samples, is significantly limited for a deep learning initiative. The consensus in the field is that a larger dataset is often necessary to train models to achieve higher accuracy and reliability. Figure 4: Discrete distribution as a horizontal bar chart to illustrate the result evaluation. Figure 5: Box plot display of the generated test results. 56 However, it is remarkable to note that despite this [2] Campbell, Robert V. D. “Evolution of Automatic constraint, our system has produced flawless results in Computation.” In Proceedings of the 1952 ACM six instances where the generated test scripts matched National Meeting (Pittsburgh), 29–32. ACM ’52. the ground truth perfectly. This achievement is New York, NY, USA: Association for Computing particularly impressive given the small sample size and Machinery, 1952. serves as a testament to the potential of our approach https://doi.org/10.1145/609784.609786. using prompt engineering with large language models. [3] Orden, Alex. “Solution of Systems of Linear The fact that our system was able to generate scripts Inequalities on a Digital Computer.” In indistinguishable from the ground truth in these cases Proceedings of the 1952 ACM National Meeting suggests that with further optimization and a more (Pittsburgh), 91–95. ACM ’52. New York, NY, USA: extensive dataset, we could see a substantial Association for Computing Machinery, 1952. improvement in the system's overall performance. https://doi.org/10.1145/609784.609793. This early success with a limited dataset is not just [4] Demuth, Howard B., John B. Jackson, Edmund encouraging; it also validates the feasibility of our Klein, N. Metropolis, Walter Orvedahl, and James methodological approach. It indicates that our system H. Richardson. “MANIAC.” In Proceedings of the has the innate capacity to learn and produce high- 1952 ACM National Meeting (Toronto), 13–16. quality outputs, even when faced with data scarcity. As ACM ’52. New York, NY, USA: Association for we continue to expand our dataset and refine our Computing Machinery, 1952. models, we are confident that the performance will see https://doi.org/10.1145/800259.808982. a marked enhancement, further solidifying the [5] Hetzel, William C. Program Test Methods. effectiveness of our AI-driven test script generation Prentice-Hall, 1973. system in the field of software testing. [6] Myers, Glenford J., Corey Sandler, and Tom Badgett. The Art of Software Testing. John Wiley 5. Conclusion and Future Work & Sons, 2011. [7] “IEEE Standard for Software Test Documentation.” This research highlights the significant impact of LLMs Accessed September 17, 2024. on enhancing software testing efficiency, particularly in https://standards.ieee.org/ieee/829/1217/. the automotive sector. Our findings underscore the [8] Martin, James. Rapid Application Development. superiority of prompt engineering over model fine- Macmillan Publishing Company, 1991. tuning, especially with smaller datasets. The [9] Khaliq, Zubair, Sheikh Umar Farooq, and Dawood Levenshtein Distance proved a reliable metric for script Ashraf Khan. “Artificial Intelligence in Software accuracy. Notably, LLMs, such as Llama3, demonstrated Testing : Impact, Problems, Challenges and remarkable consistency, indicating the robustness of our Prospect.” arXiv, January 14, 2022. framework. Even with a limited dataset, our system https://doi.org/10.48550/arXiv.2201.05371. achieved high accuracy, showcasing LLMs' potential in [10] Schäfer, Max, Sarah Nadi, Aryaz Eghbali, and software testing. Frank Tip. “An Empirical Evaluation of Using Our study introduces a novel approach to DSL Large Language Models for Automated Unit Test testing, with a user-friendly web application for our test Generation.” IEEE Transactions on Software script generation system, enhancing accessibility and Engineering 50, no. 1 (January 2024): 85–105. testing efficiency. Future work includes expanding our https://doi.org/10.1109/TSE.2023.3334955. dataset to improve script performance and integrating [11] “Domain Specific Languages.” Accessed the system into CI/CD pipelines for real-time testing. September 17, 2024. Ethical considerations and model transparency will also https://martinfowler.com/books/dsl.html. be prioritized. In conclusion, our research establishes [12] Levenshtein, Vladimir I. “Двоичные Коды с LLMs as a viable solution for automating DSL test script Исправлением Выпадений, Вставок и generation, laying the groundwork for future Замещений Символов [Binary Codes Capable of advancements in AI-assisted software testing. Correcting Deletions, Insertions, and Reversals].” Soviet Physics Doklady 163, no. 4 (February 1966): References 845–48. [1] Awedikian, Roy, and Bernard Yannou. “Design of a Validation Test Process of an Automotive Software.” International Journal on Interactive Design and Manufacturing (IJIDeM) 4, no. 4 (November 1, 2010): 259–68. https://doi.org/10.1007/s12008-010-0108-2. 57