Artificial intelligence (AI) techniques and models have been applied to assist numerous activities in software engineering, specifically in Software Quality Assurance (SQA), covering both technical and management parts. Several studies have examined the utilization of AI in various tasks within the SQA field, with a particular focus on software testing. This study seeks to investigate the overall impact of AI on SQA, while identifying the prevalent applications and possible areas for further research. We thoroughly examined a selection of articles that study the use of AI approaches in various SQA activities. We have selected relevant research papers that were published in the last 5 years. The analysis was conducted by utilizing established AI and SQA taxonomies and categorizing the chosen papers based on these taxonomies. The resulting mapping and conversations indicate that the use of AI to assist in SQA is a well-established and expanding area of scientific interest with interesting opportunities for future. Evidence of several AI benefits was found when applied to SQA such as cost reduction and process improvement, as well as challenges in case of high complexity and questionable data quality. The discussions of the impact of AI on the roles of the SQA showed better outcomes when adopting a Human-AI collaboration approach.
[
SQA as a practice, has experienced exponential growth lately following the growth of Software
Engineering. With growth come new challenges and complexities such as the difficulties to adapt
SQA to the organizational features and constraints and the high cost of implementing SQA activities
[
According to [
To ensure quality, SQA has a set of activities that starts early in a Software project’s life cycle. [
To prepare for the SQA function, the Software Quality Assurance Plan (SQAP) should be defined and signed off by all stakeholders of the software project as a first step. Once the SQAP is done then the PDCA is launched to guarantee a continuous improvement of the plan.
SQAP is a Project Planning artifact that defines the activities and tasks used to ensure that software
developed for a specific product satisfies the project’s established requirements and user needs within
project cost and schedule constraints and is commensurate with project risks [
The PDCA Cycle, also called The Deming Cycle, is a quality management procedure that ensures a
continuous process improvement [
Process assurance seeks to guarantee that the processes adopted by the project are adequate and are
followed by all stakeholders. This mitigates the risk of process risks that may lead to delays and extra
costs in the project, eventually impacting the quality of the concerned software. Process Assurance
is considerably important in case there are external participants in the project [
It is ensured through product assurance activities that software products are developed in compliance
with contractual requirements, project schedules, and established product requirements. These
products include not only the software and related documentation but also the plans associated with
the development, operation, support, maintenance, and retirement of the software. According to [
Requirement Review.
Design Review.
Software Testing.
User Acceptance Tests.
AI refers to machines capable of simulating and mimicking the human intelligence by thinking and
learning from their experiences. This technology includes a set of features such as problem-solving,
pattern recognition, understanding natural language and decision making [
AI has many definitions and can be seen from many perspectives, The taxonomy outlined in the [
Learning deals with the capacity of systems, without explicit programming, to learn, decide, forecast, adapt, and respond to changes and improve from experience automatically.
Communication relating to the capacities of recognizing, analyzing, comprehending, and producing information from oral and written human communications. Mainly, Natural Language Processing (NLP) covers this field.
Integration and Interaction addresses the combination of the previous domains to showcase an intelligent behavior through Agents and bots.
We are classified the findings of our reviews following this taxonomy in Table 1.
In this paper, we used literature review methodology to investigate the current state of the art of AI application in SQA. The selection method we adopted is based on multi-step approach. First, we build a comprehensive search strategy based on combinations of selected keywords and Boolean operators: “AI,” “Artificial Intelligence,” “Machine Learning,” “Deep Learning,” “Software Quality Assurance,” “SQA,” “Software Testing,” “Process Assurance,” “Product Assurance” “Generative AI,” and “Software Engineering,”. This search was conducted across various academic databases of computer science, engineering and software engineering disciplines. Second, we applied inclusion criteria to the retrieved articles. Articles were mainly selected if they were published within the last 5 years (20192024) in journals or conference proceedings, and if they explored the use of AI techniques in SQA tasks. Finally, after screening the title, abstract and full text, our methodology resulted in 16 articles that met our criteria and formed the basis of this review.
SQAP is an activity that falls under the Project Planning umbrella, the paper in [
There is also another recent study [
To avoid having issues in the product, the process of developing the product needs to be monitored
and tracked. The challenges that are faced in this task are usually related to the complexity of the
used processes in the project, the adherence of different contributors to the agreed-on processes and
the efficiency of processes. AI can come with great benefits to address these challenges. [
Software Configuration Management is an activity that aims to manage the software configurations
and versioning to track changes. Ensuring that all parties respect the configuration management is
an important activity of SQA, however, there might be some challenges related to the difficulty of
tracking all changes and keeping records accessible and exploitable. [
People Skill Management is an important activity of Process Assurance because the skill sets of the people involved in the process of developing the software are key to the quality assurance of the software. This task is challenging as it relies on the human personality traits and knowledge that are relative parameters and difficult to measure. According to [19], AI can be used in People Skill Management by predicting human interactions with their environment in the software project operations. The authors of [19] used Agent Based Modeling and Simulation (ABMS) that considers various human related factors such as personality traits, affective states, competencies, learnability, and individual interactions. The result showed that the prototype composes teams that perform well in terms of output quantity and development speed. The limitation of this model is that it’s based on the numeric correlations between human aspects that can be sometime hard to predict with data.
Software Development Life Cycle (SDLC) is a structured process or methodology used in software engineering to build high quality software applications with an optimized cost in the shortest time possible through planned phases and activities [20]. The Product Assurance activities depend on the adopted SDLC Model. There are many SDLC models such as Waterfall, RAD, Spiral, Agile, Prototype. In this paper we are considering the general model presented in Figure 2. In the upcoming 4 sections we are going to explore the usage of AI in each Product Assurance activity in its relationship with the SDLC corresponding stage.
AI significantly enhances the process of requirement reviews by automating the identification, analysis, and validation of software requirements. According to [21], NLP can retrieve valuable inputs from extensive documentation and identify key capabilities and potential requirements from various sources such as stakeholder interviews, documents, emails, and user feedback. NLP can also provide insights and detect any gaps or omissions in the requirements. [22] Also covered the requirement review by knowledge-based systems. These systems can manage the requirements phase efficiently by capturing and utilizing domain knowledge to ensure that the requirements are complete, consistent, and accurately reflect the stakeholders' needs.
Good software quality is a direct result of a well-defined design, but sometimes, human error and requirement complexity can leave space to gaps and functional misalignments to hide in the design. In this context, [23] discussed the importance of good software design and how the usage of model refactoring and machine learning can help maintain software quality. To identify the presence of functional misalignments and gaps in software models, the authors of [23] proposed the usage of a labeled dataset of metric values of UML class diagrams to train a deep neural network model using an adaptive supervised learning algorithm. After evaluation, the approach showed high accuracy and scalability to large and complex data.
Requirement Reviews will be performed in Stage 1,2.
Design Reviews will be performed in Stage 3.
Software Testing will be performed in Stage 4, 5, 6.
User Acceptance Tests will be performed in Stage 5, 6.
SQA is commonly mixed with Software Testing. However, Software Testing is one of the main
activities of the Product Assurance, which is an activity of SQA [
Software Testing is defined by [
Test Analysis and Design Test Implementation and execution Static Code Analysis Software Defect Prediction
Test Automation
One of the most important artifacts of Software Testing under SQA is the Test Case document [24]. It’s known in Software Testing that Test case creation is a challenging task that takes time and analytical skills to produce good quality test cases. Several Machine Learning Models have been subjects of test case generation studies. [25] compared the accuracy of these known models in the case of test case generation based on quality attribute scenarios (QAS). The results show Rain Forest combined with TF-IDF generated testcases with high accuracy, resulting in significant Test effort optimization in the project.
Test Implementation and Execution is a phase where we define the priority of each area of the software to be covered by testing. Test Case Prioritization (TCP) is a Software testing activity that is more crucial in the context of regression testing, whether manual or automated, as it reduces the costs by optimizing the testing effort and by detecting issues as early as possible. [26] discusses the various used approaches. According to this study, these are the most used techniques in TCP: Neural Networks, Bayesian Network, genetic algorithm, SVM, K-means. Results show that these techniques can achieve the continuous and adaptive TCP, improving performance of detecting faults earlier, achieve full coverage and faster fault detection.
Large-scale codebases may include possible flaws that traditional static code analysis methods miss, resulting in delayed troubleshooting and high-cost hotfixes. In [27], the authors introduced an ML prototype to enhance the Static Code Analysis, therefore, leverage SQA in sensitive and complex IT projects. The prototype is based on three ML procedures: the API Mining, the Sequential Pattern Mining, and the Frequent ItemSet Mining. Results show that the prototype can detect complex code patterns and deviations in code and automate error detection.
The importance of SDP in Software engineering comes with the challenge of discovering defect in the early stages of the software development to reduce its cost. Recently, this activity has been in the plat of many AI studies. [28] classifies the available SDP models that are based on ML and DL. A large set of techniques are widely used and have each difference advantages, we summed them up in the last section of this paper.
According to [29], Test Automation has been a trend in Software Testing for several years. It has been proven that test automation increases quality and delivery time. However, its biggest flaw is that it requires continuous human intervention that sometimes can be costly, time consuming and not efficient. This is where AI can make a difference. [29] presents a collection of AI methods used for software testing activities that are usually automated, such us GUI Testing, Black Box testing, Regression Testing, unit testing. The authors of [29] also highlighted the limitations and challenges that comes with using AI for test automation such as the Test automation complexity that becomes more complex when AI is added.
According to [30], UAT is a “Formal testing with respect to user needs, requirements, and business processes conducted to determine whether or not a system satisfies the acceptance criteria and to enable the user, customers or other authorized entity to determine whether or not to accept the system”. The authors of [31] worked on an approach for generating UAT testcases from scenarios that are captured and analyzed using NLP and Task/Method model. This approach was applied in the case of software development in the agricultural domain where the requirements and the scenarios of the agricultural processes change from a region to another. The integration of these AI techniques in the process of UAT resulted in an optimized UAT testing within the agile development environment.
Learning IntIengteraraticotnioannd
d
iignnM ii()rceeaagLLhnnnMM ltrrseeeeaokpuNNDw ()NND itsgoonB liiiftsscaaonC litrsegunC iitrssceeeonD ltseeebodhnEmM iirrseeeaSvgdpLunn ittrrcceeaSooppuhnVM iirrsseeeavgdpLunnnU lliItttseeeaggdnnnnA ttscaboh litseeeaaggoddnnnBM liitsaounm -4PTG
A
SQAP
[
AI techniques are introduced in SQA to optimize resources and address common SQA challenges. We extracted many benefits of using AI in SQA such as the cost reduction and the process optimization. We also found evidence of the challenges that necessitate further research and exploration such as the quality and representativeness of the training data. Table 2 summarizes these benefits and challenges and categorizes them by the potential impact on each SQA activity.
communication [
u P s
A -Potential biases within the training data can lead to flawed test outcomes [23,21,22,25,26,27,28,29]. -Difficulties in scaling with software complexity resulting in inaccurate testing [25,26,27,28,29].
Many of the papers we reviewed highlighted the impact of using AI on human aspects in the context
of SQA. On one hand, the positive impacts as AI proved to be efficient in repetitive task automation,
leaving more room for the SQA practitioners to focus on more complex tasks [
rod sru when relying only on human
P sA effort. The analysis of these impacts and challenging considerations points to the AI-Human Collaboration [32]. This approach is suggesting using AI techniques to support or automate software engineering tasks and incorporating human domain knowledge as starting points for designing AI techniques and using human feedback to improve these techniques, forming a continuous feedback loop, this approach seeks to reduce human efforts and the burden on human intelligence as shown in Table 3.
SQA is a crucial software engineering activity, that’s why organizations are investing efforts and resources to leverage their SQA processes using the latest technologies such as AI. We performed a paper review of AI application to all SQA activities. In these selected papers we also found evidence of its benefits such as cost reduction and Process improvement. AI application in this field also comes with challenges such as the high complexity and the data quality. As well as an impact of AI on the human aspects within the SQA field which was also a topic that we covered and discussed based on the outcomes of some of the reviewed papers that highlighted this aspect. The result of our review unrevealed many research gaps that can be covered by future research, one gap that caught our attention is the lack of research on the application of Generative AI and LLMs in SQA. In our future work we will propose an architecture of an AI Agent that helps functional roles involved in SQA such as the end users, QAs, Product Owners and Project Managers. The agent will bring a great value to SQA as it will save time and reduce test effort.
Athavale S, Balaraman V. Human behavioral modeling for enhanced software Project Management. In: 7th International Conference on Software Engineering; 2013. p. 15–7.
Viller, S., & Sommerville, I. (2000). Ethnographically informed analysis for software engineers. International Journal of Human-Computer Studies, 53(1), 169-196.
Muhairat, M., et al. (2020). An Intelligent Recommender System Based on Association Rule Analysis for Requirement Engineering. JUCS, Pensoft publishers.
K. Hema Shankari Int. Journal of Engineering Research and Applications
Brahmaleen Kaur Sidhu, et al. (2020): A machine learning approach to software model refactoring, International Journal of Computers and Applications
IEEE Std 829-2008, IEEE Standard for Software and System Test Documentation
A. Worku, et al. (2023). Test Case Generation from Quality Attribute Scenarios Using Machine Learning Approach.
Omri, S., & Sinz, C. (2021). Machine Learning Techniques for Software Quality Assurance: A Survey. arXiv preprint arXiv:2104.14056.
E. Sultanow et al. "Machine Learning based Static Code Analysis for Software Quality Assurance," 2018 ICDIM, Berlin, Germany, 2018
O.E. Emam, et al. Users Review’s on Software Defect Prediction Utilizing Machine Learning methods. vol. 6, no 1, p. 34-41.
Hayat, M. A., et al. Artificial Intelligence's Impact, Limitations, Challenges and Prospects in Software Testing.
« ISTQB® Certified Tester Acceptance Testing » Syllabus v1, 2019, [online] https://www.istqb.org/certifications/acceptance-tester
Antonelli, L., et al. (2023). User acceptance test for software development in the agricultural domain using natural language processing. Journal of Decision Systems.
Xie, T. (2013). "The synergy of human and artificial intelligence in software engineering," 2013 2nd International Workshop on (RAISE), San Francisco, CA, USA