In the domain of software engineering and quality assurance, test cases are vital for ensuring software dependability and efficiency, aligning with reliability standards and customer needs [1,2]. Despite their importance, software test cases face challenges such as standardization, interoperability, and adaptability across various testing environments [3]. Complex test cases are difficult to manage and modify, risking obsolescence and loss due to decentralized storage [4]. Most importantly, test cases still suffer from poor specifications, which lead to communication issues, inefficiencies, and increased costs [5,6]. Thus, software test cases need to be adequately specified. According to [7], an adequate test case has the following benefits: it effectively reveals defects with minimal effort, delivers accurate results, improves system performance at a lower cost, and has a strong likelihood of uncovering unknown defects.
In this work, we propose an ontology-based meta-modelling approach to support the design of adequate software test cases. This includes a new ontology-based domain-specific modelling language (DSML), called ontoST. To ensure rigor and extensibility, ontoST has been engineered by supplementing the Design Science Research (DSR) strategy [8] with the Agile Modelling Method Engineering (AMME) methodology [58].
The paper is structured as follows. Section 2 describes the background and related work, ending with the problem statement. Section 3 introduces the artifact requirements. Section 4 discusses the proposed ontology-based DSML ontoST. Section Error! Reference source not found. shows a running example of how ontoST has been implemented in the modeling tool AOAME4STC and subsequently used for the proof of concept. The paper concludes with section 6.
Software test cases are work products of test analysis and design phase of software testing as illustrated in Error! Reference source not found. by [9]. "Test Case" (TC) has been recognized as a building block for describing testing items, widely used as a work unit, metric and documentation entity" [3]. Software testing faces some challenges due to the complexities of software, financial and time constraints, and the need for high quality standards [10]. The increasing complexity of modern applications and competitive pressures further raise quality assurance standards [11]. Agile development introduces frequent requirement changes, complicating test case management [11,12]. Agile methodology's reliance on people over processes presents specific challenges during the software development lifecycle (SDLC) [14]. One major issue is the lack of traceability of test cases to other artifacts and source code [14,15,16]. This is exacerbated by inconsistent, incomplete, and inaccurate requirements [18]. Agile methods also lead to inconsistent and inadequate test cases [15], making it difficult for test cases to effectively validate software behaviours [19] and avoid errors [9]. Semantic clarity and consistency in test cases are often lacking, causing misunderstandings due to varying participant knowledge and experiences [9], [20]. Additionally, insufficient tool integration for software test cases hampers seamless testing processes [21]. These challenges highlight the need for improved test case design to ensure highquality software.
According to [22] Model-Based Testing (MBT) strives to automatically create tests (test cases) based on a model that describes specific behaviours of the system being tested (SUT). MBT offers several key motivations, including facilitating automated test case generation, providing comprehensive test coverage, and simplifying defect discovery. According to [23] MBT is becoming more and more recognized in the market as a cost-cutting strategy that automates test case generation, reducing the need for manual test suite creation and improving test cycle efficiency. Additionally, [23] also emphasize MBT's potential to lower costs and enhance test effectiveness. Compared to manual case generation, [25] assert that a complete abstraction test model allows for more comprehensive testing. According to [26], MBT can test a wider range of scenarios compared to record-based testing. Additionally, [25] demonstrate that MBT automation outperforms manual methods in error detection, citing a case study where MBT found significantly more faults than manual techniques.
Despite these benefits, MBT has some drawbacks too, such as the requirement for specialized skills, initial labour intensity, and model complexity. According to [22] testers need to be familiar with state machines, formal languages, and automata theory, as well as tools and scripts for test automation. Furthermore, [27] highlights the significant initial investment and labour needed for MBT, as careful selection of model types and division of software portions are necessary for effective modelling. The complexity of MBT models is underscored by the state-space explosion, which complicates maintenance and presents significant challenges, particularly for beginners [24,26].
Ontologies [27], given their both conceptualization and automation power, can overcome the issues posed by MBT approaches. For example, automated software test case generation is significantly enhanced by combining ontology-based requirement specification with learningbased methods, as proposed by [29]. An Ontology-based framework was proposed by [29] that automates test case generation, execution, and verdict construction using a knowledge-based system and learning-based testing algorithm. Similar approaches are suggested by [30,31]. In test scenario management, the authors in [32,33] discuss the use of ontologies to generate relevant test scenarios for complex systems, emphasizing the need for detailed and specific entity descriptions. An ontology-based approach for test case prioritization was suggested by [34] which deemed particularly useful during retesting after software updates. The importance of domain knowledge and knowledge representation in efficient software testing is emphasized in [35,36], which proposed that ontologies can solve problems such as uneven knowledge representation and focused expertise by creating semantic links between data and knowledge.
These insights collectively advance the field of software testing through improved automation, management, and knowledge sharing. However, pure ontology-based approaches in software test cases face some limitations too. According to [37] the development of ontology-based software test case generation tools is often manual and costly due to the lack of supporting tools. Furthermore, [38] emphasize the need for user-friendly ontology representations that fit the workflow of domain experts, who may not be skilled in ontology development or formal languages. Additionally, [38] highlight that ontologies require input from domain knowledge experts, who may not be familiar with the formal languages or logic needed for ontology development, making the process dependent on both domain and ontology experts.
Using domain-specific modelling language (DSMLs) in software test cases holds the promise to address such non-usability and non-understandability issues raised by ontology-based approaches. According to [38], DSML is a modelling language built for a specific area of discourse, enriching general modelling notions with domain-specific terminology and concepts reconstructed from that domain. Improved communication is one key advantage; DSMLs are expressive and concise, effectively representing concepts and relationships within a specific domain [39,40]. Empirical studies by [41] compare DSMLs and general-purpose languages (GPMLs) (e.g. UML Class Diagram) based on cognitive dimensions outlined by [42] show that DSMLs perform better in areas such as abstraction gradient, consistency, and error-proneness. This would make test cases created with DSMLs easier for stakeholders to understand and validate, ensuring alignment with domain requirements. Additionally, DSMLs promote increased productivity and consistency from the early stages of development, enhancing the quality of models produced [43]. Additionally, [44] also noted that DSMLs are quickly learnable by domain experts, improving the language's applicability.
To take the full advantage of DSMLs and ontologies, [45,46], describe an ontology-based metamodelling approach as being interpretable by both humans and machines. They elaborate by explaining that within the realm of information systems, human interpretability pertains to metamodels, while machine interpretability primarily concerns the formal semantic aspects of models.
To mitigate the requirement for specialized skills in both MBT and ontology-based approaches, the ontology-based meta-modelling approach was extended in this work. Graphics depictions of models are useful for humans, while ontologies make knowledge in models' machine interpretable. The ontology-based metamodeling technique was extended and implemented by [46] with the introduction of AOAME, an Agile and Ontology-Aided Modelling Environment. In this work we extended AOAME to accommodate the new ontology-based DSML for software test cases.
To tackle step 1 of the development process for a DSML, as outlined by [47,58], we collected requirements through semi-structured interviews with at least five experts in software testingsuch as test engineers, test managers, and developers with testing experience. We also reviewed literature on best practices and existing industry standards such as test cases. An excerpt of the requirements, including their descriptions and sources, is presented in Table 1.
This section aims to show "How can ontology-based DSML of software test cases be conceptualized?" and fulfil the suggestion phase of the Design Science Research methodology. The approach follows [47] DSML development process steps by creating a DSML through three steps: creating concrete syntax, creating abstract syntax, and defining language semantics. The abstract syntax, represented by a metamodel, depicts the language concepts and their relationships, while the concrete syntax explains how these concepts are visually and textually represented as domain-specific modelling elements. Language semantics impose structural and features to govern syntax and semantics. Error! Reference source not found. illustrates suggested abstract syntax. The meta model builds on [48] specifications for software test cases in addition to requirements gathered from additional literature and consultations with industry experts as discussed in 3. Test case, Test suite, Test expectation, Test results, and Test inputs are inferred from [48] while Message flow, Sequence flow, Gateways are additional elements added to facilitate the modelling of software test cases. This abstract syntax can be extended to customize the DSML. Table 2 represents the concrete syntax of the DSML and it visually represents the default abstract syntax conceptual elements.
Suggested concrete syntax of ontoST.
Test Suite Represents a set of test cases that will be used to test a particular functionality.
Represents instructions for testers to follow to ensure programs are functioning properly Exclusive Gateway (XOR)
Represents a decision point where only one outgoing path can be taken based on conditions. It's a mutually exclusive choice.
Splits the flow into multiple parallel paths or synchronizes multiple incoming paths. All outgoing paths are taken simultaneously.
Splits the flow into one or more paths based on conditions, and all active paths must be completed before merging.
Represents test case step. This is a step that should be executed and observed for results.
Represents input data to a test step.
Represents the actual result received after the test.
Represents the expectation of a test step after the test.
Represents test step asserts.
Sequence flow Used for connecting Test steps, Gateways and Expectation.
Used for showing input or output message flow from input and result elements.
Used to show the beginning of software test cases.
Error! Reference source not found. shows the three main ontologies of AOAME that were extended to design our software test case DSML. Specifically, the Meta-Model Ontology (MMO) that mirrors the abstract syntax, the Domain Ontology (DO) captures the semantic domain; concepts originating from the MMO are aligned with those from the DO. The Palette Ontology (PO) represents the graphical notations of the language and is directly associated with concepts within the abstract syntax. The PO contains concepts and relations regarding the modelling language's graphical notations, as well as information of how to position the graphical notations on the palette. Thus, the ME palette is supplied by the PO concepts. The MMO contains classes and characteristics that describe a modelling language's abstract syntactic elements, such as modelling elements and modelling relations, as well as the taxonomy and object properties associated with them. MMO consists of one or more modelling languages, either distinct or merged. The DO contains classes and properties that describe the semantic domain.
In this section, we evaluate the utility of the approach in two ways: first, we prove that the models can be created with the new ontology-based DSML ontoST, then we show how the ontology can be used to support the design of adequate software test cases. In both cases, a real-world scenario is considered.
In this section, we will demonstrate the use of ontoST modelling language within the AOAME4STC tool to model test cases created for evaluating the login functionality of a bank website. Error! Reference source not found. shows the current record-based software test case representation used by some test case designers. Error! Reference source not found. shows the model created using ontoST representing the test cases shown in Error! Reference source not found..
In this section we evaluated the test case model created against one of [56] test case specifications constraint that states that every test case must have at least one test step. In Error! Reference source not found. we have 2 test cases in our model where one does not meet the requirement of having a test step. We have manually labelled it as wrong. As seen in Error! Reference source not found., when we run the SPARQL query against the ontology created for the model, we receive an incorrect testcase triple.
This paper demonstrated the suitability of an ontology-based meta-modelling approach for the support of the design of adequate software test cases. For this, a new ontology-based domainspecific modelling language, ontoST, was created. The latter was implemented in the modelling environment AOAME4STC, which was used for the proof of concept. As a future work, we regard important to evaluate the perceived usefulness of the new DSML ontoST with software testers and to continue validating ontoST by modeling additional software test cases.