[Context and Motivation] Acceptance criteria are a popular textual notation, especially in connection with user stories. Typically, they express refined requirements and are used to derive test cases, sometimes automatically. [Question/Problem] While acceptance criteria are very useful, they are not always documented or not always documented well. [Principle ideas/results] In order to find indications for more acceptance criteria (semi-) automatically for a given set of user stories and acceptance criteria we want to use information that is available in user stories and acceptance criteria or public knowledge. [Contribution] This proposal discusses the research problem of improving the completeness of acceptance criteria. It then introduces potential solutions to that problem that only use available information. It also discusses the used research method, plan and progress so far as well as related work to the research.
without extra knowledge of project experts. This ensures that our approach can also be used by novice project members. We also do not want to use information from diagrams or other project documentation as it may vary from project to project which documentation and diagrams are used. Given a dataset with US and some AC we therefore only want to use the text of the US and the text of the AC and public information to find more AC for US in the dataset by using natural language processing (NLP) or machine learning (ML), especially deep learning (DL) algorithms (as defined in [25]).
The remainder of the paper is structured as follows: Section 2 describes the research method used to find solutions for our goal. Section 3 describes our proposed solution and our available datasets in more detail. Section 4 describes related work. Section 5 describes the research plan and current progress.
We use the design science research method [24]. With this method we want to achieve the following design goal: Design and evaluate an approach to improve the completeness of AC for US by using only available information. The first step of the research method is to analyze the as-is-state in research and practice. That is the problem investigation. Then, solution ideas are implemented based on the as-isstate in a software prototype as treatment design. Finally, treatment validation is performed to get empirical evidence that the solution ideas and the prototype solve the problems.
Our problem investigation consists of three parts. 1. An interview study with practitioners from industry on the creation and usage of AC. 2. The analysis of datasets from industry with US and AC. 3. A systematic mapping study conducted by the guidelines of Kitchenham and Charters [13] to find suitable algorithms that could solve our problem.
We already performed the first step with the following research question:
RQ1: What is the state of practice regarding documentation and usage of acceptance criteria and are there improvement ideas?
For the second step, we used the following research question: RQ2: What is the quality of acceptance criteria in practice and how are they documented? After the first two steps of the problem investigation we could narrow down our problem and define our research goal as the goal described in the first section. We also did some first searches for literature that describes research on how to generate more AC for US or other requirements, but only found two approaches described in section 4 that don’t use US as requirements notation. Therefore, we want to focus our systematic mapping study on possible solution ideas with the following question:
RQ3: What is the state of the art in extracting and comparing information from user stories and other requirements texts with NLP, ML and DL algorithms?
To answer this question we first will view the results of existing literature surveys [20], [25] and consider if they are relevant to answer the question. Then we will do forward snowballing on the relevant papers to see if there is more recent research that was not covered in the existing literature surveys. To be sure that there is nothing left out search terms can be defined from the found results and used in the typical search locations like ACM, IEEE, etc. A possible search term could be the following: (“requirement*” OR “user story” OR “user stories”) AND (“NLP” OR “natural language processing” OR “machine learning” OR “deep learning”) AND (“extract*” OR “compare” OR “similar*”) AND (“information” OR “concept”).
The first step of treatment design is to derive solution ideas from the as-is state. For this, the results of the systematic mapping study will be evaluated how they could be used to solve our problem. We will define criteria and select the approaches to implement. They will be implemented as algorithms in a prototype software. In the treatment validation, the algorithms will be evaluated on data sets with US and AC in natural language that we acquired from industry. Those data sets are described in the next section. We are also looking for suitable open source data, that can be used to evaluate our solution as those could be shared with the research community. The solution will be evaluated in the lab based on metrics like precision and recall. Our goal is to reach high recall (~0.8) with reasonable precision (~0.5).
The contribution of the thesis will be the interviews with practitioners on AC, coded datasets, a mapping study on algorithms for information extraction and comparison and the prototypical implementation of algorithms that can help to find more AC.
The general idea of our proposed solution is to apply NLP, ML or DL algorithms to available
information in order to suggest indications for additional AC. The information that is available are the
texts of the US, the texts of the AC and publicly available information (e.g. lexical databases like
WordNet [18], pre-trained classifiers e.g. [
We want to use the information in the following ways to find indications for additional AC (information like lexical databases or pre-trained classifiers might be used in all approaches):
The first approach is to compare the information from the US with the information from its AC to find information in the US that is not covered by its AC. The uncovered information will be presented to the requirements engineer and can give hints for additional AC. In this approach the used information will be concepts, i.e. verbs, subjects and objects of the US and AC. For the US, we only want to use the concepts of the goal part as this part is the most important part for the described functionality of the software that should be covered in AC. The role is not important in AC, as AC focus on the software functionality, not the user. The reason part describes the rationale of the goal and does not give further insights into the needed functionality. This first approach is only able to find additional AC for information that is present in the US. Therefore, it can only find missing functional AC, but no AC that concern quality criteria or other constraints that are not mentioned in the US.
Typically, AC contain more details than the US, as they should refine the US and answer questions wrt functional behavior, quality characteristics, constraints, etc. [23] (see also the example giving in the beginning). Thus, we want to use information from other US and AC to get indications for these details. Furthermore, we assume that similar US have similar AC. Datasets we acquired from industry also indicate that similar US have similar AC, but this still needs to be shown empirically. Therefore, the second approach is to find similar US for one or more given US in a given set of US and show the requirements engineer the similar US together with their AC. These AC could either be used directly or they can give ideas for new AC for the given US. The definition for similarity we use in this approach is the following: Two US are similar if they either refer to the same feature or workspace (that means group of related functions and data) or if they have a similar action. But it might be necessary to change this definition for similar US if our empirical analysis of the similarity of their AC indicates that.
For the third approach, we want to explore if there are publicly available sources of information, other than the US or AC, that we can use to find indications for additional AC. Possible ideas are to utilize forums or platforms like stack overflow, but it is not yet clear, which information from these sources can be used for our goal and how they could be used.
We also need to consider if it is possible and helpful to combine the first approach with the other approaches. E.g. if a missing concept is identified it could be used to filter the results of the other approaches to find a fitting AC.
To find suitable algorithms for these approaches we will use the systematic mapping study. Therefore, the novelty of our solution is not necessarily in the algorithms we use but in the problem context that we apply them and the insights we get on the suitability of these algorithms in this context.
To be able to evaluate our approaches we acquired two datasets from industry that contain US and AC. The characteristics of the datasets are described in Table 1. As can be seen, only a fraction of the total requirements issues contained a US with AC that we could use. The datasets originally contained 308 issues for P1 and 2462 issues for P2. From those only 142 issues for P1 and 711 for P2 contained a US or AC. Of those, only 94 issues for P1 and 97 for P2 contained both AC and US. The US have 115 AC in P1 with an average of 4.03 AC. In P2 the US have 1-6 AC with an average of 2.21 AC. The final size of the datasets is the number of requirements after removing requirements of bad quality (e.g. when the US was divided by a bullet list or when the user’s role mentioned in the US was another software). For P1 this is 86 US and 358 AC, for P2 74 US and 157 AC. Both data sets contain requirements in German language that we automatically translated into English, which we checked and corrected if needed.
We are also looking for further data sets that are publicly available to be able to share our results with the research community. We did a first search in GitHub and found it difficult to identify projects which use US and AC in natural text, not in Gherkin notation, with the available search possibilities.
#AC per US
Final #US, #AC 1-15 (mean 4.03) 1-6 (mean 2.21) 86 US and 358 AC 74 US and 157 AC
This section describes and discusses work related to our goal and to our solution ideas. There are
approaches to generate AC automatically from requirements models via test models [
We conducted first searches in literature to find algorithms for experiments with our solution ideas.
In this first literature research, we found the following algorithms for concept extraction and
comparison. This will be complemented by the mapping study. In [
In the first literature research, we found the following algorithms for similarity checks [
Barbosa et al. [
Kochbati et al. [14] aim to group user stories of a complex software project according to similarity. In order to find similar user stories, first the word-level similarity is determined. They use word2vec, which is implemented as a pre-trained neural network and produces word embeddings. This means words are reflected by vectors. In the vector space, vectors representing similar meanings are close to each other. As a similarity measure of the word vectors, the cosine similarity is used. For the extension to requirement-level similarity, the Mihalcea scoring formula (presented by Mihalcea et al. [16]) is applied. The inverse document frequency is also an ingredient of this formula. The scoring formula is applied to all user stories and the final output is an N x N similarity matrix, where N is the number of user stories.
There are different goals for the calculation of similarity, but, to the best of our knowledge, the goal to reuse AC has not been proposed yet.
Our research plan consists of the following steps 1. Conduct interviews with practitioners from industry 2. Analyze datasets with US and AC regarding quality criteria 3. Come up with solution ideas based on 1. and 2. 4. Conduct first experiments with algorithms for solution ideas a. Ground Truth creation b. Algorithm implementation c. Evaluation 5. Conduct systematic mapping study on NLP, ML and DL algorithms for solution idea 6. Choose algorithms from systematic mapping study to implement by defining and applying criteria 7. Implement chosen algorithms 8. Evaluate chosen algorithms.
We already did steps 1-4. In the first step we conducted interviews with 7 practitioners from industry. Those interviews were held online and lasted about 90 minutes each. In these interviews we asked questions on how they use and create AC, if there are any problems and if they have improvement ideas. The main results were that AC are a valuable tool to get a clear understanding of the software and that it is problematic if they are missing. One of the improvement ideas was to suggest the AC of similar US to be able to reuse them.
In the second step we analyzed the two datasets that we acquired from industry regarding quality criteria for US and AC. We found defects in all quality criteria, but as there are already tools that can find quality defects like understandability and consistency, we focused on the quality criteria completeness for further research. In the third step we came up with the solution ideas described in section 3.
Currently we are conducting experiments with algorithms for our solution ideas. First experiments
are already done. For the first approach we used the concept extraction capability of Stanford NLP to
extract the concepts similar to [
With our available datasets and our ground truth the algorithms for the second approach either have high precision with low recall or high recall with low precision, which is similar to some algorithms we found in literature. The algorithm for the first approach has low recall with medium to high precision, which is mostly due to the fact, that only few concepts could be extracted with the chosen algorithm. Therefore, we want to explore ways how those algorithms can be improved. On the one hand, we want to try other algorithms that we find in literature. On the other hand, we want to improve the combination of algorithms.
Our next experiment will be to utilize pre-trained models like USE [
I thank my advisor Barbara Paech for her valuable feedback and support.