This work aims to combine the efforts of NLP [1], GQM [2] and AHP [3] approaches for assessing overall quality of requirement documents in automated way. Techniques to process the words from the document were applied that enable the system to carry out further analysis on the syntactic and semantic structure of the text. After processing, each requirement statement and the overall requirement are assigned to numeric values based on calculations carried out by the system to determine what areas of the requirement document need modification. The ultimate goal of this work was encapsulating the best of these techniques and methods for measurement requirement quality into a single model and provide a prototype of a tool for automated validation of real-world requirements against it.
The Goal-Question-Metric method based on a system of questions and straightforward answers about properties evaluation [4]. This approach consists of three main steps: specifying goals, pointing relevant attributes, and providing measurements. GQM framework helped to define appropriate metrics and estimate the quality of requirements in our case. The goal should be defined for an object, with a purpose, from a perspective, in an environment. The overall goal of current the project is to measure quality of requirements and can be formulated by following template:
Analyze requirement quality for the purpose of improving with respect to quality attributes from the viewpoint of project managers in the context of product development.
In addition, we identified several sub-goals, which should be fulfilled to achieve the primary goal. For instance: Sub-goal: Analyze requirement unambiguity for the purpose of improving with respect to quality attributes from the viewpoint of project managers in the context of product development.
Question: How many vague words and weak phrases make requirement ambiguous? Metric: Number of ambiguous words in 1 requirement divided by an average number of words in 1 requirement.
Our model adopted the five core quality attributes to give final quality measurement for the whole requirement set evaluating by syntax and semantic analysis.
Unambiguity. It requires that only one semantic interpretation of the requirement exists. To evaluate the ambiguity of each requirement, we propose to use dictionaries with a set of words, which indicates ambiguity in the requirement [6][7]. As the metric for assessing ambiguity, we used the following formula:
Where Nambgthe number of words in the requirement, Ntotalthe number of ambiguous words in the requirement.
Singularity. Statement of the requirement must relate to only one unique requirement that does not overlap with others. The presence of several modal words tells us that the requirement contains several meanings and that the statement does not have the characteristic of singularity. These words may include could, may, might, can, should, will, shall, must, would, etc. The number of connective words may also indicate the presence of several requirements within one (mentioned above). As the metric for assessing singularity, we used the following formula:
where Ntotalthe number of words in the requirement, Nmodalthe number of modal verbs which are not zero, Nconnectivethe number of connective words in the requirement.
Readability. This attribute indicates how easily requirement text can be read and understood, it can be based on the number of syllables per word and number of words per sentence. It can be calculated by Flesch-Kincaid Grade Level [8],
Coleman-Liau Grade Level [9], and Smog Grade [10]. We chose the second one:
where Laverage number of letters per 100 words, Saverage number of sentences per 100 words. If CLI is around 10, text is easy to read, but if CLI > 15 text is too difficult for understanding. We made a mapping into percentage interpretation (if CLI index is more than 17.5, than readability is 0%) by following formula:
Completeness. It requires that the requirement contain all necessary elements, includ-ing constraints and conditions, to enable the requirement to be implemented [18]. We calculated completeness quality attribute by this formula:
where Ntotalthe number of elements in the structural template, Nfilledthe number of elements form templated that were identified in requirement sentence.
NLP is considered a branch of Artificial Intelligence that is concerned with the analysis and interpretation of natural language or human language via several techniques such as Parsing, Part of Speech Tagging, Named Entity Recognition, Tokenization, Sentiment Analysis, etc. NLP system is asked to make unambiguous decisions about word meaning, category, syntactic structure, and semantic scope [5]. In software engineering, requirements can be seen as a set of sentences written in a specific language, and as any text data requirements may suffer from ambiguity. That's why NLP is handy to extract meaning and insight from requirements and, in our case, to get know how good requirements to a set of quality attributes.
One of approaches that can help us in analyzing the priority of quality attributes is Analytical Hierarchy Process (AHP). In this case, there are 5 attributes used to analyze the requirement. Then we ask our customer to fill this questionnaire about their priority: From this table, for example, in the third row we got that unambiguity is 3 levels more important than unsubjectivity and Unambiguity and completeness are in same level of importancy.
After that we calculated pairwise matrix, where the score from questionnaire is provided and π ππ = 1/π ππ and π ππ = 1. From the average above, we got the weight of each attribute. In this case, the prioritization order is completeness, unambiguity, readability, unsubjectivity, and singularity. Requirements can be of any type expressed in the text form: functional, non-functional, use-cases. The prototype is able to perform several functions:
β’ Integration with project management system to gather textual requirements from it (via API)
β’ Perform syntax and semantic analysis of said requirements (supporting Russian language [11][12]) The core of the prototype is the Requirement Quality Model, which contains a consistent set of requirements quality metrics and is expressed in algorithms on how to measure these metrics and how to draw conclusions (average quality of a requirement/set of requirements). The prototype provides a requirement engineer with a graphical user interface or command-line interface to obtain the results of requirements measurement. For NLP were used custom alternative Python libraries Wordnet [13] and Spacy [14] with Russian language support.
We proposed the model for process of quality assessment was based on NLP tools. Different quality attributes were analyzed and adopted. We developed a prototype that capable of reducing the challenges development team face with interpreting requirements due to ambiguity, subjectivity, poor readability or incompleteness. Suggested approach was tested on sample of requirements text. Quality metrics for different attributes were calculated according to customer's priorities for every require-ment and for overall document. This prototype can be further improved by exploring other NLP techniques to furnish users with a detailed explanation of why requirements lack quality attributes.
We thank the organizers of CASE in Tools International Hackathon: Andrey Sadovykh, Alexandr Naumchev, and every single person who contributed to the success of this event. We are immensely grateful to Giancarlo Succi for comments on an earlier version of the proposed model. This research would not have been conducted without efforts of Konstantin Valeev, the challenge owner, who shed more light on grey areas of this project and provided us with enough resources. We are also indebted to appreciate Rostelecom IT company for the opportunity to work on industry-related challenge.se a third level heading for the acknowledgements