Assuring the quality of complex and highly configurable software systems is a demanding and timeconsuming process. Especially for safety-critical systems, extensive testing based on requirements is necessary. Methods for model-based test automation in agile software development ofer the possibility to overcome these dificulties. However, it is still a major efort to create formal models from functional requirements in natural language on a large scale. In this paper, we present and evaluate automated support for the requirements formalization process to reduce cost and efort. We present a new approach based on Natural Language Processing (NLP) and textual similarity using requirements and product design specifications to generate human- and machine-readable models. The method is evaluated on an industrial use case from the railway domain. The recommended requirement models for the considered propulsion system show an average accuracy of more than 90% and an exact match of the entire models of about 55%. These results show that our approach can support the requirements formalization process, which can be further used for test case generation and execution, as well as for requirements and design verification.
Increasingly complex and highly configurable software systems also increase the efort required
for their quality assurance. The rapid and simultaneously high-quality development of industrial
software products demands an increasingly efective test process. Especially for safety-critical
systems, such as in the automotive and railway domains, extensive testing based on
requirements is necessary. However, any manual processing, such as requirements verification and test
generation, from textual requirements is time-consuming and error-prone, and also requires a
lot of expert knowledge. Methods for model-based test automation in agile software
development pursue the goal to overcome these dificulties [
In previous works, a model-based testing tool chain has been developed to enable eficient test
processes [
Recent advances in natural language processing (NLP) show promising results for supporting
a wide range of requirements analysis tasks [
However, due to the shortcomings of natural language, most of the existing NL-based
approaches generate abstract models and test cases, but lack information about real entities,
their relations and design [
In this work, we want to take into consideration product design specifications for a much more precise requirements formalization process. Our new approach utilizes NLP techniques to automatically generate requirement models from natural language requirements and design specifications. We perform textual similarity and contradiction analysis between the requirements and entity descriptions using classical to modern NLP algorithms. The generated models are represented in a simple, self-created, machine- and human-readable language. The main contributions of this work are i) a new approach for requirement formalization using product design specifications and ii) the evaluation of various algorithms on an industrial use case. The integration of information from the design specifications at this early stage of the testing process shall provide much faster feedback to the requirements or test engineer whether the requirements are verifiable and correctly designed.
There is a wide range of sophisticated methods and tools for requirement formalization, see e.g.
the surveys of Zhao et al. [
There are only a few works that take architecture and design specifications into account during
the formalization of requirements. Bernaerts et al. [
The use of architecture and design specifications has been investigated for various other
requirements analysis tasks. Leitão and Medeiros [
The application of similarity approaches is widely used in requirements analysis. For example,
several techniques for identifying requirement duplicates and interdependencies have been
studied [
However, most previous work has focused on the level of the entire requirement, which has the advantage that it can be processed at once and most NLP techniques are directly applicable. In contrast, our approach goes into much more detail on the individual entities and their dependencies between the requirements and design descriptions. In addition, not only similarity but also contradiction is studied to identify specific properties of the requirement model. To the best of our knowledge, the application of NLP-based methods for automated requirements formalization using requirements and design specifications has not yet been considered in the literature.
In order to conduct experiments and evaluate our approach, we consider a use case from the rail industry. We use the data of the Propulsion Control (PPC) system in Bombardier Transportation, an Alstom Group Company. The PPC is part of a large, complex, safety-critical system. It handles the control of the entire propulsion system, including both control software as well as the electrical functions.
The requirements are written in textual format in the requirements management tool IBM DOORS. They are written in English and may contain several sentences. They do not follow a prescribed format in order not to focus on syntax when writing them. To meet the standards Requirement specification
Requirement + ID + Description + LinkedComponents 1..*
Component 1..* + Name + Description + Parameters + Signals 1 1..*
Design specification
of Safety Integrity Level 2 (SIL2), a design document is written in textual format and is also
handled in DOORS. The requirements and the design document are created by hand and can
be manually linked. The system architecture and software modules are modeled as blocks in
Matlab Simulink using a model-based design approach [
We assume the following tool and use case specific characteristics of the requirements and design specifications, cf. Figure 1: i) components are not mentioned in the textual requirement descriptions but are defined in the design specification and linked to the requirements, ii) all signals and parameters are defined with textual descriptions in the design specification and linked to the corresponding components, and iii) requirement descriptions have a clear and consistent structure. In particular, we assume that a short description in natural language is available for each entity. This assumption should hold true in most industrial product developments since requirements engineers also need to identify entities by some textual description. This information can be written inside the implemented code, in the company’s internal standardization documentation, or more structured as a table in a design specification document.
In the following, we investigate suitable NLP methods to automate the process of creating requirement models from the requirements and design specification. Figure 2 demonstrates the pipeline of our approach using an exemplary requirement with related signal descriptions. Note that the requirement is shown somewhat generalized, as the data we use is confidential.
The process can be divided into the following steps. The requirements in natural language and the design specification serve as inputs to the pipeline. The requirement is initially decomposed into sub-requirement clauses (Section 4.1). Each clause is compared to the signal descriptions in the design specification to identify the signals (Section 4.2) and then the corresponding parameters are identified (Section 4.3). A relation is formed for each clause, which is then translated into a logical structure and finally into a requirement model (Section 4.4).
The ground truth for the individual steps was created manually and reviewed by a requirements engineer with expert knowledge. While presenting the methodology, we show experimental results for the individual steps and discuss which methods are most suitable. CMD_Close_Valve = true EVT_Ex_Hi_Temp = true Device_Temp > T_ExHi Device_Temp < T_RST Requirement Model if (Device_Temp > T_ExHi) then (CMD_Close_Valve = true and
EVT_Ex_Hi_Temp = true) until (Device_Temp < T_RST) Requirement Specification The valve shall be closed and event 'Extremely high device temperature' shall be indicated, when the device temperature exceeds [T_ExHi] ºC until the device temperature falls below the reset value [T_RST] ºC.
Decomposition
Signal Detection
Parameter Detection
Model Generation Design Specification
It is imperative to initially decompose the long, complex industrial requirements into shorter
requirement clauses that describe elementary instructions individually. This process requires
understanding both the structure (syntax) and meaning (semantics) of the requirement text. In
our first trials to decompose the requirements, we used a naive keyword-based approach. The
conjunctions for introducing conditional clauses like if, when, while and until and for connecting
clauses like and and or were identified in the span of the requirement text and were considered
the boundary of clauses. In the considered use case with well structured requirements, there
were only a few cases where this approach did not work. For example, the phrases "between
sensor 1 and 2" or "less than or equal" should not be considered a boundary despite the presence
of a conjunction. To decompose such requirements correctly, we considered the syntactic
dependencies1 of the requirement text (e.g., using the ancestor tokens of the conjunctions) in
addition to the keywords. Fig. 2 illustrates the decomposition of a requirement (by line breaks)
from our use case. Our use case demands only a simple decomposition algorithm, as proposed
here, to work correctly. However, this could be extended to more linguistic patterns as proposed
in [
For the further steps in our pipeline, decomposition into requirement clauses is best suited. A shorter chunk of text (with just the noun phrases) leads to a loss of information and a longer clause (a whole sentence) leads to erroneous detection of signals and parameters.
Once the requirement is decomposed, we compute the cosine similarity between the requirement clause and all the signal descriptions in the design specification. The signal with the highest similarity score is then retrieved as the most relevant. To obtain a vector representation for each requirement clause and signal description, we apply classical techniques including Term Frequency (TF), static word embeddings including GloVe2 and fastText3 and contextual embeddings from Sentence-BERT4 (SBERT). We also use FuzzyWuzzy5, which looks for partial (or inexact) matches between sentence pairs. We also consider ensembles of these methods by taking the average of the individual similarity scores.
The evaluation results for signal detection using a total of 107 clauses and 207 signals from the use case are shown in Table 1. We use TF instead of TF-IDF as we observed that the IDF weights do not change the result significantly (probably due to the shortness of texts). In agreement with this argument, we found that removing stop words deteriorated the similarity scores. However, stemming6 the text before vectorizing with TF gives better results. Pre-trained static embedding models performed hardly well for detecting signals. We observed that these models have dificulty distinguishing between, e.g., "extremely high temperature" and "too high temperature" within the signal description. Though the SBERT model performs very well, there are also some cases where the model could not distinguish between "device on fault" and "device of fault" . This could be attributed to the modality of their training, which aims to bring embeddings of similar words in their representation space as close as possible. As a result, the model gives very high cosine similarity scores for such pairs and occasionally detects an incorrect signal. Though FuzzyWuzzy individually did not perform as well as the other methods, when combined with the relatively better performing TF-stem and SBERT models, the best results were obtained.
3using wiki-news-300d-1M.vec from https://fasttext.cc/docs/en/english-vectors.html 4using paraphrase-distilroberta-base-v1 from https://github.com/UKPLab/sentence-transformers 5https://github.com/seatgeek/fuzzywuzzy 6using Porter stemmer from NLTK, https://www.nltk.org/_modules/nltk/stem/porter.html
After detecting the signals, we need to determine the parameters that are assigned or compared
to the signal values. When parameter names are used within the requirement (e.g., T_ExHi),
they can be easily identified by simple pattern matching in the design specification. In many
cases, however, the parameters are of boolean data type and can only be identified by a semantic
comparison between the two textual statements. In these cases, we need to determine whether
the requirement clause agrees or contradicts with the corresponding signal description to obtain
the parameter. For example, the requirement clause "The valve shall be opened" and the signal
description "close valve .." contradict each other and the parameter is set to false. Conversely, if
the signal description agrees with the requirement clause, the parameter should be detected as
true. Such NLP tasks for detecting contradictions are known to be dificult [
Our first trial was antonym synonym detection, where we checked whether the verb in the
requirement clause falls into the antonym sets in WordNet7 of the verb in the signal description.
Similar to this approach, we used the model of Mixture-of-Experts for Antonym-Synonym
Discrimination (MoE-ASD) [
We also utilize the similarity scores from the signal detection step and identify the parameter as true if the scores are above a threshold (empirically set to 0.6) and false otherwise. This is motivated by the idea that when paraphrasing a sentence into another, one could identify whether it agrees or contradicts with the other sentence.
Since textual inference methods can reliably classify a pair of sentences as either agreement or contradiction, we also resort to sentiment analysis8 to assess the sentiment of the pair and determine the parameter from it. We also tried to infer the entailment relationship using a pretrained model9 trained with 3 classification labels. We considered the softmax probabilities for
8using GLoVe-LSTM from https://demo.allennlp.org/sentiment-analysis/glove-sentiment-analysis 9using ELMo-based Decomposable Attention from https://demo.allennlp.org/textual-entailment/elmo-snli the labels agreement and contradiction and predicted the parameter accordingly. In neutral cases, we employed a negation rule that checks for the presence of words that identify a negation.
Table 2 shows our evaluation results for parameter detection using 85 signals with boolean parameters, of which 67 are actually true and 18 are false. It is evident from the table that textual entailment combined with the negation rule gives the best result.
Once the signals and parameters are identified for each requirement clause, we formulate a relation (of the form Signal-Operator-Parameter) for each of these clauses. We rely on a dictionary-based approach10 to identify comparison operators (if any) in the clauses. In Fig. 2 the relations are illustrated for the exemplary requirement. To generate requirement models from these relations, we introduce a domain-specific language (DSL) with abstract logical blocks. It maps the relations coming from a conditional clause (if /when/while) to an if-block, those coming from main clauses to a then-block and those coming from an until-clause to an until-block. Conjunctions (and/or) identified between relations are also accommodated in these DSL blocks. Though this mapping appears rather trivial, our aim is to make this translation simple and flexible, so that ways are open for integration with other sophisticated languages. The resulting models can also be further transformed into Matlab Simulink models or UML sequence diagrams, depending on what the end user desires.
To evaluate the entire pipeline against the 31 requirements from our use case, we combine all the individual steps from above. We take the requirement texts as input and generate requirement models with the help of the design specification.
The evaluation results are shown in Table 3. The average accuracies are calculated as follows. For each requirement, we calculate the percentage of correctly identified relations. Similarly, we calculate the accuracy of the logic by counting the correctly connected relations with logical conjunctions (and/or) and the correctly assigned relations to the logical blocks (if /then/until). For the accuracy of the model, we count all correctly identified signals, parameters, relations, and logic. Then, the macro-average over all requirements is calculated and shown in Table 3. The exact match counts the percentage of completely correct models. While this is a tough metric, it shows the extent to which the requirements formalization process can be fully automated.
We observe the best results with the two best ensembles for signal detection together with the best method (TE-neg) for parameter detection. This combination yields an average accuracy of 90.7% in evaluating the model across all requirements. The rather low values for exact match of 54.8% reflect the toughness of the metric, which invalidates a requirement model even if a single constituent (mostly one parameter in our case) is misidentified, although the remaining constituents are correctly identified. Some generalized resources of this paper can be found on GitHub11.
10using Roget’s Thesaurus, http://www.roget.org/scripts/hier.php/?class=I&division=0§ion=III. 11https://github.com/ifak-prototypes/nlp_reform
Although our approach shows promising results, it has several limitations. One of the main limitations of our pipeline could be the simple decomposition algorithm that works for our specific requirements. Therefore we referred to more elaborated algorithms when needed. The model language is also very limited and contains only the basic behavioral elements. Finally, we assume that links are provided between artifacts, especially to the components.
In this work, we presented an NLP-based approach for automated requirements formalization using natural language requirements and design specifications. We investigated various NLP methods for the individual steps of our pipeline and evaluated them on an industrial use case from the railway domain. We have shown that for signal detection we obtain the best results using Sentence-BERT combined with FuzzyWuzzy-TSR, and for (boolean) parameter detection using Textual Entailment supported by a negation rule. When evaluating the entire pipeline, we found that the requirement models generated using the combination of the two aforementioned methods yield the highest average accuracy of more than 90% and an exact match of about 55%. These results show that our approach can highly automate the process of requirements formalization, which can support the requirements engineer in e.g. requirements verification and test case generation.
In the future, we plan to integrate this approach into our model-based testing pipeline. Furthermore, there are several features that could improve our methodology, such as handling non-functional properties (like durations), prioritizing methods with lower execution time, domain-specific pre-training and fine-tuning of available models (like SBERT), or learning from corrected predictions of a domain expert (online learning). Another interesting enhancement would be to support the requirements design process and the manual creation of the design specification by providing recommendations for related components and signals in advance.
This research was funded by the ITEA3 project XIVT. The German partner was funded by the BMBF (grant no. 01IS18059E). We thank Bombardier Transportation, an Alstom Group Company, for providing an industrial use case for the evaluation of the presented method.