In requirements engineering (RE), software requirements typically come in the form of unstructured text, written in natural language. Consequently, identifying related requirements becomes a time-consuming task. In this paper, we propose a novel method for measuring semantic relatedness between software requirements, with the aim to develop RE tools that can automate the process of detecting traceability links in requirements documents. The proposed method is underpinned by an embedding-based representation of semantic frames in FrameNet, trained on a large corpus of user requirements. Applying the method to the task of detecting semantically related software requirements, the performance of the proposed method was evaluated against a manually labelled corpus and baseline system. Our method obtained satisfactory performance in terms of F-score (86.36%) against a manually labelled data set of software requirements, and outperformed the baseline system by 24 percentage points. These encouraging results demonstrate the potential of our method to be integrated with RE tools for facilitating software requirement analysis and traceability tasks.
Requirements engineering (RE) is concerned with the elicitation and speci cation of user requirements for the purposes of software development [HJD17]. In the initial phases of RE, requirements are typically written in natural language [FDE+17], examples of which are shown in Figure 1.
Finding related or similar requirements within a document is an essential task in order to understand linkages between them [HJD17]. Considering the four requirements presented in Figure 1, we can say that both Req-1 and Req-2 are related to le restriction, while Req-3 and Req-4 describe the necessity to update the bank's client data. However, it is a challenging RE task to analyse requirements to nd relations between them, either explicit or implicit, because requirements documents are often very long, and written inconsistently by di erent stakeholders [HJD17]. This problem is due to the semantic ambiguity and incompleteness inherent to natural language [FDE+17]. Moreover, manually analysing requirements is a time-consuming and error-prone procedure [HJD17, GF94, MNX12].
In this paper, detecting semantic relatedness between requirements is de ned as the process of nding semantically connected events (e.g., \ The user shall use the software to send les." and \The user shall receive documents as well") whereas detecting requirements similarity is de ned as the process of nding requirements that discuss similar events (e.g., \ The user shall use the software to send les." and \The user shall subCopyright c 2019 by the paper's authors. Copying permitted for private and academic purposes. mit les via the system."). Detecting such relatedness and similarity in requirements thus requires semantic processing [MNX12].
For instance, semantic relatedness has drawn the attention of RE researchers with the goal of automating the creation of traceability links within requirement documents [MNX12, MN15]. In addition, semantic relatedness constitutes a more intuitive method for tracing requirements, since it provides a holistic view and mimics the human mental model in nding similar or related concepts [MN15]. Tracing such relatedness requires representing the natural language|hence, unstructured|requirements, in a more structured manner. Afterwards, the structured requirements can be analysed for semantic relatedness between them.
Techniques in natural language processing (NLP) o er a viable solution to many tasks in RE [DFFP18], including requirements analysis and traceability [MNX12, MN15]. However, majority of the available NLP techniques and resources are not domain-speci c, i.e., they are trained or built on general-domain text (e.g., news articles) [DFFP18]. For this reason, a recent research direction in RE has called for \customizing general NLP techniques to make them applicable for solving the problems requirements engineers face in their daily practice" [DFFP18].
One of the recent trends in NLP research is the use of word embeddings, which are a semantic representation of a word that captures the context in which it has been used, within a corpus of documents [MCCD13]. They are learned based on the principle of distributional semantics, which posits that words occurring in the same context have similar or related meanings [Har54]. Deep learning o ers a framework for representing word context as real-valued vectors, that goes beyond the counting of co-occurrences and takes into account word order as well [BDVJ03].
In this paper, we present and demonstrate a method for measuring semantic relatedness between software requirements, which is underpinned by embeddings representing semantic frames in FrameNet [FB01, Bak17], trained on a domain-speci c corpus using deep learning techniques. In our previous work, we constructed FNRE, a corpus of software descriptions enriched with semantic frame annotations [ABNZ18a, ABNZ18b]. We also proposed a method for detecting implicit relations between semantic frames, through the measurement of similarity between them [AZBN18]. Speci cally, we made use of pre-trained word embedding vectors [ECS18] as a means for representing semantic frames, and then computed their similarity, i.e., the cosine distance between any two FrameNet frames X and Y annotated in the FN-RE corpus. Our initial validation, as reported in [AZBN18], yielded an F-score of 83% against human . In this paper, we extend our previous work to demonstrate the measurement of relatedness|based on semantic frames|at the requirement statement level.
The rest of this paper is organised as follows: Section 2 presents a brief background on FrameNet semantic frames. Section 3 discusses our proposed method for measuring semantic relatedness between software requirements. In Section 4, we explain our evaluation procedures, and then we report results relative to a manually labelled data set and a baseline system. Finally, we conclude and brie y discuss our ongoing work in Section 5. 2
A semantic frame is a coherent structured representation that sums up the meanings given by a statement. According to Fillmore's theory, a semantic frame allows for full understanding of the meaning of a word (e.g., an action signi ed by a verb) by providing access to related essential knowledge (e.g., participants involved in the action such as the \who" and \what") [Fil76].
FrameNet is a computational lexicon1 implemented based on the semantic frames theory [BFL98, Bak17]. The lexicon, accessible via a web-based interface, holds more than 1,200 semantic frames. For every semantic frame in FrameNet, the following information is given: the frame title, de nition, a list of frame elements and lexical units (LUs). LUs are words that evoke the frame, represented as a combination of their lemmatised form and part-of-speech (POS) tag. For example, the concept of creation which is included in FrameNet as a semantic frame entitled Creating, can be evoked by the LUs create.v and generate.v (where v stands for verb). Its frame elements specify the participants involved, namely, the Creator, Created entity and Cause, among many others.
The meaning of a given piece of text can be represented in a more structured and explicit form, by annotating the semantic frames it contains. Figure 2 shows examples of requirement statements which have been annotated based on semantic frame information in FrameNet. Each LU evokes a semantic frame that conveys a speci c concept. Words such as \authorised" and \restricted", for instance, evoke the Deny or grant permission frame in FrameNet, while words such as \con rm", and \verify" evoke the Veri cation frame. It can be observed that although Req-1 and Req-2 both pertain to le restriction, they contain words that are lexically di erent, e.g., \authorised" vs. \restricted", \records" vs. \logs", \kept" vs. \stored". However, these words evoke the same semantic frames, e.g., \kept" and \stored" both evoke the Storing frame.
Tracing the similarity or relatedness between requirements, e.g., between Req-1 and Req-2, or between Req-3 and Req-4, is a non-trivial task. Apart from lexical variation, the syntax structures of these software statements are not identical, thus making the use of heuristic techniques (e.g., word matching) unsuitable. To overcome such issues, there is a need to formulate a method that represents the under-speci ed meanings of such requirements in a structured form, which can then be used as a basis for automatically measuring relatedness. 3
In this section, we present our method for detecting semantic relatedness between software requirements. That is, for any two requirement statements, we measured the semantic relatedness between them on the basis of the FrameNet semantic frames that their lexical units evoke. 3.1
Whereas our previous work [AZBN18] employed pre-trained word embeddings learned on a corpus of Stack Over ow messages [ECS18], in this work we decided to train our own embeddings on a corpus of documents that are more similar to software requirements, i.e., a collection of user reviews of mobile applications. Using the web-based AppFollow tool2, reviews from di erent mobile application repositories (e.g., Apple Store and Google Play) were retrieved. The user reviews covered di erent categories of mobile applications, i.e., business, sports, 1https://framenet.icsi.berkeley.edu 2https://appfollow.io health, travel, technology, security, games, music, photos, videos, shopping, lifestyle, books, social networking, nance. While each review came with metadata such as review date, title and per-user application rating, we took into consideration only the textual content of the reviews. This resulted in a total of 3,019,385 unique reviews/documents in our training data set.
The documents in the training data set were then preprocessed with the following steps: sentence splitting, tokenisation, stop-word removal, part-of-speech (POS) tagging and lemmatisation. The preprocessing results allowed us to automatically check for the occurrence of LUs (associated with semantic frames) catalogued in FrameNet, in order to assess the data set's coverage of semantic frames. Based on this, we were able to determine that our mobile application reviews data set covers all of the 123 semantic frames annotated in the FN-RE corpus3. 3.2
Utilising the preprocessed mobile application reviews data set as a corpus, we trained word embeddings using the continuous bag-of-words (CBOW) learning method of Word2Vec [MCCD13]. A word embedding vector was trained for each LU, which was represented as a combination of its lemmatised form and POS tag. Taking into account the POS tag of an LU makes it possible to train di erent vectors for words with the same lemma but di erent parts of speech. It is preferable, for example, to train a vector for \form" as a verb (form.v ) that is di erent from the vector for \form" as a noun (form.n). The size of each vector was set to 300, following previously reported work [SP18, MCCD13]. 3.3
The word embedding vectors resulting from the previous step were then used to form an embedding-based representation of semantic frames, i.e., frame embeddings. That is, for any given semantic frame F, we collected the vectors corresponding to the LUs that evoke it. The average of these LU vectors is then computed and taken as the frame embedding for F. For instance, as 11 LUs are associated with the Creating frame in FrameNet, a vector containing the average over the 11 word embedding vectors corresponding to these LUs was obtained as part of this step. 3.4
The generated frame embeddings were employed in computing relatedness between semantic frames. Following our method described in [AZBN18], we used the cosine similarity metric. For FrameNet frames X and Y, let R(X,Y) denote the relatedness between these two frames:
R(X; Y ) = cos(X; Y ) =
FX
FY kFXkkFYk where FX and FY are the frame embedding vectors for X and Y, respectively.
The cosine similarity metric measures the angle between two vectors (i.e., frame embeddings). If the vectors are close to parallel (e.g., with R(X,Y) 1) then we consider the frames as similar, whereas if the vectors are orthogonal (i.e., with R(X,Y) 0), then we can say that the frames are not related. 3.5
A requirement statement may evoke one or more frames in FrameNet. To measure the semantic relatedness between any two requirement statements A and B (i.e., a requirement pair), we generated a frame-to-frame similarity matrix. The rows of this matrix correspond to the frames evoked in statement A while its columns correspond to the frames evoked in the other statement B. Its cells hold the cosine similarity between two frames FA and FB, where FA is a frame evoked in statement A while FB is a frame evoked in the other statement B. Given that statement A contains m frames while statement B contains n frames:
A = (FA1; FA2; : : : ; FAm) B = (FB1; FB2; : : : ; FBn) (1) (2)
The values of the cells of the n m matrix M were determined based on the following equation, where i is the i th frame of statement A and j is the j th frame of statement B, and R is a relatedness score based on cosine similarity, as described above:
Mi;j = R(FAi; FBj ) (3)
With the cells of matrix M populated in the manner just described, we took the average over all of its cells. This resulted in a real-valued score that corresponds to the semantic relatedness between statement A and statement B. 4
For evaluating the proposed method for determining semantic relatedness between requirement statement pairs| which we will henceforth refer to as the F2F (or frame-to-frame) method|we used a data set containing human judgements (or labels) as a gold standard. Furthermore, for comparison, we implemented a baseline system underpinned only by word embeddings. 4.1
We constructed a new data set, SEM-REQ, in which pairs of requirement statements have been manually assigned labels indicating semantic relatedness. This was drawn from the FN-RE corpus which consists of 18 documents, containing a total of 220 requirement statements. At least three requirement statements were randomly selected from each document in FN-RE, where each statement evokes at least one FrameNet semantic frame. This resulted in a set of 60 requirement statements. Each of these was paired with every other statement in the set, generating a total of 1,770 requirements pairs which were all included in SEM-REQ.
We devised a simple scheme for annotating each pair, whereby \Yes" was assigned to a pair if its two requirements are semantically related, and `No" otherwise. Each pair in SEM-REQ was manually annotated by three annotators: Annotator A, a requirements engineer; Annotator B, an expert in NLP; and Annotator C, the lead author of this paper who is a PhD candidate investigating the use of NLP techniques in RE. Working independently, the annotators carried out the labelling task over a period of six weeks.
In order to assess the consistency of annotations between our annotators, inter-annotator agreement (IAA) was calculated based on the F-score, i.e., the harmonic mean of precision (P) and recall (R). In this case, F-score was computed for each of Annotators A and B, treating Annotator C's annotations as gold standard. Here, true positives are annotations from each of Annotator A and B that overlap with those of Annotator C, while false negatives are those which were missed. The false positives, meanwhile, are comprised of the annotations from each of Annotator A and B which are not in Annotator C's annotations. We obtained an F-score of 83% for Annotator A and 72% for Annotator B. These results indicate that there is a more than satisfactory level of consistency between our annotators, implying that the semantically related requirements that they annotated, can be considered as reliable.
In producing the nal set of annotations, we harmonised the labels provided by our three annotators to resolve any disagreements between them. That is, Annotator C revisited every discrepancy, e.g., a requirement pair for which Annotator A has a di erent opinion from Annotator B, or Annotator C assigned a label that di ers from that of Annotators A and B. In cases where she was convinced that Annotator A's (or B's) perspective is more correct, she modi ed the labels assigned to requirement pairs. Annotations resulting from this harmonisation procedure formed the basis of the nal SEM-REQ data set. 4.2
To evaluate the proposed F2F method further we implemented a baseline system underpinned by pre-trained word embeddings, i.e., Google's Word2Vec model4 which was trained on approximately 100 billion words from a corpus of Google News articles.
Given a pair of requirement statements, the text of each statement was rstly preprocessed through tokenisation, stemming and stop-word removal. The pre-trained word embedding vector for each token in a statement is then retrieved, resulting in n vectors (corresponding to n unique tokens in a statement). We note that we simply ignored out-of-vocabulary words, i.e., tokens for which the word embedding vectors do not exist in the 4https://code.google.com/archive/p/word2vec/
Word2Vec model. The vectors retrieved for each requirement statement are then averaged, resulting in a vector representation of the statement.
Finally, the semantic relatedness between a pair of requirement statements was determined by computing the cosine similarity between their vector representations. As with our proposed method, the above-mentioned steps were implemented using the Gensim, NLTK, and numpy Python packages. 4.3
We compared the F2F method with the baseline system by applying each of them to the same 1,770 pairs of requirement statements extracted from the SEM-REQ data set. The result obtained, a score s from each of the approaches, corresponds to the average of cosine similarities over the represented embeddings between requirement statements in a pair, where 0 s 1. To help us determine what values of s can be considered as indicative of semantic relatedness, we experimented with di erent values of a minimum score or threshold t, whereby we say that the two statements in a pair are semantically related if s t, and not semantically related otherwise.
For each value of t that was explored, ranging from 0.1 to 0.9 in increments of 0.1, a round of performance comparison was conducted based on the F-scores, obtained by each of the F2F method and the baseline system against the manual labels in SEM-REQ. The results of this comparison are shown in Figure 3. It can be observed that while the F2F method obtained an F-score as high as 86.36% (with t = 0:4), the best performance obtained by the baseline system is only 62.42% (with t = 0:1). It is also worth noting that the baseline system is unable to detect semantically related statements unless only very low cosine similarity scores, i.e., those between 0.1 and 0.3, are taken into account. This implies that the cosine similarity scores obtained using the baseline system contradict the intuition that scores closer to 0 indicate lower semantic relatedness.
Examples of results obtained by the F2F method are shown in Figure 4. Although the statements in any of the ve pairs shown are lexically di erent, our proposed F2F method detected them as semantically related, consistent with the judegment of human experts on the same pairs. In the rst pair, for example, both statements explain a process for registering a new user (or customer) with di erent software systems. However, personal details (e.g., name and address of the user) required by the registration process were speci ed in Statement B, which is not the case in Statement A.
By detecting semantically related requirement statements, the F2F method can help in tracing connected events that a system is involved in. The statements of the third pair in Figure 4, for instance, discuss two events involving a cash withdrawal machine (the user specifying his desired amount, and the machine con rming that said amount can be withdrawn), with the latter having a dependency on the former. Tracing such relations within a requirement document is desirable, as it can aid, for example, in identifying dependency links between system features.
The obtained relatedness scores as presented in Figure 4 indicate various grades of semantic relatedness. For example, in the third requirement pair (which has the highest relatedness score of 58.18% among the examples shown), the statements refer to similar (e.g., Removing in the withdrawal sense) and connected events, e.g., Veri cation and Indicating (the user's bank account), where the frames share LUs. Meanwhile, in the fourth requirement pair (which has the lowest relatedness score of 40.07% among the examples), the frames in the two statements are not similar although they are semantically connected (i.e., Deny or grant permission and Preventing or letting ). Since these frames share contextual information in the trained frame embeddings, the statements in the requirement pair were correctly detected as semantically related. This indicates that, as desired, our proposed method accounts for contextual similarity in detecting semantic relatedness between requirement statements.
As shown in previous work, semantic frames are a promising means for capturing the meaning of requirements (e.g., [JM17]). Our encouraging results demonstrate that with careful selection of a threshold or minimum similarity score (for determining semantic relatedness), our proposed F2F method|which combines the strengths of semantic frames and embedding-based representations|can be applied to RE tasks such as software requirements analysis and traceability. 5
We presented a novel method for measuring semantic relatedness between software requirements, with a view to enhancing their traceability in requirements documents. The proposed method is based on the development of an embedding-based representation of semantic frames in FrameNet (i.e., frame embeddings), trained on a large corpus of user requirements, consisting of more than three million mobile application reviews. The performance of the method is encouraging as indicated by very good results (i.e., an F-score of 86.36%), outperforming a baseline system by 24 percentage points. In our immediate future work, we shall integrate this method with an RE tool for analysing and tracing semantic relatedness of software requirements. This in return, will aid in organising and grouping related system features described in requirements documents.
Jennifer Sikos and Sebastian Pado. Using embeddings to compare framenet frames across languages. In Proceedings of the First Workshop on Linguistic Resources for Natural Language Processing, pages 91{101, 2018.