The recent evolution of NLP has enriched the set of DL-based approaches to include a number of generalpurpose Large Language Models (LLMs). Whereas new models have been proven useful for generic text handling, their applicability to domain-specific NLP tasks still remains doubtful, particularly because of the limited amount of dataset available in certain domains, such as Requirements Engineering. In this study, diferent pre-trained embeddings were tested in three requirements classification tasks, in search of a tradeof between accuracy and computational complexity. The best F1-score results were obtained with BERT (90.36% and 84.23%), with DistilBERT identified as optimal tradeof (90.28% and 82.61%).
of high-quality software which meets the expectations of customers and end-users. Given the
importance of this part of the software development lifecycle, it is necessary to ensure that
requirements are stated clearly, adhere to quality criteria, are appropriately classified, and are free
from errors. While it is possible, and often the norm, to carry out the requirements engineering
processes manually, automated NLP approaches stand to ofer significant improvement to the
process [
This paper seeks to assess the efectiveness of LLMs in accurately classifying requirements into their respective categories. In doing so, this paper considered pre-trained language models and evaluated their ability to create semantic representations from requirements specifications.
The contribution of this work can be summarized as follows: • comparison of the semantic representational power from available pre-trained LLM with application to RE. • an explorative study trying to optimize the tradeof between computational resources and accuracy
The remainder of this paper is organized as follows. Section 2 summarizes the state of the art and related work in NLP tasks for RE. Section 3 describes the experiment design, the research questions, and the evaluation methodology. Results and discussion are reported in Sections 4 and 5 respectively. Finally, conclusions are drawn in Section 6.
Among NLP tasks in RE, requirement classification is one of the most common [
Similar to [
The experiment aimed at comparing diferent pre-trained models for word and sentence
embeddings to verify their suitability for requirements classification. Evaluation was conducted on
the PROMISE-NFR dataset [
The word embeddings used in the experiments were GloVE [
1SBERT was used with the all-MiniLM-L6-v2 model. See https://www.sbert.net/docs/pretrained_models.html As illustrated in Fig. 1, the aim was to train a small size classifier (<10M parameters) on a domain-specific context with limited amount of data, relying on pre-trained language models for semantic representation of words/sentences.
All embeddings were tested under the same conditions, using two MLP structures (≃100k parameters and 2M parameters)2. The MLP models were implemented using dense layers, with ReLU activation function and Stochastic Gradient Descent (SGD) optimizer. Learning rate values of 0.1 and 0.01 (default) were used. The embeddings and the MLP structures were evaluated in three diferent tasks:
Task 1 compares these general-purpose embeddings without an extreme imbalance. Tasks 2 and 3 allow the evaluation to be made on the impact of class imbalance in the more complex cases distinguishing 6 and 12 classes. The 6 classes of Task 2 were chosen as the most frequent classes (comprised of at least 50 sentences and 500 words) as indicated in Table 1. The experiment aimed at answering the following research questions: RQ 1 Which embedding provides better accuracy in requirement classification? RQ 2 Which embedding provides the best trade-of between accuracy and model’s complexity?
The dataset was split into train and test set, 70% and 30% respectively. With the training set further split into train and validation (10%). The splits were done using the stratify options, i.e. preserving class imbalance, and preventing from separating sentences appearing in the same requirement. The evaluation was done using a 5-fold procedure comparing the embeddings with the two MLP structures on the three tasks. Models were trained with early stopping using a patience of 25 epochs and saving only the models with highest accuracy in the validation set. The data imbalance was handled using weighted loss. Finally, a k-Nearest Neighbors (kNN) classifier was used as a baseline. Since kNN makes direct use of embeddings for classification, and because of its non-parametric nature, it is therefore a suitable approach to measure how well the embeddings obtained from pre-trained models could be used directly to classify requirements.
In the experiment, results covering all combinations between the large and small MLP classiifers were calculated on the three tasks. For the sake of conciseness only some combinations are reported. Additional results including confusion matrices are available in the published repository. Table 2 reports results obtained using the large MLP structure on Task 1 and the small MLP architecture on Task 2.
Additional combinations were tested considering the cased and uncased versions of pretrained BERT and DistilBERT models. Table 3 report results obtained with the best performing model on Task 2, the small MLP architecture using the uncased version of BERT. Fig. 2 illustrates the confusion matrix and the normalized confusion matrix obtained in Task 2 using BERT uncased and the small MLP.
Table 4 summarizes results obtained in Task 3, including the baseline results using kNN as a classifier. Table 5 reports precision, recall and f-score values for all classes obtained with the best performing combination on Task 3. Fig. 3 illustrates the normalized confusion matrices obtained with the diferent embeddings on Task 3. Finally, Fig. 4 summarizes the macro-average F1 score obtained with all the embeddings on Task 2 and Task 3. * Macro-average +Weighted - in () are the baseline values obtained using kNN
Precision* MLP kNN
Results on Task 1 highlight BERT and DistilBert as the best performing embeddings (RQ1), however results obtained with the other embeddings are comparable and might be considered for resource constrained cases. In particular, SBERT is the fastest model (except for GloVe) for generating vectors of 384 dimensions, which also reduces the complexity of the final classifier. Similarly, GloVe embeddings are obtained by simple lookup on a dictionary data structure and are 300 dimensions embeddings. GloVe, however, is exposed to out-of-dictionary (OOD) words limiting its working ability in the presence of OOD words. Similarly, in Task 2 and 3, BERT and DistilBERT were the best performing models, with DistilBERT a good candidate to reduce the computational overhead of BERT without causing detrimental efects on the accuracy performance (RQ2). No major diferences were observed when using the small and the large MLP classifiers, possibly due to the limited size of the dataset that does not allow to maximize the benefit of using a classifier with a higher number of trainable parameters. The lack of data is further exacerbated in the case of sentence embeddings with the MLP trained on fewer data points. The comparison with the baseline highlighted how, despite the limited amount of data, training an MLP classifier outperforms the baseline kNN approach of using embeddings to classify new data. The worst performing baseline results were obtained with GloVe, possibly attributable to out-of-dictionary words. MLP classifiers trained on GloVe vectors, however, appear to reduce the gap, leading to results comparable with SBERT and USE.
Construct Validity Standard evaluation metrics were used as macro-averages to prevent majority
classes from masking less represented ones. All mandatory steps of the ECSER pipeline for
evaluating classifiers were performed [
Internal Validity One major factor afecting internal validity is the correctness of the annotation of the dataset that authors have questioned in the past. Nevertheless, it still represents the most widely used dataset for requirements classification, facilitating the comparison with previous work. Since this type of ML tasks typically include a high degree of randomness, 5-fold cross-validation was used to calculate results.
External Validity The dataset includes requirements written by students, which may not be representative of industrial requirements and the evaluation of the language models is limited to the three examined tasks. Diferent results may be obtained when other classification schemes are used, or other types of requirements-related information (e.g., user stories, or app reviews) are adopted. Concerning the coverage of possible pre-trained embeddings, we have considered a representative set of basic and deep learning-based ones, not only limited to those derived from BERT. Therefore, we argue that our analysis can be considered representative of the usage of diferent embeddings for requirements classification. As for the classification algorithm, we use two MLP structures and a kNN as a baseline. Diferent results may be obtained when using other classifiers (e.g., SVM, Naive Bayes).
This paper reported on the evaluation of pre-trained embeddings for RE. Some of the most common embeddings were tested under the same circumstances on a public dataset. Results obtained identify BERT and its smaller variant DistilBERT as the best performing embeddings, with the latter being an optimal tradeof between accuracy and model complexity. GloVE and SBERT despite a slightly lower accuracy were found to be the fastest in prediction time and could be suitable for cases in which time represents a key factor or resource constrained environments. Future work will aim to extend the evaluation on additional datasets to verify the validity of these results on diferent RE tasks and datasets.
This research is supported by the ARC (Advanced Research Engineering Centre) project, funded by PwC3 and Invest Northern Ireland.
3PricewaterhouseCoopers LLP a limited liability partnership incorporated in England with its registered ofice ofice at 1 Embankment Place, London WC2N 6RH