Recently, Large Language Models like ChatGPT have demonstrated remarkable proficiency in various Natural Language Processing tasks. Their application in Requirements Engineering, especially in requirements classification, has gained increasing interest. This paper report an extensive empirical evaluation of two ChatGPT models, specifically gpt-3.5-turbo, and gpt-4 in both zero-shot and few-shot settings for requirements classification. The question arises as to how these models compare to traditional classification methods, specifically Support Vector Machine and Long Short-Term Memory. Based on five diferent datasets, our results show that there is no single best technique for all types of requirement classes. Interestingly, the few-shot setting has been found to be beneficial primarily in scenarios where zero-shot results are significantly low.
• RQ1: What is the best technique for requirements classification between SVM, LSTM and
ChatGPT? • RQ2: What are the performance diferences between GPT-4 and 3.5? • RQ3: What are the performance diferences between Zero-Shot and Few-Shot settings? The rest of the paper is organized as follows: Section II outlines the methodology, providing insights into our research approach. Section III presents the analysis of our experimental results. Section IV comprises the threats to validity of our research, while Section VI presents the conclusion, synthesizing our findings and suggesting future directions in the field.
In our methodology, we follow the modeling paradigm of Li et al. [
Acknowledging the possibility that a requirement may encompass both functional and quality aspects, Dalpiaz et al. subsequently formulated four distinct binary classification problems: • IsFunctional: does a requirement possess functional aspects? • IsQuality: does a requirement possess quality aspects? • OnlyFunctional: does a requirement possess only functional aspects? • OnlyQuality: does a requirement possess only quality aspects?
While our initial goal was to utilize the exact same datasets as the ones curated by Dalpiaz et al., we
had access to only five public requirements datasets out of the eight used by Dalpiaz et al. in their
experiments: PROMISE, Dronology, ReqView, Leeds Library and WASP [
Table 1 indicates the number of requirements and their distribution among the four classes.
2.3.1. SVM
We directly used the datasets from the work of Dalpiaz et al. for their SVM-based classification approach
[
Mirroring the approach adopted for the SVM model, we trained the LSTM on 75% of the PROMISE dataset and assessed its performance on the global evaluation dataset. 2.3.3. ChatGPT ChatGPT, developed by OpenAI, is a Generative Pre-trained Transformer model designed to generate human-like text based on the prompts it receives.
In our study, we used:
Prompt Engineering Prompt engineering is a critical aspect of interacting with ChatGPT models. Crafting efective prompts can significantly influence the model’s responses, particularly when precise or specific outputs are desired.
In the context of our study, two major strategies were employed: zero-shot prompting and few-shot
prompting [
Zero-Shot Prompting In Zero-shot prompting, models are given the task without the benefit of specific contextual examples in the prompt. As such, the model draws purely from its vast pre-trained knowledge and the immediate context provided in the prompt. Zero-shot scenarios ofer a unique insight into the model’s innate understanding of the task and its ability to generalize from its training data without needing explicit examples to guide its response.
For the zero-shot setting in our study, our prompts were engineered following these three basic principles: 1. Each prompt should include all necessary details to get more relevant answers; 2. Each prompt should ask the model to adopt a software requirements expert persona. 3. The words used in each prompt should come from the Dalpiaz et al. reference paper. For example, the designed zero-shot prompt for the "IsFunctional" classification is: You are a software requirements expert tasked with categorizing software requirements into: IsFunctional: if the requirement possesses functional aspects (functional goals or functional constraints). IsQuality: if the requirement possesses quality aspects (quality goals or quality constraints).
OnlyFunctional: if the requirement possesses only functional aspects, with no quality aspects.
OnlyQuality: if the requirement possesses only quality aspects, with no functional aspects.
Given the requirement below, determine if it falls under the class "IsFunctional" Requirement: "[requirement_text]"
Respond as "yes" or "no".
Few-Shot Prompting Few-shot prompting is another approach to guide the model towards desired outputs by providing a limited number of example inputs and their corresponding outputs. By providing these examples, the intention is to give the model a clear context of what is expected from it, thus aiding it in generating more accurate and task-specific responses.
In our study, few-shot prompting was applied with adherence to a set of structured guidelines. Firstly, we adopted the same principles as those used in the zero-shot setting. Additionally, for the selection of examples, we ensured that all were sourced from the PROMISE training set. A balanced representation was a key focus; for each class — "IsFunctional", "IsQuality", "OnlyFunctional", and "OnlyQuality" — we curated two examples each of the positive and negative cases. This approach was designed to ensure that the model was exposed to a diverse range of scenarios, thereby providing a comprehensive understanding of the classification nuances. For instance, in the case of "IsFunctional", the positive instances included one scenario that combined both "IsFunctional" and "IsQuality", and another that incorporated "IsFunctional" without "IsQuality".
It is important to note that ChatGPT was also evaluated on the same global evaluation dataset as used for SVM and LSTM, which includes the 25% of the PROMISE dataset and the entirety of the datasets from Dronology, ReqView, Leeds Library, and WASP. The examples for few-shot prompting were specifically chosen from the PROMISE training set to maintain consistency.
Querying ChatGPT The OpenAI API was used to query ChatGPT. Every individual instance of each test set was provided to ChatGPT in one API call. The parameters of ChatGPT API calls were set as follows: • Temperature: We set the temperature parameter to 0 in order to minimize randomness in outputs and enhance the reproducibility of our evaluation results. However, due to the non-deterministic nature of ChatGPT, some variability may still occur. • Other parameters: Beyond the temperature setting, all other parameters were set at their default values.
While running the experiments, we encountered multiple server-side errors (e.g., BadGateway – HTTP 502, ServiceUnavailableError – HTTP 500). We incorporated exception handling in our approach in order to handle them.
To provide a clear understanding of the implementation details and ensure transparency and reproducibility, the code used in this study has been made publicly available [8].
In the context of our requirements classification task, which is characterized by a pronounced imbalance in the dataset as shown in Table 1, the selection of an appropriate evaluation metric is important. Dalpiaz et al. have previously employed a range of metrics including precision, recall, F1-score, and AUC to assess model performance. Since ChatGPT does not ofer probability distribution or score confidence which are essential for the calculation of the AUC metric, we cannot employ this metric in our evaluation. The F1-score, while commonly used, assumes that precision and recall are of equal importance by attributing them equal weight. This assumption does not hold in our context, especially considering that achieving high recall is more challenging and crucial in manual processes. Typically, it is more feasible to manually reject a false positive identified by a tool than to manually find a true positive within the tool’s input. Consequently, we propose the use of the score, which allows us to weight recall over precision to reflect its greater importance. The value of should be empirically determined based on the inverse frequency of the target class within the dataset [9]. As explained by Berry [9], the inverse frequency indicates the average number of items that must be examined in the search space to find one true positive, providing a lower bound estimate for .
Table 2 is constructed based on the inverse frequencies derived from the datasets, illustrating the values for each class.
The scores in Table 3 reveal that no single technique consistently outperforms the others across all four classifications. Rather, the optimal classification technique is contingent upon the specific requirements class pursued. For instance, in the "IsFunctional" classification, GPT-3.5 and GPT-4 are highly competitive in both Zero-Shot and Few-Shot settings, with GPT-3.5 Few-Shot reaching an score of 0.899. Meanwhile, LSTM leads in the "IsQuality" classification with a score of 0.708. The "OnlyFunctional" classification is best addressed by the GPT-4 Zero-Shot model, which scores 0.872. Lastly, for requirements that are solely quality-centric ("OnlyQuality"), the GPT-3.5 Few-Shot setting also shows a commendable score of 0.732.
An additional nuance to consider regarding the performance of the LSTM model is that although it is the best performer in the "IsQuality" classification, its efectiveness is not consistent across all the individual test sets. Indeed, its performance is particularly good on the PROMISE test set, likely due to being trained on similar data. However, its efectiveness drops when applied to other test sets, as illustrated in Table 4, indicating a limitation in its ability to generalize. translates directly to a score of zero. While the score is designed to provide a more nuanced evaluation in the presence of class imbalance by emphasizing recall, this example shows that even a well-chosen metric may not be suficient in the context of highly imbalanced datasets. The "OnlyQuality" performance metrics comparison
Test set
SVM
LSTM
presented in Table 6, one can observe that GPT-3.5 outperforms GPT-4 in three out of four scenarios. This is particularly notable in the "IsQuality" and "OnlyQuality" classifications, where GPT-3.5 has higher scores. GPT-4 only stands out is in the "OnlyFunctional" classification, demonstrating a significant improvement over GPT-3.5 in both zero-shot and few-shot settings. Therefore, if the classification of purely functional requirements is critical and the budget allows for the higher cost of GPT-4, it would be the recommended choice.
The deltas show that few-shot setting does not consistently improve upon the zero-shot baseline, with marginal declines in performance across most classifications. Conversely, GPT-3.5 displays significant positive deltas in "OnlyFunctional" and "OnlyQuality" classifications, indicating that few-shot learning has a pronounced beneficial impact when the zero-shot performance is weak. These insights suggest that few-shot learning can be particularly valuable in scenarios where a model struggles to perform adequately without prior examples, as it can lead to marked performance gains and help models like GPT-3.5 overcome initial deficiencies in understanding and classifying requirements.
In our study, we identified several threats to validity across diferent dimensions. Internally, we noted biases such as the sensitivity of ChatGPT’s responses to prompt variations and the non-deterministic nature of its output, which could vary slightly even with identical prompts. The choice and representation of few-shot examples and the potential impact of server errors also posed challenges to the reliability of our findings. Externally, the use of specific datasets and the subjectivity of tagger annotations could introduce biases. Regarding construct validity, we acknowledged that traditional metrics might not fully reflect the model’s capabilities in classifying requirements, accentuating the importance of interpretability. Finally, our conclusion validity faced threats from unbalanced datasets, which could bias our results.
The work of Rashwan et al. [10] introduced a novel approach by developing a new corpus with annotations for diferent types of NFR based on a requirements ontology and employing a SVM classifier to categorize requirements sentences into diferent ontology classes automatically. The proposed approach showed promising results in two diferent software requirements specification corpora, underlining the importance of semantic analysis and ontological representation.
Ray et al. [11] introduced the aeroBERT-Classifier, a specialized model designed for the aerospace domain. The model’s architecture, leveraging a domain-adapted BERT, underscores the merits of industry-specific adaptations. Their comparative analysis against models like GPT-2 and Bi-LSTM further delineates the robustness of transformer architectures in similar tasks.
In a distinct contribution, Rahimi et al. [12] proposed an ensemble approach combining a suite of machine learning classifiers, including Naïve Bayes, SVM, Decision Tree, Logistic Regression, and SVC. Their ensemble method achieved a remarkable 99.45% accuracy in classifying Functional Requirements, afirming the eficacy of harnessing multiple models to optimize classification outcomes.
Lastly, adding a linguistic dimension, Yucalar [13] introduced BERTurk, a model specifically finetuned for classifying software requirements in the Turkish language. Through rigorous empirical validation, Yucalar established that BERTurk achieves a commendable 95% F1-score in diferentiating between functional and non-functional requirements. This research accentuates the value of integrating linguistic nuances with state-of-the-art NLP techniques within the sphere of RE.
In this study, we conducted a comprehensive assessment of two ChatGPT models: gpt-3.5-turbo, and gpt-4. We evaluated these models in both zero-shot and few-shot settings, comparing them against established methods such as SVM and LSTM in requirements classification. Our findings show that there is no single best technique for all requirements classifications. The best technique varies depending on the specific requirement classification. For instance, GPT-3.5 Few-Shot configuration leads in the "IsFunctional" and "OnlyQuality" classifications, LSTM model performs best in the "IsQuality" classification, and GPT-4 Zero-Shot setting stands out in the "OnlyFunctional" classification. Our results also indicate that GPT-3.5 is generally more efective than GPT-4, except when it comes to "OnlyFunctional" requirements classification, where GPT-4’s higher cost may be justified by its enhanced performance. Interestingly, the few-shot setting has been found to be beneficial primarily in scenarios where zero-shot performance is notably weak. From a practical perspective, these findings suggest that for "IsFunctional" and "OnlyQuality" classifications, the default choice should be GPT-3.5 Few-Shot. In scenarios where the classification of "OnlyFunctional" requirements is paramount and resources are available, GPT-4 emerges as a reasonably good cold start. If budget constraints are a factor, GPT-3.5 Few-Shot remains a viable alternative. For "IsQuality" classification, LSTM stands out as the most efective tool, however it needs to be trained.
In the future, we intend to undertake a series of rigorous comparisons of ChatGPT with other LLMs, such as Llama-2 and Mistral, in performing RE tasks. Such analyses could ofer valuable insights into the relative advantages and limitations of each model for various RE tasks. To complement this research direction, it is also essential to build high-quality benchmark requirements datasets for training and comprehensively evaluating LLMs on a wider range of RE tasks. D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, D. Amodei, Language models are few-shot learners, in: H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, H. Lin (Eds.), Advances in Neural Information Processing Systems, Curran Associates, Inc., 2020. [8] A. El-Hajjami, N. Fafin, C. Salinesi, Requirements-classifiers-evaluation-materials, 2024. URL: https://doi.org/10.5281/zenodo.10802076. [9] D. M. Berry, Empirical evaluation of tools for hairy requirements engineering tasks, Empirical
Software Engineering 26 (2021) 111. [10] A. Rashwan, O. Ormandjieva, R. Witte, Ontology-based classification of non-functional requirements in software specifications: A new corpus and svm-based classifier, in: Proceedings of the 2013 IEEE 37th Annual Computer Software and Applications Conference (COMPSAC), IEEE, 2013. [11] A. T. Ray, B. F. Cole, O. J. P. Fischer, D. N. Mavris, aerobert-classifier: Classification of aerospace requirements using bert, Aerospace 10 (2023) 279. [12] N. Rahimi, F. Eassa, L. Elrefaei, An ensemble machine learning technique for functional requirement classification, Symmetry 12 (2020) 1601. [13] F. Yucalar, Developing an advanced software requirements classification model using bert: An empirical evaluation study on newly generated turkish data, Applied Sciences 13 (2023) 11127.