Efective issue classification is crucial for eficient software project management. However, labels assigned to issues are often inconsistent, which can negatively impact the performance of supervised classification models. In this work, we investigate how label consistency and training data size afect automatic issue classification. We first evaluate a few-shot learning approach on a manually validated dataset and compare it to fine-tuning on a larger crowd-sourced set. The results show that our approach achieves higher accuracy when trained and tested on consistent labels. We then examine zero-shot classification using GPT-3.5, finding that its performance is comparable to supervised models despite having no fine-tuning. This suggests that generative models can help classify issues when annotated data is limited. Overall, our findings provide insights into balancing data quantity and quality for issue classification.
Ital-IA 2024: 4th National Conference on Artificial Intelligence, organized by CINI, May 29-30, 2024, Naples, Italy $ giuseppe.colavito@uniba.it (G. Colavito); iflippo.lanubile@uniba.it (F. Lanubile); nicole.novielli@uniba.it (N. Novielli); luigi.quaranta@uniba.it (L. Quaranta)
0000-0003-3871-401X (G. Colavito); 0000-0003-3373-7589 (F. Lanubile); 0000-0003-1160-2608 (N. Novielli); 0000-0002-9221-0739 (L. Quaranta)
© 2024 Copyright for this paper by its authors. Use permitted under Creative CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g CCoEmmUoRns LWiceonsrekAstthribouptionP4r.0oIncteerenadtiionnagl s(CC(CBYE4U.0)R.-WS.org) mains. With the advent of recent GPT-like Large Lan- Table 1 guage Models (LLMs), researchers have started investi- Distribution of labels in the extracted samples. gating their potential in solving software engineering Label Train set Test set challenges [16, 17]. To better understand how GPT-like Bug 47 24% 53 27% LLMs can be leveraged in automated issue labeling in the Documentation 33 17% 32 16% absence of training data, we formulate and investigate Feature 60 30% 55 28% our second research question as follows: Question 44 22% 47 24%
RQ2: To what extent we can leverage GPT-like LLMs to Discarded 16 8% 13 7% classify issue reports? Total 200 200
To address it, we evaluate GPT3.5-turbo [18] in a zeroshot learning scenario, where the model is prompted ifers using the small manually validated training dataset by only providing the task and label descriptions. We described in Section 2. In particular, we train and evaluate compare the performance of classifiers based on GPT-like a model based on SETFIT [14] using the manually labeled LLMs with fine-tuned BERT-like LLMs [19]. train and test sets. Then we compare its performance
In this paper, we discuss our ongoing work on using with the one obtained by fine-tuning RoBERTa [ 15] using
LLMs to address software engineering challenges, with a the full dataset of 1.4M crowd-annotated issues [
our dataset as follows. First, non-textual items, such as links, code snippets, and images, are identified and replaced with tokens (e.g., <link> for links) in the dataset. Next, we use the ekphrasis Text Pre-Processor 1 to normalize the text by detecting and replacing items such as URLs, email addresses, symbols, phone numbers, mentions, time, date, and numbers with specific tokens.
400 GitHub issues labeled as bug, features, question, and documentation. The dataset is split into two subsets of 200 issues which we use as train and test sets, respectively.
Both subsets are equally distributed and include 50 issues Choice of GPT-like models Several LLMs have been
per class. Our dataset is obtained by manually labeling proposed in the last few years, with GPT-3 [23] being
the 400 randomly selected items from the dataset of 1.4M one of the most popular. There is a significant prevalence
GitHub issues distributed by the NLBSE’23 tool competi- of studies leveraging GPT3.5-turbo [24], an
instructiontion organizers [
To address our first research question, we investigate the eficacy of few-shot learning for training robust classi• Input Format: The format of the input issues,
which includes a title and a body;
1https://github.com/cbaziotis/ekphrasis
• Task Description: A description of the classifica- beled gold standard and tested on the raw test set. When
tion task to be performed, including the possible trained and evaluated on the manually labeled dataset
labels that can be assigned to the issues; (a), SETFIT performs better than RoBERTa (b1 and b2),
• Label Descriptions: A brief description of each la- regardless of whether the training set used for RoBERTa
bel. Label descriptions are generated by ChatGPT is raw or manually labeled. However, when trained on
and then manually reviewed to ensure they are the manually-labeled dataset (b1), RoBERTa struggles to
clear and informative. deliver good performance due to a shortage of training
• Input Issue: The issue to be classified; data. On the other hand, when trained on the raw dataset
• Output format instructions: The desired output (b2), RoBERTa achieves competitive performances, but it
format. We ask the model for a JSON object con- is unable to outperform SETFIT (b).
taining a reasoning and the predicted label. This As the manually-labeled dataset embodies the ideal
is done to inject some Chain-of-Thought reason- labeling criteria for classifiers, comparing SETFIT (a) and
ing into the model, as suggested in previous stud- RoBERTa (b2) provides a practical scenario in which we
ies about prompting LLMs [
4.2. Leveraging GPT for automatic issue report classification (RQ2) 4.1. Impact of label consistency on the In Table 3, we report the classification performance of classifier performance (RQ1) GPT compared to the SETFIT model. As already explained in the previous section, we experimented with In Table 2, we present the results obtained by training the several versions of GPT 3.5 that were available at the SETFIT classifier on the hand-labeled gold standard and time of the study. For a full report of the results, see evaluating it on both the hand-labeled test set (a) and the Colavito et al. [19]. In this paper, we include considerfull test set distributed for the challenge (c). To ensure a ation of the 16k-0613 model only as this achieves the fair comparison, we compared the SETFIT model’s per- best performance in terms of a combination of F1 and formance with the performance obtained by RoBERTa percentage of discarded items due to nonsensical model on the same test set, when trained on the hand-labeled output. Specifically, none of the predictions from this gold standard set (b1). Furthermore, we also include the model were discarded. We observe that the Feature class performance obtained by training the RoBERTa classifier achieves the best F1, while the Documentation class is on the full train set distributed by the organizers (b2). the most problematic to identify, showing a lower recall
To assess the ability of the models to generalize on than the other classes.
a broader dataset, we also include a comparison with While the zero-shot GPT model achieves a slightly
the NLBSE ’23 challenge baseline [
The SETFIT model achieved an F1-micro score of .7767 GPT for this application since it can perform the classifi(see model (c) in Table 2) when trained on the manually la- cation in absence of labeled data, i.e., without the need for fine-tuning. This evidence could help maintainers of new projects, for which historical data is not available or is scarce. In such cases, API calls to GPT could be used to classify issue reports, providing a valuable tool for project management. Once the project has accumulated enough labeled data, the maintainer could switch to a ifne-tuned model to improve the classification accuracy.
Although this could be a viable solution for open-source projects, it is worth noting that the cost of API calls and the privacy of data could limit its practical feasibility in commercial projects. In such cases, project maintainers might consider using open-source models or building and deploying a classifier on-premise. Nonetheless, the construction and maintenance of LLMs is expensive both in terms of resources and time, and this constitutes a barrier to their adoption in most cases. state-of-the-art performance in the absence of manually annotated issues, i.e. when a gold standard is not available for fine-tuning state-of-the-art approaches based on BERT-like models. Our empirical results show that GPTlike models can achieve a performance comparable to the state-of-the-art without the need for fine-tuning. This suggests that when manual annotation is not feasible or a gold standard for training is not available (i.e., on a new project), maintainers could rely on generative AI to successfully address the issue classification task.
However, using LLMs to build issue classifiers might pose important challenges due to licensing and computational limitations. As such, we plan to extend this benchmark with open-source LLMs, also including issue-report datasets. This will enable evaluating the generalizability of our findings.
In this paper, we summarized the outcomes of our re- This research was co-funded by the NRRP Initiative,
cently published studies on the use of large language Mission 4, Component 2, Investment 1.3 - Partnerships
models for automated issue classification. Specifically, extended to universities, research centres, companies
we investigated the impact of improving data quality on and research D.D. MUR n. 341 del 15.03.2022 – Next
issue classification performance. We trained and eval- Generation EU (“FAIR - Future Artificial Intelligence
Reuated a model based on few-shot learning using SET- search”, code PE00000013, CUP H97G22000210007) and
FIT with a subset of manually verified data. The model by the European Union - NextGenerationEU through
achieves better performance when trained and tested the Italian Ministry of University and Research, Projects
on data for which label consistency was manually veri- PRIN 2022 (“QualAI: Continuous Quality Improvement
ifed [
Furthermore, we explored the performance of GPTlike models for automatic issue classification [ 19] to understand if we can leverage GPT-like LLMs to achieve