The Software Engineering Information Retrieval (IRSE) track focuses on developing automated methods to evaluate code comments using a machine learning framework. This year, the track featured two key tasks: (i) predicting the usefulness of code comments and (ii) estimating code quality. The first task focuses on distinguishing code comments as either useful or not useful. The dataset comprises 9,048 pairs of code comments sourced from open-source C-based projects on GitHub, along with an additional dataset generated by teams utilizing large language models (LLMs). A total of 12 teams from various universities contributed to this efort, conducting experiments that were evaluated using both quantitative and qualitative metrics. Notably, while labels generated by large language models introduce bias into the prediction model, they also contribute to reducing overfitting, leading to more generalizable results. The sub-track pertaining to code quality estimation was introduced this year. Given a problem description, and a list of large language model (LLM) generated software code, the objective of the task is to automatically estimate the functional correctness of each generated code. For the purpose of evaluation, each problem-solution pair is then ranked by these estimated probabilities of functional correctness, the quality of which is then reported with standard ranking performance measures.
Evaluating the quality of comments is essential for optimizing codebases and improving code maintainability. Clear and well-organized comments can greatly enhance the readability and understanding of the code as long as they are consistent and informative.
Perceptions of comment quality, especially regarding their "usefulness," are context-dependent and
can vary across diferent situations. Bosu et al. [
Majumdar et al. [
In the first iteration of the IRSE track at FIRE 2022, the research in [
The performance of these models, especially in understanding the relationships between code and comments, can provide valuable insights into the quality of the generated data and its potential for scaling the existing classification model. Additionally, this approach can be generalized to any classification model based on software metadata.
In addition to the core track of comment usefulness detection, this year - as a pilot track, we introduced
a new sub-track pertaining to quality estimation of large language model (LLM)-generated code. More
precisely speaking, given a problem description comprised of a natural language description of a
programming task along with a partially written code, e.g., the function prototype, a common practice
is to employ generative AI models (instruction tuned LLMs specifically fine-tuned on software code,
e.g., CodeLlama, Codestral etc.) to automatically generate code to solve the given task. Given a pair
comprised of a problem description along with an LLM-generated solution the objective is to estimate
the likelihood that the code is functionally correct, i.e., it provides a correct solution to the problem.
As analogy, this task is somewhat similar to the task of query performance prediction (QPP) [
Software metadata is essential for code maintenance and understanding. Numerous tools [
In the context of mining code comments and assessing their quality, authors [
With the rise of large language models [16], it is crucial to compare the quality assessment of code
comments generated by standard models like GPT-3.5 or LLaMA with human interpretations. The IRSE
track at FIRE 2023 builds on the approach proposed in [
Comment Classification : The task involves binary classification of source code comments as either Useful or Not Useful given a comment and its associated code snippet as input. The output is based on whether the information contained in the comment is relevant, and would help comprehend the surrounding code, i.e., it is useful. • Useful Comments have suficient software development concept → Comment is Relevant, and these concepts are not primarily present in the surrounding code → Comment is not Redundant. • Not Useful Comments have suficient software development concept → Comment is Relevant, and these concepts are mostly present in the surrounding code → Comment is Redundant.
Dataset: For the IRSE track, we use a set of 9048 comments (from Github) with comment text, surrounding code snippets, and a label that specifies whether the comment is useful or not.
Task and Evaluation Measures We scope the code quality estimation task to estimate the functional correctness of code snippets generated via LLMs in response to a prompt specifying a programming task. In particular, we make use of the HumanEval1 dataset for this task, which constitutes of 161 programming problem descriptions.
Given a programming task description , and a list of solutions = {1 , . . . , } generated by an LLM, a predictor model should estimate a likelihood score of the functional correctness of each solution, i.e., : , ↦→ R.
An efective model should estimate a high likelihood value for a functionally correct solution (the ground-truth being computed via a set of test-cases), which means that a standard evaluation metric for a ranking task may also be applied here - the only diference being the notion of ‘relevance’ replaced with that of ‘functional correctness’ ( being analogous to a query and to that of a set of top- retrieved documents). Motivated by this analogy, we report nDCG@ (in our setting, = 10, i.e., 10 solutions are generated for each problem) as an evaluation measure.
Additionally, we also report a global ranking efectiveness measure to compare across the performance over all problem tasks. Specifically, we use the predicted likelihoods to rank all the , pairs for each ∈ (the set of all problem tasks in a benchmark), and compute the nDCG value of this set, i.e., nDCG@(||).
To diferentiate the two measures, we call the former local nDCG ( l-nDCG) and the latter global nDCG (g-nDCG). More precisely speaking, to calculate l-nDCG, we rank , pairs for each problem , calculate nDCG and then calculate the average of nDCG values for all ∈ (i.e., ∑︀|=|1( /||)), whereas to calculate g-nDCG, we rank all || pairs of , for all ∈ and then calculate the nDCG value.
IRSE 2024 received a total of 12 experiments from 12 teams for the two tasks. As this track is related to software maintenance, we received participation from several research labs of educational institutes.
The various teams with the details of their submissions are characterized in Table 1. The dataset provided was balanced and had 4015 useful comments and 4033 not useful comments. The participants used various pre-trained embeddings such as one hot encoding, TF-IDF vectorizer, word2vec, or contextaware like ELM or BERT to generate vectors for the word sequence. Teams have used several machine learning models like support vector machine, logistic regression, and deep-learning based models such as BERT, Recurrent neural network, and so on.
Some participants were observed to achieve a slight increase in test accuracy when the model was trained with the addition of an LLM-generated dataset (Table 2). However, in many cases, the accuracy reduces (2%-4%). This behavior is due to the incorporation of silver standard data that reduces the over-fitting of the models.
A team from IIT-KGP participated in this task. Similar to the methodology proposed in [18], they employed GPT-3.5 Turbo zero-shot inference on a problem description and a solution pair to estimate how likely is the solution to be functionally correct. They submitted three runs with three diferent temperature ( ) settings for the GPT decoder (specifically, = 0.7, = 0.8 and = 0.9). The prompt used by the participating team is shown in Figure 1.
To set a reference point for comparison purposes, we employed a relatively simple heuristic baseline which given a problem description and a list of solutions measures the variance across the semantic Given the problem {Problem} and the solution {Solution}, generate a likelihood score between 0 and 1 indicating how relevant the solution is to the problem. Only state the score. similarities between each solution pair. For measuring the semantic similarity between a pairs of code solutions, we use the CLS embeddings obtained from CodeBERT [19], a BERT model fine-tuned on large volumes of source code data.
The assumption of using the variance across generated code solutions as an estimate is that topical diversity of the solutions may indicate lack of consistency in the solutions being generated, which could be be associated with a risk of the solutions being incorrect. Making this assumption is appropriate since HumanEval dataset contains the method signature for each problem. Although there can be diferent ways to provide solution (i.e., implement code) for a problem (e.g., by using diferent data structures or implementing a recursive algorithm instead of an iterative one, the method signature limits the way one can provide a solution for a given problem .
Table 3 shows that the GPT-based 0-shot inference produced better results than the in-house heuristicbased baseline of estimating code quality as a measure of the topical diversity between the LLMgenerated solutions.
The first sub-task of the IRSE track focused on exploring various automated approaches for evaluating comment quality. The assessment criteria were based on whether a comment provided useful information that enhanced the comprehension of the surrounding code. A total of 12 teams participated, employing diverse machine learning models, embedding techniques, feature sets, and LLM-generated data. The highest F1-score achieved was 0.853, with a notable improvement to 0.892 when incorporating LLMgenerated data. The inclusion of LLM-generated labels not only mitigated overfitting in classification models but also enhanced performance. Furthermore, when combining data from all participants with gold standard labels from industry practitioners, the overall F1-score improved, demonstrating the efectiveness of data augmentation through LLM-generated annotations.
The second sub-task of the IRSE track focused on assessing the efectiveness of predictive models in estimating the functional correctness of LLM-generated code. The evaluation revealed that LLM-based approaches for code quality estimation outperformed embedding-based baselines, demonstrating their superior ability to capture contextual and functional correctness aspects of generated code.
During the preparation of this work, the author(s) used ChatGPT in order to: Grammar and spelling check. After using these tool(s)/service(s), the author(s) reviewed and edited the content as needed and take(s) full responsibility for the publicationâs content. [16] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., Language models are few-shot learners, Advances in neural information processing systems 33 (2020) 1877–1901. [17] S. Majumdar, A. Varshney, P. P. Das, P. D. Clough, S. Chattopadhyay, An efective low-dimensional software code representation using bert and elmo, in: 2022 IEEE 22nd International Conference on Software Quality, Reliability and Security (QRS), IEEE, 2022, pp. 763–774. [18] T. Y. Zhuo, Ice-score: Instructing large language models to evaluate code, 2024.
arXiv:2304.14317. [19] Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, M. Zhou, Codebert: A pre-trained model for programming and natural languages, CoRR abs/2002.08155 (2020). URL: https://arxiv.org/abs/2002.08155. arXiv:2002.08155.