The Information Retrieval in Software Engineering (IRSE) track focuses on the automated assessment of software artifacts using machine learning techniques. The 2025 edition emphasizes two tasks: (i) binary classification of source code comments into useful and not useful categories, and (ii) a pilot task on estimating the functional correctness of code generated by Large Language Models (LLMs). The primary dataset comprises 9,048 commentcode pairs extracted from open-source C projects on GitHub. Fourteen teams from academia and industry submitted a total of 45 experimental runs. Results indicate that transformer-based models with software-specific embeddings outperform traditional classifiers, while LLM-based inference proves highly efective for code quality estimation.
Steidl et al. [
Existing methods largely focus on evaluating comment quality by identifying irrelevant or repetitive
words and phrases in relation to nearby code constructs. However, the definition of quality is inherently
context-dependent. For instance, Rahman et al. [
The task is defined as a binary classification problem aimed at categorizing source code comments as either Useful or Not Useful based on a given comment and its corresponding code snippet.
A Useful comment contains meaningful software development concepts that are not explicitly reflected in the surrounding code, thereby enhancing its comprehensibility. Conversely, a Not Useful comment either contains very few relevant concepts or includes suficient software development information that is already evident in the surrounding code, making it redundant. /* Initializes the buffer only once to avoid redundant allocations */ ---- USEFUL void init_buffer() { static int initialized = 0; if (!initialized) { buffer = malloc(BUFFER_SIZE); initialized = 1; }
} /* Function to initialize buffer */ ---- NOT USEFUL, REDUNDANT void init_buffer() { static int initialized = 0; if (!initialized) { buffer = malloc(BUFFER_SIZE); initialized = 1; }
}
The IRSE track utilizes a robust dataset comprising 9,048 comment-code pairs extracted from opensource C-based projects on GitHub. Each data instance is a tuple consisting of the comment text, the associated surrounding code snippet (function or block), and a binary ground-truth label indicating whether the comment is useful or not useful. The rigorous data collection process ensured that the comments evaluated were pertinent to real-world software maintenance scenarios.
The pilot task focuses on predicting functional correctness of LLM-generated solutions using the HumanEval benchmark. Given a problem description and multiple candidate solutions, systems rank solutions by estimated correctness using IR-style ranking metrics such as nDCG.
The 2025 edition of the IRSE track witnessed active engagement from the software engineering research community, receiving experimental submissions from 14 diverse teams. The participation featured a strong mix of academia and industry, reflecting the universal relevance of the problem statement. The cohort comprised 7 teams from the Indian Institute of Technology (IIT) Kharagpur, 4 from IIT Goa, 2 from SSRN College, and 1 from Salesforce. The track focuses on software maintenance, a critical phase in the software lifecycle, and participant details are outlined in Table 1.
The provided dataset was meticulously balanced to prevent class bias during training, containing 4,208 instances labeled as “useful” and 4,235 instances labeled as “not useful.” This balance allowed for a fairer evaluation of model performance using metrics like the F1-score. In terms of feature extraction, participants primarily employed feature engineering techniques centered on text mining to isolate significant keywords and phrases. Additionally, almost all teams utilized string matching and overlap coeficients to quantify the redundancy between the comment text and the source code tokens.
A significant number of teams utilized state-of-the-art transformer-based models such as BERT and GPT to learn the comment quality labels. These models were typically fine-tuned over 50 to 75 epochs, utilizing binary cross-entropy loss functions and the Adam optimizer to minimize classification error. While the GPT architecture achieved the highest F1 score—demonstrating superior capability in understanding context—its deployment requires substantial computational resources, which are typically more accessible to large software companies like Salesforce.
In contrast, other teams employed more traditional architectures including Recurrent Neural Networks (RNNs), Support Vector Machines (SVMs), Random Forests, and Logistic Regression. These models often relied on a hybrid of textual features (e.g., length, keyword frequency) and code-correlation features. Interestingly, the F1 scores obtained from RNNs and SVMs were comparable to those obtained from the more complex BERT variants. This competitiveness is attributed to the balanced nature of the dataset and the high discriminative power of explicit textual features when combined with numerical vectors, suggesting that lighter models can still be efective for this specific task.
The experiments involved a comprehensive exploration of embedding spaces, utilizing both
contextaware and context-independent embeddings. These were either trained from scratch on specific corpora
or fine-tuned with software development concepts. The best results were consistently achieved using
the recently released contextualized CodeBERT embeddings [
Comparable results were also obtained by using CodeELMo [
The Information Retrieval in Software Engineering (IRSE) track represents a significant step forward in empirically investigating a diverse range of approaches within a machine-learning framework to automate the evaluation of comment quality. Central to this evaluation is the definition of quality, which is rigorously assessed based on whether a comment provides semantic information that aids in the comprehension of the surrounding code, rather than mere syntactic correctness.
This iteration of the track saw robust engagement from the research community, with 14 teams participating and submitting a total of 45 distinct experiments. This diverse participation facilitated a comprehensive comparative analysis of various machine learning models, embedding spaces, and feature engineering strategies, ranging from traditional statistical classifiers to advanced neural networks.
The results highlight a clear trend toward the eficacy of large language models in software maintenance tasks. The highest F1-Score of 0.9102 was achieved by the Salesforce team using the GPT-2 architecture, combined with hybrid textual and numerical features derived from CodeBERT vector space embeddings. This superior performance in identifying ‘useful’ versus ‘not useful’ comments underscores the potential of combining generative pre-trained transformers with domain-specific embeddings to build the next generation of intelligent automated software maintenance tools.
In the course of preparing this manuscript, the author(s) employed the generative AI tool ChatGPT. Its use was limited to performing checks for grammar and spelling. Following this, the author(s) conducted a thorough review and revision of the text and assume full responsibility for the final published content.