This study presents our submission to the FIRE 2025 IRSE shared task, which focuses on automatically determining whether a comment within source code is genuinely helpful or redundant. Unlike prior approaches that primarily relied on bag-of-words features or large transformer based models, we propose a graph oriented approach in which each code-comment pair is transformed into a graph structure. Here, each node corresponds to individual tokens from the code-comment pairs, edges capture diferent sorts of relationships between code-comment pair tokens such as normal sequential order of tokens, skip connections for longer range dependencies, and dedicated bridge links that explicitly connect code elements to comment tokens. For said node representation, we use 53-dimensional vectors obtained from pretrained embeddings and we train a compact Graph Neural Network comprising of only 9762 learnable parameters. To further assess the robustness of this approach, we further evaluate it on synthetically constructed datasets, where the trained model attains 62.5% accuracy on a smaller set of 16 samples and 83% on a larger set of 100 samples.
Comprehending source code is a fundamental prerequisite in modern software development. When engineers are tasked with updating or extending an application, they frequently rely on accompanying comments to understand the purpose of implementations and the rationale behind design decisions. However, the quality and usefulness of such comments in real-world projects can vary considerably depending on multiple factors.
The Forum for Information Retrieval Evaluation (FIRE) 2022 edition introduced a shared task aimed at automatically classifying whether code comments are useful or not useful for given code snippets written in the C programming language. As part of this initiative, a dataset of manually labeled code–comment pairs was built, where domain experts assessed the usefulness of each comment.
Previous research addressing this problem has explored a wide range of computational approaches.
Majumdar [
Most existing studies treat source code purely as natural language text, applying standard natural language processing techniques without accounting for the inherent structural properties of code. A graph based formulation can potentially enable a model to better determine whether a comment is genuinely useful. Graph Neural Networks (GNNs) have demonstrated strong performance across various code analysis tasks but, to the best of our knowledge, have not yet been applied to this specific problem.
This work makes three primary contributions. First, we propose a graph based representation for each code - comment pair that explicitly connects tokens from both sides through multiple types of links. Second, we develop and train a lightweight GNN classifier that utilizes 53-dimensional node features with only 9,762 trainable parameters. Finally, we evaluate the model’s generalization capability using synthetically generated datasets of varying sizes.
The FIRE 2022 IRSE shared task attracted several participants who explored diverse computational
approaches for assessing the usefulness of code comments. [
Graph based representations have gained increasing attention in recent years for a variety of
codeanalysis tasks. Chen and Monperrus [
Research on the quality and consistency of code comments has been active for over a decade. Tan [
The dataset employed in this study was introduced as part of the FIRE 2022 IRSE shared task [
Each code - comment pair is represented as an undirected, weighted graph using the NetworkX library. The graph construction process consists of three primary stages: tokenization, node creation, and edge formation.
• Tokenization: For the source code, text is segmented into tokens using the regular expression [+[]\w]+, which retains words, operators, and punctuation as separate entities. All tokens are converted to lowercase. For comments, comment markers are removed, and alphabetic words are extracted using [a-zA-Z][a-zA-Z0-9]*. Tokens consisting of a single character are discarded to reduce noise. • Node Creation: Each token is represented as a node that stores its textual content, source (code or comment), and positional index within the sequence. Code tokens are positioned before comment tokens to preserve a logical structure. • Edge Formation: Multiple types of edges are added to capture both local sequential dependencies and long-range relationships. Sequential edges with a weight of 1.0 connect adjacent tokens within the same sequence. Additional edges include skip links for non-adjacent dependencies, bridge edges connecting related code and comment tokens, and self-loops for all nodes to retain self-referential information.
Each node is encoded as a 53-dimensional feature vector. These embeddings are pretrained on the training corpus and capture token identity, contextual information from neighboring tokens, origin type (code or comment), and normalized positional attributes. The embeddings are saved and reused across all subsequent experiments to maintain consistency and reduce computational overhead.
The proposed model builds upon the Graph Convolutional Network (GCN) framework introduced
by Kipf and Welling [
The pooled representation is then passed through a fully connected layer that reduces its dimensionality from 64 to 32, followed by a final linear layer that outputs two logits corresponding to the binary classes. Overall, the model comprises 9,762 trainable parameters, emphasizing its lightweight design. The GCN layer updates each node representation according to the following formulation: ⎛ ℎ(+1) = ⎝
∑︁ ∈ ()∪{}
1 √︀
⎞ ()ℎ() ⎠ , () denotes the feature vector of node at layer , () represents the set of neighboring nodes, where ℎ is the degree of node , () is the learnable weight matrix, and is the ReLU activation function.
The dataset was divided into 80% for training (9,161 samples) and 20% for testing (2,291 samples). The training subset exhibited a class imbalance, with 3,510 not useful samples and 5,651 useful samples. To address this issue, an upsampling strategy was adopted by generating synthetic variants of the not useful samples. Gaussian noise with a standard deviation of = 0.05 was added to the 53-dimensional embeddings, producing a balanced training set with 5,651 examples per class.
Model training was performed using the Adam optimizer with a learning rate of 0.001 and a weight decay of 10−4 . A batch size of 64 was used, and training proceeded for 30 epochs. The binary crossentropy loss function was employed as the objective. At the end of each epoch, the model checkpoint yielding the highest validation accuracy was preserved for evaluation.
To evaluate the model’s generalization capability beyond the original training distribution, two synthetic test sets were constructed. The smaller set contained 16 examples evenly divided between useful and not useful classes, while the larger set consisted of 100 similarly balanced examples. The useful comments described non trivial programming concepts such as binary search algorithms or memory management, whereas the not useful comments corresponded to trivial statements such as “set x to 1” or “loop through array.” Each synthetic code - comment pair was processed using the same graph construction pipeline as the original dataset to ensure methodological consistency.
The proposed model achieved an accuracy of 80.8% on the original test set comprising 2,291 samples. The weighted F1 score was 0.807, indicating well-balanced performance across both classes. For the not useful class, precision, recall, and F1 scores were 0.77, 0.71, and 0.74, respectively. For the useful class, precision was 0.83, recall was 0.87, and F1 was 0.85.
The confusion matrix revealed that, among 879 not useful samples, 626 (71%) were correctly classified, while 253 were misclassified as useful. Conversely, for 1,412 useful samples, 1,226 (87%) were correctly identified, and 186 were misclassified as not useful. Table 1 presents a comparison of model performance under diferent experimental configurations. The baseline configuration, which utilized 6-dimensional features, achieved 79.8% accuracy. Increasing the feature dimensionality to 53 improved accuracy to 80.4%, and incorporating upsampling further enhanced it to 80.8%.
When evaluated on the small synthetic dataset containing 16 samples, the model attained an accuracy of 62.5%, corresponding to an 18.3% decrease relative to the original test set. For the larger synthetic dataset of 100 samples, accuracy increased to 83.0%. These findings suggest that model performance improves with larger and more diverse synthetic datasets, indicating sensitivity to dataset size and variability.
During training, the model demonstrated stable convergence behavior. The training loss decreased from 0.607 in the first epoch to 0.421 by the 30th epoch, while validation accuracy exhibited steady improvement during the initial 15–20 epochs before plateauing around epochs 25–26. No significant overfitting was observed, as validation accuracy remained consistent and did not deteriorate toward the end of training. These trends indicate that the model efectively generalized to unseen data without sacrificing stability.
The proposed graph-based representation ofers distinct advantages over treating code and comments purely as plain text. By incorporating multiple types of connections, the graph structure enables the model to better capture relationships between code tokens and corresponding comment tokens. Sequential edges preserve the inherent order of tokens, while skip connections allow the model to recognize long-range dependencies across non-adjacent tokens. Bridge connections, on the other hand, explicitly link code and comment elements, allowing the model to associate linguistic context with program structure. Collectively, these connections help the model discern whether a comment provides meaningful information or simply restates the code.
The proposed approach has certain limitations. First, the graph construction process relies on manually defined rules, rather than being learned adaptively from data. Second, the 53-dimensional embeddings, though efective, may not fully capture the semantic richness of complex programming constructs or natural-language comments. Incorporating larger pre-trained representations such as CodeBERT or GraphCodeBERT could provide more expressive features.
Future extensions of this work may focus on: (1) learning optimal graph structures and edge weights directly from data, (2) integrating attention-based mechanisms for more efective information propagation, and (3) employing large-scale pre-trained models trained on diverse code repositories to enhance generalization across programming languages and comment styles.
On the FIRE 2022 IRSE dataset, the proposed model achieved an accuracy of 80.8%, which is comparable
to, though slightly below, the performance of CommentProbe [
This paper presented a graph-based framework for classifying the usefulness of code comments using Graph Neural Networks (GNNs). The proposed model achieved an accuracy of 80.8% on the FIRE 2022 IRSE shared task, while remaining lightweight with only 9,762 trainable parameters. Evaluation on synthetic datasets demonstrated that accuracy decreased to 62.5% on a smaller set of 16 samples, but improved to 83.0% on a larger set of 100 samples. These findings suggest that the proposed graph-based approach efectively captures structural relationships between code and comments, striking a balance between the simplicity of bag-of-words models and the representational power of transformer-based architectures.
The results underscore the potential of graph-oriented models as eficient alternatives for comment usefulness classification, especially in scenarios where computational resources are limited. Future research may explore scaling this framework with larger pre-trained embeddings and automated graph construction techniques to further enhance generalization.
All source code and experimental configurations are publicly available at: https://github.com/ CrimsonCoderAadit/code_comment_project_clean
We thank the organizers of FIRE 2025 IRSE shared task for providing the dataset and evaluation framework. The authors would also like to express their gratitude to the faculty of the Department of Computer Science and Engineering, SSN College of Engineering, for their valuable guidance and support throughout this work.
In the course of preparing this manuscript, the author(s) employed the generative AI tool ChatGPT. Its use was limited to performing checks for grammar and spelling. Following this, the author(s) conducted a thorough review and revision of the text and assume full responsibility for the final published content.