Code commenting is a practice developer pursues to improve the readability of the code. Hence, it is essential to evaluate comments based on whether they increase code understandability for software maintenance tasks. Although some studies have been conducted for detecting useful code comments, most of them exploit various handcrafted features, and the recent advancement of transformer-based models is still under-explored. Therefore to fill the gap, we explore transformer-based models and propose a fusion-based solution for detecting Useful comments based on the shared task, "Information Retrieval in Software Engineering (IRSE) 1" at FIRE 2022. We observe that our fusion-based model BERT+CodeBERT(PP), which uses features from both code comments and snippets, achieves the highest Macro F1 score of 90.739 among all the models and ranked first in this task.
Software development refers to a software deliverable’s design, documentation, programming,
testing, and ongoing maintenance 1. During the development and ongoing maintenance of
the product, a developer has to write a lot of code to fix bugs and extend/add new features.
Although during the development phase, the developer writes codes based on his thinking and
code writing style, it may not be the case when a bug comes, or new features have to be added;
the same developer may be working on the same code he wrote earlier. Therefore, the new
developer must understand the existing code before modifying it. However, it is not always
easy to understand the current code just by going through it without proper documentation.
Even if documentation is found, without adequate updates, it becomes outdated, which may not
benefit the new developers. Therefore, developers must manually search and mine source files
and other knowledge sources like emails, and defect trackers, to understand the application’s
overall design [
Commenting code is not always as easy as described in the definition. Depending on the
products and company guidelines, the technique of writing comments may difer. Although, to a
large extent, the commonality among all the guidelines is that a comment should be informative
and meaningful and represent whatever is written in actual coding without any ambiguity.
Nevertheless, it is undeniable that comments can be noisy, unstable, and may not evolve with
the source code [
In addition, merely placing irrelevant comments in the codes does not add any significance
to enhance the readability of the source code. Thus it is necessary to identify useful comments
in the source code given a code snippet. Therefore to engage and facilitate the research around
useful comment detection, the organizers of the “Information Retrieval in Software Engineering
(IRSE)" [
To this end, several strategies have been explored to classify comments based on various
handcrafted features such as explicit syntactic information, presence of specific tags (e.g., @param,
@deprecated, etc.), words and symbols; or implicit elements, such as comment length,
parts-ofspeech of comment words or the cosine similarity of words in code-comment pairs [
In this paper, we attempt to use the existing transformer-based models for our classification
task, which have already been seen to outperform several baselines and stand as a
state-of-theart model for various downstream tasks [
This section discusses some of the proposed strategies in the literature to investigate and evaluate comments quality by detecting inconsistencies and classifying comments.
Tan et al. [15] devised a tool iComments, to analyze comments written in natural language to extract implicit program rules and used those rules to automatically detect inconsistencies between comments and source code, indicating either bugs or irrelevant comments. For this purpose, the author incorporates Machine Learning, Natural Language Processing(NLP), Statistics, and Program Analysis methods and evaluates the tools on four large code bases: Linux, Mozilla, Wine, and Apache. Ratol el al. [16] designed a new rule-based approach called Fraco to detect fragile comments. It incorporates the identifier’s type, its morphology, the identifier’s scope, and the location of the comment. The author evaluated the method by comparing its precision and recall against hand-annotated benchmarks created for six targets Java systems and compared the results against the performance of Eclipse’s automatic in-comment identifier replacement feature.
Haouari et al. [
Although the existing methodologies established several baselines for meaningful code comments analysis, none of the approaches used the recent advancement of transformer-based models. Hence to fill the research gap, in this work, we propose a fusion-based technique using pre-trained transformer-based models to classify Useful comments based on the dataset shared by the organizers of IRSE.
The shared task on Useful Comment Classification (given the surrounding code snippet) in
Information Retrieval in Software Engineering (IRSE) [
This section discusses the methodology we followed for detecting Useful code comments. The detail of the pipeline is shown in Figure 1.
While manually going through the dataset, we observe the code comments contain lots of special characters, blank spaces, newlines, etc. Therefore we apply pre-processing steps to Code Snippet
Pre Processing Cleaned Code
Pre Processing Code Comments
CodeBERT (768 x 1)
Dense Layer (256 x 1)
Dense Layer (128 x 1)
Dense Layer (64 x 1) r ayeL )1x isonu (281 F
Classification
Head
Post Processing Cleaned Code Comments
BERT (768 x 1)
Dense Layer (256 x 1)
Dense Layer (128 x 1)
Dense Layer (64 x 1) remove all the non-English characters. Further, we observed the comments are associated with the code snippet, so we removed the code comments from the code snippet as well.
As part of our initial experiments, we pose the problem as a unimodal text classification task.
Here, instead of using the comments along with their associated codes, we only use the code
comments to find out whether the comment is useful or not. The idea is that although the code
associated with the comment is not utilized explicitly for the classification, sometimes developers
write code snippets in the comment to make it more transparent, which can indicate the model
to determine the usefulness of the comments. For this purpose, we use the transformer-form
model BERT [
BERT3, which stands for Bidirectional Encoder Representations from Transformers, is pretrained on a large corpus of English data utilizing masked language modeling (MLM) and Next sentence prediction (NSP) objectives in a self-supervised manner. It consists of a stack of transformer encoder layers with 12 “attention heads," i.e., fully connected neural networks augmented with a self-attention mechanism. The model can handle a maximum of 512 tokens as input. To fine-tune BERT, we add a fully connected layer with the output corresponding to the CLS token in the input. This CLS token output usually carries the representation of the sentence provided to the model.
As discussed above, the uni-modal model does not explicitly consider the associated code snippet along with the code comments. However, to decide whether a comment is useful, the use of surrounding code is crucial. Therefore we design a fusion-based model to take into consideration both code comments and code snippets to understand better the relationship between the comments and surrounding code snippets.
Although programming languages are primarily written in the English language, the grammar
of programming languages does not follow the rule of natural language. Hence using a model
like BERT, which is pre-trained on natural language, won’t be an excellent choice to represent
the code. Thus we use the model CodeBERT 4 [
Here we extract the 768-dimensional features vector from the last layer of the BERT and CodeBERT model using the code comments and code snippets, respectively. Then these feature vectors are separately provided through three dense intermediate layers of size 256, 128, and 64, respectively. Finally, we concatenate all the nodes(BERT+CodeBERT) and reduce them to a feature vector of length 2(Useful or Not-Useful).
We further apply the following post-processing step to improve the classification performance of the models. We interviewed two software developers and asked how they felt about the short-length comments. Based on their prior experience, both of them have noticed that shorter code comments are mostly Not-Useful. Thus as a post-processing step, we relabeled all the comments less than five tokens to Not-Useful.
We have trained the models for ten epochs with binary cross entropy loss for both the uni-modal and fusion-based models. We have used the Adam optimizer with an initial learning rate of 2e-5 and epsilon of 1e-8. For the unimodal text-based BERT model, we have used the batch size of 16 and maximum token length of 100; for the fusion-based model, we have used the batch size of 32. Additionally, as no validation set was given for the experiments, we divided the training data points into 85% and 15% split and used the 15% as a validation set. We predict the test set for the best validation performance. We have used HuggingFace[19] and PyTorch [20] for implementing all the models.
Model Accuracy BERT 89.810 BERT(PP) 92.607 BERT+CodeBERT 90.909 BERT+CodeBERT(PP) 92.807
Table 3 demonstrates the performance of each model. As expected, the BERT+CodeBERT model(Accuracy: 90.909, Macro F1: 88.804), which utilizes features from code comments and snippets, performs better than the standalone BERT(Accuracy: 89.810, Macro F1: 87.595) model. Further, we observe post-processing improves the model’s performance. In conclusion, BERT+CodeBERT(PP) model performed the best in terms of accuracy(92.807) and macro F1(90.739) score. We plot the confusion matrix in Figure 2 to further assess the models. Although post-processing makes the majority of the correction for Not-Useful classes; however, some of the Useful classes’ test data points got misclassified, which further reduces the recall for Useful classes, as shown in Table 3. The observation holds true for both the unimodal text-based model and the fusion-based model. Nonetheless, the post-processing technique improves the overall performance.
While Majumdar et al. [
In this shared task, we deal with a novel problem of classifying Useful code comments given the surrounding code snippet. We evaluated the state-of-the-art transformer-based models. We found that the fusion-based model BERT+CodeBERT, which uses features from code comments and snippets, performs better than the standalone text-based BERT model. In the future, we plan to explore other transformer-based models, such as Roberta [21], CodeT5 [22], etc., for code understanding and useful comments detection. We also plan to explore knowledge graphs relevant to Software Engineering to improve the classification performance. [13] M. Das, P. Saha, R. Dutt, P. Goyal, A. Mukherjee, B. Mathew, You too brutus! trapping hateful users in social media: Challenges, solutions & insights, in: Proceedings of the 32nd ACM Conference on Hypertext and Social Media, 2021, pp. 79–89. [14] M. Das, S. Banerjee, P. Saha, Abusive and threatening language detection in urdu using boosting based and bert based models: A comparative approach, arXiv preprint arXiv:2111.14830 (2021). [15] L. Tan, D. Yuan, G. Krishna, Y. Zhou, /* icomment: Bugs or bad comments?*, in: Proceedings of twenty-first ACM SIGOPS symposium on Operating systems principles, 2007, pp. 145– 158. [16] I. K. Ratol, M. P. Robillard, Detecting fragile comments, in: 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE), IEEE, 2017, pp. 112–122. [17] Y. Padioleau, L. Tan, Y. Zhou, Listening to programmers—taxonomies and characteristics of comments in operating system code, in: 2009 IEEE 31st International Conference on Software Engineering, IEEE, 2009, pp. 331–341. [18] H. Aman, S. Amasaki, T. Yokogawa, M. Kawahara, Empirical analysis of words in comments written for java methods, in: 2017 43rd Euromicro Conference on Software Engineering and Advanced Applications (SEAA), IEEE, 2017, pp. 375–379. [19] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. L. Scao, S. Gugger, M. Drame, Q. Lhoest, A. M. Rush, Huggingface’s transformers: State-of-the-art natural language processing, 2020. arXiv:1910.03771. [20] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, S. Chintala, Pytorch: An imperative style, high-performance deep learning library, 2019. arXiv:1912.01703. [21] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, V. Stoyanov, Roberta: A robustly optimized bert pretraining approach, arXiv preprint arXiv:1907.11692 (2019). [22] Y. Wang, W. Wang, S. Joty, S. C. Hoi, Codet5: Identifier-aware unified pre-trained encoderdecoder models for code understanding and generation, in: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021, pp. 8696–8708.