Code Comments increase the readability of the surrounding code if they highlight concepts that are not evident from the source code itself. Hence, evaluation of the quality of code comments is important to de-clutter large code bases and remove not useful comments. The Information Retrieval in Software Engineering (IRSE) track aims to develop solutions for automated evaluation of code comments. In this track, there is a binary classification task to classify comments as useful and not useful. The dataset consists of 9048 code comments and surrounding code snippet pairs extracted from open source github C based projects. Overall 34 experiments have been submitted by 11 teams from various universities and software companies. The submissions have been evaluated quantitatively using the F1-Score and qualitatively based on the type of features developed, the supervised learning model used and their corresponding hyper-parameters. The best performing architectures mostly have employed transformer architectures coupled with a software development related embedding space.
Assessing comment quality can help to de-clutter code bases and subsequently improve code
maintainability. Comments can significantly help to read and comprehend code if they are
consistent and informative. Comment analysis approaches have mainly focused on detecting
inconsistent comments [
Several approaches have been proposed to classify comments based on explicit syntactic
information, such as the presence of specific tags (e.g., @param, @deprecated, etc.), words,
and symbols; or implicit details, such as the type of associated code construct, length of the
comment, parts of speech (POS) and dependency relations of comment words or the cosine
similarity of vector representation of words in code-comment snippets [
These approaches do not target comment quality evaluation based on the interpretation of the information contained in comments. The syntactic methods used for comment classification need to be augmented so as to extract the semantics of a comment in order to develop an overall quality assessment model.
Further, the perception of quality in terms of the ’usefulness’ of the information contained in
comments is relative and hence is perceived diferently based on the context. Bosu et al. [
Majumdar et al. [
The IRSE track of FIRE 2022, extends the work in [
Several approaches exist that attempt to assess the quality of comments by detecting inconsistencies with source code or by classifying comments based on syntactic properties.
Tan et al. [
Ying et al. [
Comment quality evaluation: Steidl et al. [
The available approaches mostly target to evaluate the quality of the comments by mining
for irrelevant words and phrases, coupled with the repetitiveness in the surrounding constructs.
The context based on which the quality is defined is essential, like Rahman et al. [
The following section outlines the task descriptions and the characteristics of the dataset.
Comment Classification : A binary classification task to classify source code comments as Useful or Not Useful for a given comment and associated code pair as input.
Input: A code comment with surrounding code snippet (written in C) Output: A label (Useful or Not Useful) that characterises whether the comment helps developers comprehend the associated code
Therefore in this classification task, the output is based on whether the information contained in the comment is relevant and would help to comprehend the surrounding code, i.e., it is useful.
Useful: Comments have suficient software development concept → Comment is Relevant, and these concepts are not mostly present in the surrounding code → Comment is not Redundant, hence the comment is Useful
Not Useful: Comments have suficient software development concept → Comment is Relevant, and these concepts are mostly present in the surrounding code → Comment is Redundant, hence the comment is Not Useful
It may also be the case that comments do not contain suficient software development concepts → Comment is Not Relevant, hence the comment is Not Useful.
It is left to the participants to decide on the threshold value for how many concepts retrieved make a comment relevant or how many matches with surrounding code make a comment redundant.
The notion of relevant comments refers to those that developers perceive as important in comprehending the associated or surrounding lines of code. These concepts are related to the outline of the algorithm, data-structure descriptions, mapping to user interface details, possible exceptions, version details, etc. In the below examples, the comments highlight useful details about the input data to the function, which is not evident from the associated code itself. 1 2 # works on a two dimensional data matrix (each of size 8) generated from the light rider bot module int* flood_fill(self, position, visited) {...} 1 /* uses png_calloc defined in pngriv.h*/ 2 PNG_FUNCTION(png_voidp,PNGAPI 3 png_calloc,(png_const_structrp 4 png_ptr, png_alloc_size_t size),PNG_ALLOCATED) 5 { }
Therefore, a relevant comment provides more information for the surrounding code and subsequently aids better comprehension that can improve software maintenance.
However, in the example below, even if the comment contains relevant information, it is already available in the associated code rendering the comment redundant. 1 // PHP Shutdown method to destroy the global php hash map, using zend hash api’s 2 PHP_MSHUTDOWN_FUNCTION(hash) { ... zend_hash_destroy(&php_hashtable); . }
We select 5 projects from Github and use the modified random sampling approach of Cochran [15] to sample source files with equal probability and hence provide an unbiased representation of the population (C code files with comments). We gather a total of 318 files with 20,206 comments.
Ground Truth Generation: For every comment, a label (Useful or Not Useful) has been generated by a team of 14 annotators. Every comment has been annotated by 2 annotators with a kappa ( ) value of 0.734 (Cohen’s metric [16]). The annotation process has been supervised through weekly meetings and brainstorming sessions and peer review. Out of the total 16,000 comments, 2,285 comments were annotated by every individual annotator. A total of 156 man-hours were required to complete the annotation process.
For the IRSE track, we use a set of 9048 comments (from Github) with comment text, surrounding code snippets, and a label that specifies whether the comment is useful or not. Sample data has been characterised in Table 1.
• The development dataset contains 8048 rows of comment text, surrounding code snippets, and labels (Useful and Not useful). • The test dataset contains 1,000 rows of comment text, surrounding code snippets, and labels (Useful and Not useful).
In its first edition, IRSE 2022 received a total of 34 experiments from 11 teams. As this track is related to software maintenance, we received participation from companies like Amazon and Mentor Graphics along with several research labs of educational institutes.
The various teams with the details of their submissions are characterised in Table 3.
Evaluation Procedure: Candidates were asked to submit predicted labels (’useful’ or ’not useful’) for every data point in the test set of 1000 comments. This was used by our script to generate the precision, recall, and the F1-Score (Macro) using the annotated (golden) labels.
Features: Apart from evaluating the prediction metrics, we analysed the types of features the teams have used to devise the machine learning pipeline. The teams have performed routine pre-processing and have retained the significant words or letters only for both the code and comment pairs. Further, some of the teams have also used morphological features of a comment like a length, significant words ratio, parts of speech characteristics, or occurrence of words from an enumerated set as textual features. To correlate code and comment and detect redundancies, the teams mostly used grep-like string match to find similar words.
Vector Space Representations: Code and comments belong to diferent semantic granularity which is unified by a vector space representation. The participants have used various pre-trained embeddings to generate vectors for the words like those based on one hot encoding, tf-idf based, word2vec or context aware like ELMo and BERT. Each of the employed embedding models are trained or finetuned using software development corpora.
The F1-Scores have been analysed based on the machine learning models used, the features and pre-trained embeddings for projection to vector space.
The dataset provided was balanced and had 4015 useful comments and 4033 not useful comments. The textual features used by the teams were mostly related to mining specific words and determining significant words. Similarly, almost all teams used string matching to locate overlapping words between code and comment.
Significant diferences were analysed in terms of the pre-trained embeddings used and the machine learning models which contributed to the improvement in the F1-Score.
Machine Learning Architectures: The best F1 score was obtained using GPT architecture, although it is resource critical and can be aforded by software companies. The other machine learning models commonly used are recurrent neural networks, support vector machines, random forest and logistic regression with textual and correlation features. The F1-Score obtained using recurrent neural networks, support vector machines are comparable to the ones obtained from BERT. This is because the dataset is balanced and also due to the use of various textual features apart from numerical vectors (as features).
Pre-Trained Embeddings: Both context-aware and context-independent pre-trained
embeddings trained from scratch or finetuned with software development concepts have been used.
Results are better with the recently released codeBERT pre-trained embeddings [17] where
natural language and programming language pairs are used from software projects of diferent
domains to train using masked language modeling. Comparable results have been obtained
by using CodeELMo [
The IRSE track in its first edition empirically investigates various approaches in a machinelearning framework for automated comment quality evaluation. The comments are evaluated based on whether they contain information that can aid in understanding the surrounding code. A total of 11 teams participated and submitted 34 experiments that used various types of machine learning models, embedding spaces, and features. The best F1-Score of 90.8 was reported by experiments conducted using GPT-2 architecture with textual and numerical features from CodeBERT vector space embeddings, to classify comments as ’useful’ and ’not useful’. [13] M. Liu, Y. Yang, X. Peng, C. Wang, C. Zhao, X. Wang, S. Xing, Learning based and context aware non-informative comment detection, International Conference on Software Maintenance and Evolution (ICSME), IEEE, 2020, pp. 866–867. [14] M. Alver, N. Batada, Jabref: Cross-platform citation and reference management software,
Open Source, 2003. https://github.com/JabRef/jabref, Last Accessed: December 12, 2020. [15] J. Kotrlik, C. Higgins, Organizational research: Determining appropriate sample size in survey research, Information technology, learning, and performance journal 19 (2001) 43. [16] N. Gisev, et al., Interrater agreement and interrater reliability: key concepts, approaches, and applications, Research in Social and Administrative Pharmacy 9 (2013) 330–338. [17] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805 (2018).