Source code is usually accompanied by comments which help in improving code comprehension. However, not all comments are helpful in this respect, with some comments being redundant, some being unclear which results in a poorer code readability. Sifting through large volumes of code to identify such comments manually being tedious, the task of automatically evaluating the usefulness of a comment in context of the code it lies in can add value. In this work, we evaluate the performance of various pretrained transformer encoders to solve this task. We demonstrate decent performance of a few models alongside abnormally high performance obtained by a few other pretrained architectures.
Programmers rely on source comments to understand the functioning of code. It represents a way of documenting the code, with the comments explaining the logic behind the diferent parts of the code, the expected behavior, inputs, and outputs. Code comments not only help programmers working with the code, but they also aid in aspects related to software maintenance and reusability. However, not all comments are of high quality. Some examples include inconsistent comments, redundant or superfluous comments explaining the code rather than its rationale, and irrelevant comments. Such comments reduce the comprehensibility and readability of the code and are thus not desirable.
This motivates the need for a way to evaluate the quality of code comments given the piece
of code they accompany. Diferent methods based on have been proposed in the literature for
solving this problem, such as those based on heuristics and subjective and objective quality
attributes [
Unlike previous work, which makes use of custom-trained embeddings and explicit features, we investigate the eficacy of pretrained language models in this particular task. Large pretrained models have achieved SoTA performance across many downstream tasks in diferent domains. Models such as CodeBERT [6] have also emerged recently, explicitly designed for software engineering tasks and capable of understanding both source code semantics and natural language semantics.
Seven language models were chosen to experiment with for this task. These are - ALBERT [7], BERT [8], CodeBERT [6], DeBERTa [9], MPNet [10], RoBERTa [11], and XLNet [12]. Of-theshelf pretrained versions of these models were then finetuned on the training set and optimized using the dev set. We also used a Transformer [13] encoder initialized with random weights for comparison against the pretrained models. Finally, the results were reported according to the performance on the test set. This method benefits from being agnostic to language, with the pretrained models capable of working with code in any language.
We have open-sourced our code for replication and further experimentation1.
3.1. Data The provided dataset for the shared task consisted of separate splits for training and evaluation (test) of sizes 8000 and 1001, respectively. However, duplicates were found in the samples provided, and some samples in the test data were present in the training set. Hence, we first removed all the duplicates from both sets and cleaned the test to remove all the samples already preset in the train set to not tamper with the integrity of the evaluation. We also separated 10% of the data from the deduplicated train set as the validation (dev) set for saving the best checkpoint during training.
# samples train 5354 dev 595 test 678
Following data duplication and test cleaning, the sizes of the data splits obtained are indicated in Table 1.
For each of the experiments performed, base versions of the model were trained for 5 epochs on a batch size of 4 for training, and gradients were accumulated for two steps, thus providing an efective batch size of 8. The learning rate was increased till 40% of the training steps peaking at 6e-5, followed by a cosine decay. Validation was performed at the end of each training epoch using the dev set, and the best-performing model checkpoint was saved based on the F1 score. The best checkpoint thus determined was used to evaluate the test data.
Model Transformer
BERT ALBERT XLNet MPNet
The results of the code comment usefulness classification on the deduplicated and cleaned test data are in Table 2. In this table, we have only included models that gave a decent performance on the dataset. Other models that gave an anamolously high performance are included separately in Section 4.1. Evaluation was done using Accuracy, F1, Matthew’s Correlation Coeficient (MCC) and ROC-AUC metrics 2.
The BERT model was performing only slightly ahead of the transformer encoder trained from scratch. There were diferences in the training process, however, with the BERT model achieving the optimal performance within 3 epochs, but the transformer encoder showed a gradual increase in performance, thus distinguishing the efect of pretraining. XLNet and MPNet - the two permutation language modeling-based architectures - performed gave a decent performance with the former having an edge in performance over the latter. ALBERT performed close to BERT in accuracy but showed a significant gap w.r.t. the other three metrics.
Model RoBERTa CodeBERT DeBERTa
The transformer architectures that gave an unusually high performance are shown in Table 3. As can be seen, the models achieved an almost-perfect performance. Although CodeBERT and RoBERTa tied up in the performance on the test set, CodeBERT achieved optimal performance by the 3rd epoch in training while RoBERTa at the 4th epoch, CodeBERT being a model pretrained on the RoBERTa architecture on programming language data, thus highlighting the faster convergence achieved due to domain-specific pretraining. Both the models maintained accuracy of 100% on the dev set in the subsequent epochs. The closest performance to these models was achieved by DeBERTa. An ROC-AUC score of 1 indicates an almost perfect confidence achieved in the classification task, which indicates that the models have been easily able to fit almost perfectly on the distribution of the data, and the test and train data distributions do not have any significant variation in the distribution. Such an anomalous behaviour might have resulted from the annotations for the train and test set being done on the basis of a rule-set, and the models being able to learn those rules.
In this work, we evaluated the performance of diferent pretrained transformer architectures on the downstream task of code-comment usefulness prediction. While some of the models gave a decent performance, some of them performed abnormally well. The latter aspect needs to be investigated further by experimenting with diferent sets of data prepared over diferent programming languages, and diverse set of annotations. A further investigation of the mechanics behind some architectures performing exceedingly well on the dataset - which does not have a direct overlap between its train and evaluation splits - can add value to the efectiveness of a transformer-based solution to the problem. 2The results submitted to the shared task were based on the actual test set, and not the cleaned one - the values for accuracy and F1, in that case, is for these two models being 99.7% and 99.46% respectively. The other model results also slightly difer from the ones in the table due to this reason.