In the domain of software development, the utility of code comments varies, necessitating methodologies capable of distinguishing their substantive value. This study delves into enhancing code comment usefulness classification by adopting a hybrid approach, combining manually tagged datasets with synthetic data augmentation. For augmentation, we employed GPT-3.5-turbo, a state-of-the-art language model, to label additional comment examples. A baseline model was established using random forests for classification. Interestingly, despite the data augmentation, the model performance remained consistent, with an F1 score of approximately 0.79 both before and after the synthetic data integration. This research ofers insights into the potential and limitations of synthetic data augmentation in the realm of code comment usefulness classification.
Developers often need to fix bugs, develop new source code, or upgrade already deployed
applications on a reduced time frame. This can lead to improper coding practices. As the
software changes dynamically, the documentation, such as requirement specification,
highlevel design etc., becomes outdated and incomplete, and the knowledge transfer process or
help from the earlier developers is often unobtainable. These types of situations demand a
systematic quality-controlled development process. Automated Program comprehension is one
such method of maintaining existing source code in a better way.[
Since the software design of a codebase is a moving target, the real source of truth are the traces of test execution, static analysis of the programs and, to a large extent, code comments. This paper focuses on code comments as information about the program design - both for developers, and for automated program comprehension. Code comments ofer deep insights into the logic, decisions, and intentions behind the code, thereby aiding in code comprehension, maintenance, and debugging. However, not all comments are equally informative or useful, creating a compelling need to develop automated methods to classify the usefulness of code comments efectively.
A common hurdle across studies for code comment usefulness is the scarcity of extensive, well-annotated datasets that encompass the diverse nature of comments in various programming contexts. This necessitates innovating strategies to enhance available data for improved model generalization on unseen, real-world comments. Recognizing this gap, we aim to integrates manual annotation with synthetic data augmentation. We employ GPT-3.5-turbo, a state-of-theart language model, to label code comment samples scraped from open source code bases.
In this paper, we propose a binary classification task to understand the source code comments present in a program written in C language. We classify each comment into two classes - Useful and Not Useful. We start with a training data set of over 11000 manually-annotated samples. We use Random Forests to create a baseline for comment classification. Then, we augment the data with over 200 GPT-labelled samples to examine the improvement in performance. We observed that the model performance remained consistent, with an F1 score of 0.79 both for the baseline and model trained on augmented data.
By exploring the intricate interplay between manual annotation and synthetic data augmentation, this study aspires to contribute a novel perspective to the existing body of knowledge on code comment usefulness classification. It strives to ofer an innovative solution to the prevailing challenges in the field, inspiring further exploration and development of robust, scalable models that can seamlessly adapt to the dynamic, evolving landscape of software development.
The rest of the paper is organized as follows. Section 2 discusses the background work done in the domain of comment classification. The task and dataset are described in 3. Our methodology is discussed in section 4. Results are addressed in section 5. Section 6 concludes the paper.
Software metadata is integral to code maintenance and subsequent comprehension. A significant
number of tools [
In terms of mining code comments and assessing the quality, authors [
With the advent of large language models [19], it is important to compare the quality assessment of code comments by the standard models like GPT 3.5 or llama with the human interpretation. The IRSE track at FIRE 2023 [20] extends the approach proposed in [17] to explore various vector space models [21] and features for binary classification and evaluation of comments in the context of their use in understanding the code. This track also compares the performance of the prediction model with the inclusion of the GPT-generated labels for the quality of code and comment snippets extracted from open-source software.
In this section, we have described the task addressed in this paper. We aim to implement a binary classification system to classify source code comments into useful and not useful. The procedure takes a code comment with associated lines of code as input. The output will be a label such as useful or not useful for the corresponding comment, which helps developers comprehend the associated code. Classical machine learning algorithms such as random forests can be used to develop the classification system. The two classes of source code comments can be described as follows: • Useful - The given comment is relevant to the corresponding source code. • Not Useful - The given comment is not relevant to the corresponding source code.
A dataset consisting of over 11000 code-comment pairs written in C language is used in our work. Each instance of data consists of comment text, a surrounding code snippet, and a label that specifies whether the comment is useful or not. The whole dataset is collected from GitHub and annotated by a team of 14 annotators. A sample data is illustrated in table 1.
There is another similar dataset that is created and used in this work. That dataset is created by getting code-comment pairs from Github, and the label of useful or not useful was given by GPT. This dataset has a similar structure to the original dataset, and is used to augment the original dataset later on.
We use random forests to implement the binary classification functionality. The system takes comments as well as surrounding code snippets as input. We create embeddings of each piece of code and the associated comment using a pre-trained Universal sentence encoder. The output of the embedding process is used to train both machine learning model. The training dataset consists of 80% data instances along with their labels. The rest is used for testing, in both experiments. The description of the model is discussed in the following section. 4.1. Random Forest Random Forest (RF) is employed for binary comment classification in our study, leveraging an ensemble of decision trees to improve the model’s predictive accuracy and control overfitting. The basic premise of Random Forest is to generate numerous decision trees during training, and output the class that is the mode of the classes output by individual trees during the prediction phase.
Each tree in the Random Forest is constructed as follows: 1. A subset of the training data is selected with replacement (bootstrap sample). # 1 2
/*cr to cr,nul*/ 3 /*convert minor status code (underlying routine error) to text*/ 2. A subset of features is randomly chosen at each node. 3. The best split based on a criterion (such as Gini impurity or entropy) is chosen to partition the data. 4. Steps 2 and 3 are repeated at each node until the tree is fully grown.
The classification decision is obtained by aggregating the predictions made by all trees in the forest through majority voting: () = majority ({()}=1) (1) where () denotes the prediction of the -th tree for the input vector , and is the number of trees in the forest. A threshold of 0.5 is conventionally used for binary classification, although this can be adjusted to favor the useful comment class, similar to the threshold adjustment in random forests.
Random Forest inherently handles multi-dimensional feature space and does not require the feature scaling. It handles missing values by choosing the split that minimizes the impurity among non-missing values, hence imputing the missing ones based on the majority class or mean/mode value.
During training, the out-of-bag (OOB) error, computed on the data not used in bootstrap samples, serves as an unbiased estimate of the generalization error and can be employed for hyper-parameter tuning.
We train our random forests model on both datasets. The original dataset has 11,452 samples and the GPT generated data has 233 samples. The first experiment uses only the original data and produces the following scores.
After augmenting the original dataset with the GPT generated data, the following results were seen.
Accuracy Original Dataset 81.05630729 Augmented Dataset 81.0012837
Precision 0.790190835 0.790785274
Recall 0.801640488 0.801383776
F1 Score 0.794906015 0.795175139
The very slight change in the scores across metrics suggests that the newly generated data was practically indiferentiable from the original dataset, highlighting the validity of using GPT generated data for data augmentation.
This paper has addressed a binary classification problem in the domain of source code comment classification. The classification has been done based on the usefulness of the comment present within a source code written in C language. We have used random forests as our base classiifcation method. We conducted two experiments, one with the original dataset and another with the original dataset plus the synthetic GPT generated data. The similar results in both cases show that the synthetic data falls in line with the original dataset, and how synthetic data creation can help in efectively increasing data volume required for training models. The synthetic data’s correctness as compared to the original dataset is proven by the results shown above. Synthetic data generation can help a lot with data augmentation, finding its use in many pipelines. Repositories (MSR), IEEE, 2017, pp. 215–226. [16] A. Bosu, M. Greiler, C. Bird, Characteristics of useful code reviews: An empirical study at microsoft, Working Conference on Mining Software Repositories, IEEE, 2015, pp. 146–156. [17] S. Majumdar, A. Bansal, P. P. Das, P. D. Clough, K. Datta, S. K. Ghosh, Automated evaluation of comments to aid software maintenance, Journal of Software: Evolution and Process 34 (2022) e2463. [18] S. Majumdar, S. Papdeja, P. P. Das, S. K. Ghosh, Comment-mine—a semantic search approach to program comprehension from code comments, in: Advanced Computing and Systems for Security, Springer, 2020, pp. 29–42. [19] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., Language models are few-shot learners, Advances in neural information processing systems 33 (2020) 1877–1901. [20] S. Majumdar, S. Paul, D. Paul, A. Bandyopadhyay, B. Dave, S. Chattopadhyay, P. P. Das, P. D.
Clough, P. Majumder, Generative ai for software metadata: Overview of the information retrieval in software engineering track at fire 2023, in: Forum for Information Retrieval Evaluation, ACM, 2023. [21] S. Majumdar, A. Varshney, P. P. Das, P. D. Clough, S. Chattopadhyay, An efective lowdimensional software code representation using bert and elmo, in: 2022 IEEE 22nd International Conference on Software Quality, Reliability and Security (QRS), IEEE, 2022, pp. 763–774.