In software development, code comment quality holds varying importance, highlighting the need for efective assessment methods. This study aims to improve the classification of comment utility by combining traditional annotated datasets with synthetic data. For augmentation, we utilized GPT-3.5-turbo to label additional comments. A baseline model using logistic regression was implemented for classification, yielding an F1 score of around 0.80, with little change upon adding synthetic data. The research explores both the potential and limitations of using synthetic data augmentation to assess code comment relevance.
In today’s digital era, software plays a crucial role in key sectors such as finance, healthcare, and transportation. Organizations must continually adapt to changing demands, resulting in frequent updates to existing software and the development of new applications. This continuous growth increases code complexity, which is necessary to support new features. Managing this large codebase is an essential phase of the Software Development Life Cycle (SDLC).
Rapid development cycles often lead to quick fixes, adding new code, or updating existing applications. These accelerated timelines can lead to less-than-optimal coding practices. As software evolves, associated documents, such as requirements and designs, may become outdated. In many cases, the original developers are not available for consultation. This underscores the need for structured, quality-driven processes in software development, with program comprehension being a key approach to maintaining existing code bases.
Given the dynamic nature of software, primary sources of reliable information become test execution traces, static code analysis, and code comments. This research focuses on code comments as indicators of software design for developers and automated tools. Comments provide context about the rationale and goals behind code, aiding in understanding and maintenance. However, comment quality varies, highlighting the need for automated tools to assess their usefulness.
One challenge in assessing code comment utility is the limited availability of well-annotated datasets that cover the diversity of comments across diferent programming contexts. To address this, innovative methods are needed to augment existing datasets and improve model performance on new, real-world comments. Our study combines manual data labeling with synthetic data augmentation using GPT-3.5turbo.
In this paper, we tackle a binary classification task for source code comments in the C programming language, categorizing them as ’Useful’ or ’Not Useful.’ Starting with a dataset of over 11,000 manually labeled comments, we use logistic regression as an initial classification model. We then augment this dataset with over 200 additional samples labeled using GPT to evaluate performance improvements.
Notably, the model’s performance remained consistent, maintaining an F1 score of 0.80 for both the original and augmented datasets.
This research contributes to the understanding of code comment utility classification by combining manual annotation with synthetic data augmentation. Our goal is to tackle current challenges and promote the development of adaptable models for the evolving landscape of software development.
The paper is organized as follows. Section 2 covers background work in comment classification. Section 3 details the task and dataset. Our methodology is discussed in Section 4. Results are presented in Section 5, and Section 6 concludes the study.
Understanding a program automatically is a well-known research area among people working in the
software domain. Numerous tools have been developed to aid in the extraction of knowledge from
software metadata, including elements such as runtime traces and structural attributes of code [
New programmers often rely on existing comments to comprehend code flow. However, not all
comments contribute efectively to program comprehension, necessitating a relevancy assessment
of source code comments prior to their use. Numerous researchers have focused on the automatic
classification of source code comments in terms of quality evaluation. For instance, Omal et al. [
With the advent of large language models [21], it is important to compare the quality assessment
of code comments by the standard models like GPT 3.5 or llama with the human interpretation. The
IRSE track at FIRE 2024 [22, 23] builds upon the methodologies proposed in [
# 1 2
/*cr to cr,nul*/ 3 /*convert minor status code (underlying routine error) to text*/
Code -10. int res = 0; -9. CURL *curl = NULL; -8. FILE *hd_src = NULL; -7. int hd; -6. struct_stat file_info; -5. CURLM *m = NULL; -4. int running; -3. start_test_timing(); -2. if(!libtest_arg2) { -1. #ifdef LIB529 /*test 529*/ 1. fprin -1. else /*cr to cr,nul*/ 1. newline = 0; 2. } 3. else { 4. if(test->rcount) { 5. c = test->rptr[0]; 6. test->rptr++; 7. test->rcount–; 8. } 9. else 10. break; -10. break; -9. } -8. gss_release_bufer(&min_stat, &status_string); -7. } -6. if(sizeof(buf) > len + 3) { -5. strcpy(buf + len, ".\n"); -4. len += 2; -3. } -2. msg_ctx = 0; -1. while(!msg_ctx) { /*con
In this paper, we tackle the task of developing a binary classification system to categorize source code comments as either useful or not useful. The system takes a comment and its associated code as input and outputs a label indicating the comment’s relevance. Classical machine learning algorithms, such as logistic regression, can be employed to perform this classification. The two categories of comments are: • Useful - The comment accurately describes or provides relevant information about the code. • Not Useful - The comment does not convey relevant information about the code.
Our dataset comprises over 11,000 C code-comment pairs annotated by a team of 14 annotators to label comments as useful or not. Additionally, we created a supplementary dataset by obtaining code-comment pairs from GitHub and labeling them using GPT. This secondary dataset, structured similarly to the original, serves as an augmentation for our main dataset.
We use logistic regression to implement the binary classification functionality. The system takes comments as well as surrounding code snippets as input. We create embeddings of each piece of code and the associated comment using a pre-trained Universal sentence encoder. The output of the embedding process is used to train both machine learning model. The training dataset consists of 80% data instances along with their labels. The rest is used for testing, in both experiments. The description of the model is discussed in the following section. 4.1. Logistic Regression We use logistic regression for the binary comment classification task which uses a logistic function to keep the regression output between 0 and 1. The logistic function is defined as follows: = + () =
1 1 + (− ) (1) (2)
The output of the linear regression equation (refer to equation 1) is passed to the logistic function (see equation 2). The probability value generated by the logistic function is used for binary class prediction based on the acceptance threshold. We keep the threshold value of 0.6 in favor of the useful comment class. We have a three-dimensional input feature extracted from each training instance which is passed to the regression function. The Cross entropy loss function is used during training for the hyper-parameter tuning.
We train our logistic regression model on both datasets. The original dataset has 11,452 samples and the GPT generated data has 233 samples. The first experiment uses only the original data and produces the following scores.
After augmenting the original dataset with the GPT generated data, the following results were seen.
The very slight change in the scores across metrics suggests that the newly generated data was practically indiferentiable from the original dataset, highlighting the validity of using GPT generated data for data augmentation.
This study investigated using synthetic data for code comment quality classification and established a logistic regression baseline with an F1 score of 0.80. Results indicate that synthetic augmentation did not improve classification, suggesting that traditional annotated data can still be highly efective in some cases. Future work could involve exploring more complex models or incorporating domain-specific knowledge.
During the preparation of this work, the author(s) used ChatGPT in order to: Grammar and spelling check. After using these tool(s)/service(s), the author(s) reviewed and edited the content as needed and take(s) full responsibility for the publication’s content. of the irse track at fire 2022: Information retrieval in software engineering., in: FIRE (Working Notes), 2022, pp. 1–9. [18] S. Majumdar, A. Bandyopadhyay, P. P. Das, P. Clough, S. Chattopadhyay, P. Majumder, Can we predict useful comments in source codes?-analysis of findings from information retrieval in software engineering track@ fire 2022, in: Proceedings of the 14th Annual Meeting of the Forum for Information Retrieval Evaluation, 2022, pp. 15–17. [19] S. Majumdar, P. P. Das, Smart knowledge transfer using google-like search, arXiv preprint arXiv:2308.06653 (2023). [20] P. Chakraborty, S. Dutta, D. K. Sanyal, S. Majumdar, P. P. Das, Bringing order to chaos: Conceptualizing a personal research knowledge graph for scientists., IEEE Data Eng. Bull. 46 (2023) 43–56. [21] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., Language models are few-shot learners, Advances in neural information processing systems 33 (2020) 1877–1901. [22] S. Paul, S. Majumdar, R. Shah, S. Das, M. Ghosh, D. Ganguly, G. Calikli, D. Sanyal, P. P. Das, P. D Clough, A. Bandyopadhyay, S. Chattopadhyay, Generative ai for code metadata quality assessment, in: Proceedings of the 16th Annual Meeting of the Forum for Information Retrieval Evaluation, 2024. [23] S. Paul, S. Majumdar, R. Shah, S. Das, M. Ghosh, D. Ganguly, G. Calikli, D. Sanyal, P. P. Das, P. D Clough, A. Bandyopadhyay, S. Chattopadhyay, Overview of the irse track at fire 2024: Information retrieval in software engineering, in: FIRE (Working Notes), 2024. [24] S. Paul, S. Majumdar, A. Bandyopadhyay, B. Dave, S. Chattopadhyay, P. Das, P. D. Clough, P. Majumder, Eficiency of large language models to scale up ground truth: Overview of the irse track at forum for information retrieval 2023, in: Proceedings of the 15th Annual Meeting of the Forum for Information Retrieval Evaluation, 2023, pp. 16–18. [25] S. Majumdar, S. Paul, D. Paul, A. Bandyopadhyay, S. Chattopadhyay, P. P. Das, P. D. Clough, P. Majumder, Generative ai for software metadata: Overview of the information retrieval in software engineering track at fire 2023, arXiv preprint arXiv:2311.03374 (2023). [26] S. Majumdar, A. Varshney, P. P. Das, P. D. Clough, S. Chattopadhyay, An efective low-dimensional software code representation using bert and elmo, in: 2022 IEEE 22nd International Conference on Software Quality, Reliability and Security (QRS), IEEE, 2022, pp. 763–774.