This paper proposes a framework for source code comment classification, which classify a comment based on its usefulness within the source code. This qualitative classification assists new developers in correctly comprehending the source code. We implement two binary classification mechanisms of source code comments based on two machine learning models: Naive Bayes and support vector machine. The classifier will classify each comment into two categories - useful and not useful. We extract comment features such as comment length, the position of comment within source code, and significant word ratio before training both models. We use a source code database of over 9000 instances written in C language in our work. Both models achieve an F1-score value of 0.632 and 0.765, respectively.
extraction process was systematically carried out using regular expressions, allowing us to obtain these code-comment pairs.
To evaluate the enhanced performance of our model, we leveraged a Large Language Model (LLM), specifically the chatgpt-3.5-turbo model, which played a pivotal role in generating labels for our data. We classified each code-comment pair as either "Useful" or "Not Useful" with the help of this LLM. This categorization was essential for assessing the efectiveness of our model in understanding and contextualizing code comments. The integration of ChatGPT not only expanded our dataset but also significantly contributed to an increase in accuracy, to 70.84%, further validating the robustness of our comment classification model. This augmentation process underscores the potential of combining state-of-the-art language models with machine learning techniques to enhance the performance and versatility of software development tools.
It’s worth noting that, while there are alternative options available for LLMs, such as BARD, LLaMA, and several others, we chose to utilize the OpenAI LLM for several reasons. One of the primary considerations was cost eficiency, as it ofers a cost-efective solution for our research needs. Additionally, the accessibility of OpenAI LLM’s API keys was a significant factor, facilitating our research eforts when compared to other alternatives.
The rest of the paper is organized as follows. Section 2 discusses the background work done in the domain of comment classification. Details of existing methods are discussed in section 3. We discuss the proposed method in section 4. Results are addressed in section 5. Section 6 concludes the paper.
Software metadata is integral to code maintenance and subsequent comprehension. A significant
number of tools [
In terms of mining code comments and assessing the quality, authors [
With the advent of large language models [19], it is important to compare the quality
assessment of code comments by the standard models like GPT 3.5 or llama with the human
interpretation. The IRSE track at FIRE 2023 [
Our previous experience with LLMs involved their application in generating data related to voting processes and project selections. Beyond label generation, we have also employed machine learning models for various classification tasks, encompassing binary classification, multiclass classification, and multilabel classification, utilizing the extensive dataset at our disposal.
In this section, we describe the task addressed in this paper. We aim to implement a binary classification system to classify source code comments into useful and not useful. The procedure takes a code comment with associated lines of code as input. The output will be a label such as useful or not useful for the corresponding comment, which helps developers comprehend the associated code. Classical machine learning algorithms such as Naive Bayes and SVM are used to develop the classification system. The two classes of source code comments can be described as follows: • Useful - The given comment is relevant to the corresponding source code. • Not Useful - The given comment is not relevant to the corresponding source code.
A dataset consisting of over 9000 code-comment pairs written in C language is used in our work. Each instance of data consists of comment text, a surrounding code snippet, and a label that specifies whether the comment is useful or not. The whole dataset is collected from GitHub and annotated by a team of 14 annotators. A sample seed data is illustrated in table 1. The development dataset consists of 8000 instances, and the test dataset consists of 1000 instances. The model is trained again using an additional dataset, consisting of 500 instances, generated using ChatGPT. A sample LLM-generated data is illustrated in table 2.
We try two machine learning models - Multinomial Naive Bayes and support vector machine (SVM) to implement the binary classification functionality. The system takes comments as well as surrounding code snippets as input. We extract features such as comment length, the position of comment within source code, and significant word ratio[ 17] from the given input. The output of the feature extraction process is used to train both machine learning models. The training dataset consists of 8591 data instances (including an additional 500 data instances generated using ChatGPT) along with their labels. Among them, 3791 data instances are labeled as not useful and 4796 data instances are marked as useful. The description of each model is discussed in the following section.
Multinomial Naive Bayes is a classification algorithm used for text data. It assumes that features (e.g., word counts) are conditionally independent given the class label.
# 1 2
/*cr to cr,nul*/ 3 /*convert minor status code (underlying routine error) to text*/ -10. int res = 0; -9. CURL *curl = NULL; -8. FILE *hd_src = NULL; -7. int hd; -6. struct_stat file_info; -5. CURLM *m = NULL; -4. int running; -3. start_test_timing(); -2. if(!libtest_arg2) { -1. #ifdef LIB529 /*test 529*/ 1. fprin -1. else /*cr to cr,nul*/ 1. newline = 0; 2. } 3. else { 4. if(test->rcount) { 5. c = test->rptr[0]; 6. test->rptr++; 7. test->rcount–; 8. } 9. else 10. break; -10. break; -9. } -8. gss_release_bufer(&min_stat, &status_string); -7. } -6. if(sizeof(buf) > len + 3) { -5. strcpy(buf + len, ".\n"); -4. len += 2; -3. } -2. msg_ctx = 0; -1. while(!msg_ctx) { /*con
In the probability model, We want to find the class label that maximizes the posterior probability (|), where is a comment.
(|) = () · (|) () In text classification, we use a multinomial distribution for the likelihood model: #
{ "nosombrero",
&nosombrero, PL_OPT_BOOL, "nosombrero", "No sombrero plot" }, free( img ); *width = w; *height = h; *img_f = imf; return 0; } if ( f_name ) save_plot( f_name ); } plFree2dGrid( z, XDIM, YDIM );
Where is the count of each term in the comment.
To classify a comment, calculate ⎛
argmax ⎝log( ()) + ∑︁ log( ( |))⎠ =1 ⎞ for each class and choose the class with the highest probability.
Tune the smoothing parameter ( ) and class prior probabilities ( ()) during training. For binary classification, we set a threshold on the posterior probabilities. Multinomial Naive Bayes is efective for text classification tasks, especially when features represent word counts.
We have incorporated a Support Vector Machine (SVM) model into our binary classification task. The decision boundary is determined by the linear function outlined in Equation 1. In this setup, instances with an output exceeding 1 are assigned to one class, while those with an output less than -1 are categorized into the other class. The training of the SVM model involves the utilization of the hinge loss function, as shown below.
(, , ) = 0,
We train both models in a system having an Intel i5 processor and 16GB RAM. We test both models using the test dataset. The test dataset consists of 1001 data instances, among which 719 data instances are labeled as not useful and 282 instances are marked as useful. Our Multinomial Naive Bayes model has been tested on this dataset and achieved an F1-score value of 0.632 (without ChatGPT-generated data) and 0.640 (with the augmented model). Similarly, the SVM model achieves a 0.765 F1-score value (without ChatGPT-generated data) and 0.776 (with the augmented model). Both models achieve high recall values of 0.652 and 0.763 without the LLM-generated data and 0.659 and 0.774, respectively with the augmented model. It shows that both models correctly predict useful comments in a better way. Both models achieve lower precision, such as 0.608 and 0.763, compared to the recall value. Apart from this, our model does not use any qualitative feature, which may be important to understand the usefulness of a comment within a source code. Using these qualitative features may increase the overall accuracy of the binary classification.
This paper has addressed a binary classification problem in the domain of source code comment classification. The classification has been done based on the usefulness of the comment present within a source code written in C language. We have used two machine learning models, Multinomial Naive Bayes, and support vector machine, to implement the binary classification task. We extracted three structural features: the length of the comment, the position of the comment within the source code, and the significant word ratio from each data instance. Both models have been trained using a training dataset with more than 8,000 data instances. Hinge loss has been used during hyper-parameter tuning for the SVM model. The models are tested on a test dataset of 1000 data instances. The Multinomial Naive Bayes model has been tested on this dataset and achieved an F1-score value of 0.632 (without ChatGPT-generated data) and 0.640 (with the augmented model), and the SVM model achieves a 0.765 F1-score value (without ChatGPT-generated data) and 0.776 (with the augmented model). Currently, we are using structural features for the classification task, which may not be suficient for the qualitative analysis of the source code comments. In the future, we will use some qualitative features of the comment, which may increase the accuracy of the comment classification task. [10] Y. Wang, H. Le, A. D. Gotmare, N. D. Bui, J. Li, S. C. Hoi, Codet5+: Open code large language models for code understanding and generation, arXiv preprint arXiv:2305.07922 (2023). [11] D. Steidl, B. Hummel, E. Juergens, Quality analysis of source code comments, International
Conference on Program Comprehension (ICPC), IEEE, 2013, pp. 83–92. [12] S. Majumdar, A. Bandyopadhyay, P. P. Das, P. Clough, S. Chattopadhyay, P. Majumder, Can we predict useful comments in source codes?-analysis of findings from information retrieval in software engineering track@ fire 2022, in: Proceedings of the 14th Annual Meeting of the Forum for Information Retrieval Evaluation, 2022, pp. 15–17. [13] S. Majumdar, A. Bandyopadhyay, S. Chattopadhyay, P. P. Das, P. D. Clough, P. Majumder, Overview of the irse track at fire 2022: Information retrieval in software engineering, in: Forum for Information Retrieval Evaluation, ACM, 2022. [14] J. L. Freitas, D. da Cruz, P. R. Henriques, A comment analysis approach for program comprehension, Annual Software Engineering Workshop (SEW), IEEE, 2012, pp. 11–20. [15] M. M. Rahman, C. K. Roy, R. G. Kula, Predicting usefulness of code review comments using textual features and developer experience, International Conference on Mining Software Repositories (MSR), IEEE, 2017, pp. 215–226. [16] A. Bosu, M. Greiler, C. Bird, Characteristics of useful code reviews: An empirical study at microsoft, Working Conference on Mining Software Repositories, IEEE, 2015, pp. 146–156. [17] S. Majumdar, A. Bansal, P. P. Das, P. D. Clough, K. Datta, S. K. Ghosh, Automated evaluation of comments to aid software maintenance, Journal of Software: Evolution and Process 34 (2022) e2463. [18] S. Majumdar, S. Papdeja, P. P. Das, S. K. Ghosh, Comment-mine—a semantic search approach to program comprehension from code comments, in: Advanced Computing and Systems for Security, Springer, 2020, pp. 29–42. [19] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., Language models are few-shot learners, Advances in neural information processing systems 33 (2020) 1877–1901. [20] S. Majumdar, A. Varshney, P. P. Das, P. D. Clough, S. Chattopadhyay, An efective lowdimensional software code representation using bert and elmo, in: 2022 IEEE 22nd International Conference on Software Quality, Reliability and Security (QRS), IEEE, 2022, pp. 763–774.