This paper reports a detailed analysis on the viability of a classification framework which can classify a comment based on its usefulness within the source code. This classification will be helpful for new developers for correctly comprehending the source code. Three machine learning models: such as logistic regression, support vector machine, and multinomial naive Bayes are trained using an initial dataset called seed dataset. Each comment is classified into one of the two categories - useful and not useful. An accuracy of 82.92%, 83.92%, and 50.75% respectively is achieved from the initial training of three models. The dataset is then augmented using a new set of data extracted from several online resources. The corresponding class for the new set are generated using chatGPT large language model (LLM). The augmented dataset is then again used to train those three machine learning models. It is observed that for the new augmented dataset, the accuracy drops down for all three models due to inclusion of noise and biasness owing to the LLM generated dataset.
Software is emerging as the backbone of modern technology as they warrant many promising applications owing to their integration into electronics and appliances. They are simplifying the challenges of our daily life and making it easier in all aspects. For example, GPS software facilitates driving from one place to another. Constant modification of existing software and building of new software is the key of improving the software functionality which leads to an increase in the source code. Maintaining this large amount of source code is a crucial phase of Software Development Life Cycle (SDLC). In most of the cases, developers face numerous challenges during this source code maintenance including comprehending large code base in short period of time, outdated and incomplete required documents, unavailability of knowledge from previous developer to name a few.
This type of scenario can be tackled by following systematic process flow. New developers
generally have the source code, sample test cases, requirement documents, and a debugger to
implement new functionality. For further modification of the code, the developer must understand
the existing source code. So, they repeatedly run the current application on the sample test cases
to identify execution patterns, understand the design, and to comprehend the program. But this
whole process is time-hungry, efort-intensive, monotonous and sometimes becomes
unmanageable. To overcome those bottlenecks, developers often follow shortcut method which further
introduce errors that are dificult to filter. This results in the degradation of software quality
and developer’s eficiency. These types of situations demand a systematic quality-controlled
development process for ease of use by the developers. Program comprehension is one such
process for maintaining existing source code in a better way. This reverse engineering process
is beneficial for reuse, inspection, maintenance, and many others in the context of software
engineering[
Inserting comments within program is incumbent for better understanding of the sourcecode. Many developers are working on the same code-base and describe diferently while inserting comments. Diferent description of same code-base decreases the program’s readability. Therefore, a standardized procedure of writing codes and comments is imperative in order to enhance the readability. Still this approach is not efective for understanding an already registered code. An intuitive understanding of source code comments may be a good idea to solve the readability program. In recent years, researchers are exploring this domain to develop new applications aiming to enhance the eficiency of new programmers through understanding the existing code.
In this paper, we present a classification framework applied on a dataset of code and comment pairs written in C language. The work is done in three stages - Training of classification framework based on seed dataset, augment the dataset volume using large language model, and again train the classification framework using the augmented dataset. In the first stage, the framework takes code and comment pair as an input and classify them into one of the two classes - Useful and Not Useful. In this case, logistic regression method, support vector machine (SVM), and multinomial naive Bayes techniques are employed for comment classification. A training dataset of 9000 samples and test dataset of 1001 samples are used for this purpose. The model is validated using five fold cross-validation process. We employ linear kernel for the SVM strategy and L2 regularization method for the logistic regression strategy. In the next stage, another set of code-comment pair is gathered from online sources such as github. Consequently, chatGPT-4 large language model is used to categorise the newly gathered code-comment pair into two classes- useful and not-useful. This generated dataset is augmented with previous seed dataset. The newly generated dataset is used to train these classification frameworks again. A small reduction in all the F1 score values and accuracies is obtained which can be due to noise inclusion as part of the newly generated dataset.
The rest of the paper is organized as follows. Section 2 discusses the background work done in the domain of comment classification. Details of existing methods are discussed in section 3. We discuss the proposed method in section 4. Results are addressed in section 5. Section 6 concludes the paper.
Software metadata [
In the realm of mining code comments and assessing their quality, several authors have
conducted research. Steidl et al. [
In light of the emergence of large language models, such as GPT-3.5 or llama [19], it becomes
crucial to assess the quality of code comments and compare them to human interpretation. The
IRSE track at FIRE 2023 [20] expands upon the approach presented in a prior work [
The task of implementing binary classification framework is accomplished in three consecutive steps. Those are designing of classification framework with seed dataset, augmenting the seed dataset volume using large language model and train the same framework using the new augmented dataset. The source code and comments pairs are classified into two classes useful and not useful using the trained framework. The procedure takes a comment description with associated lines of code as input and generates a label such as useful or not useful corresponding to each code-comment pair. The classification system was developed using classical machine learning models such as logistic regression, naive Bayes, and SVM.
• Useful - The specified comment is appropriate for the corresponding source code. • Not Useful - The specified comment is not appropriate for the corresponding source code.
The seed dataset has 9000 code-comment pairs which are written in C language. Each data contains comment text, surrounding code snippet, and a label that describes its usefullness. The whole dataset is gathered from the GitHub and is annotated with the help of a 14 annotators group. An example of dataset is presented in table 1. Another set of code-comment pairs is collected from a diferent online resources and is augmented with the above mentioned dataset. The set of code-comment pairs is categorized into two above-mentioned classes using a large language model. This newly generated dataset in then added with the seed dataset.
The classification model is then again trained using this augmented dataset in order to understand the efect of augmentation. Diferent factors are analysed including noise inclusion, distribution of dataset, which cause the change in accuracy during the training of classification framework with augmented dataset.
A binary classification system was implemented with the help of three machine learning models - logistic regression, support vector machine, and multinomial naïve Bayes. The system took both the source code and corresponding comments as input. Considering the task criteria we did not use any deep learning frameworks in our classification system. The comments are first tokenized using English word lemmatizer. Following that the set of tokens are vectorized using TF-IDF vectorizer. The TF-IDF matrix generated from the vectorization step along with class labels was used as feature fed into the classification models. These models are trained using the primary seed data and tested using test dataset. The unlikeliness in training data was controlled using five-fold cross validation. We will briefly elucidate each machine learning models in the subsequent subsections.
We use logistic regression for the binary comment classification task where a logistic function is used in order to keep the output between 0 and 1. The logistic function is defined as follows: = + () =
1 1 + (− )
Equation (1) is referred as the linear regression equation whose output (Z) is passed to the logistic function. The logistic function is defined in equation 2. The binary class is predicted from the probability value generated by the logistic function based on the acceptance threshold. The threshold value is kept to 0.6 which is in favor of the useful comment class. A three-dimensional input feature is extracted from each training instance which is passed to the regression function. During training the Cross entropy loss function is used for the hyper-parameter tuning.
In the next step, a support vector machine model is implemented for binary classification task. Classification is done based on the output of the linear function (equation 1). If the output is greater than 1, then it is classified with one type of class, and if the output is less than -1, then it is classified it with another class. We train the SVM model using the hinge loss function, as shown below.
(, , ) = 0,
* ≥ 1 = 1 − * , ℎ
It is noticed from the loss function, that the cost will be 0 if the predicted and actual values are of the same sign. In this case, the loss value is called if the predicted and actual values are of diferent signs. The Hinge loss function is used for the SVM model hyper-parameter tuning.
Multinomial Naïve Bayes model is also used in this task mainly for text classification. This model uses the Bayes’ theorem mentioned as below: (|) = (|). () () where, (|) is the posterior probability of class y given features X. (1) (2) (3) (4) (|) is the likelihood, representing the probability of observing features X given class y. () is the prior probability of class y. () is the probability of observing features X, which acts as a normalization constant.
Multinomial Naive Bayes operates on the assumption that each of the features are conditionally independent of the other given some class.
A system with Intel i5 processor and 32 GB RAM is employed for the task implementation. The whole task has three steps, as mentioned earlier. At first, the seed dataset is divided into two segments training data (90%) and validation data (10%). Training dataset is exploited to train the three ML models - logistic regression, support vector machine, and multinomial naive Bayes. The test dataset contains 1001 instances. Among them, 719 instances are labeled as not useful and 282 instances are useful. All three models are tested on this test dataset and an overall accuracy of 82.92%, 83.92%, and 50.75% are achieved for logistic regression, support vector machine, and multinomial naive Bayes model respectively. The corresponding confusion matrix are plotted in figure 1. We noticed that the naive Bayes algorithm fails to predict the not useful data eficiently which leads to a degradation in the overall accuracy.
Another dataset is generated using large language model which consists of 311 useful samples and 21 not useful samples. This generated data is then augmented with seed data. This new dataset is then again divided into two parts - training and validation dataset. Furthermore, those new training and validation dataset are used to train the same classification models. The newly trained models are tested with the same test data. An overall accuracy of 82.92%, 84.12%, and 50.05% are achieved from the three models respectively. The individual confusion matrix for all three models are also displayed in figure 2. The evaluation results for all three models are illustrated in table 2. It is evident that the models trained with the augmented dataset experience a slight decrease in the accuracy compared to the previously achieved accuracy from seed dataset. This may be attributed to the incorporation of noises in the seed data from large language models. This noises are mainly generated because of the imperfection of large language model such as chatGPT 4 in our case, which leads to a decrease in the overall accuracy. Still we can argue that the augmented dataset is well-balanced for machine learning model training and generates a similar accuracy as for the initial seed dataset.
(a) Logistic regression (b) Support vector machine (c) Multinomial naive Bayes
This paper proposes a framework for source code comment classification, which classify a comment based on its usefulness within the source code. Three machine learning models such as logistic regression, support vector machine, and multinomial naive Bayes are implemented and trained using seed dataset. These classifier classify each comment into two categories useful and not useful. These three models exhibit an accuracy of 82.92%, 83.92%, and 50.75% respectively. Subsequently this seed dataset is augmented with a newly generated dataset that are gathered from online sources. The corresponding class for the new set are generated using chatGPT large language model (LLM). The newly generated augmented dataset is again used to train all the models. It is observed that the new augmented dataset drops down the accuracy for all three models due to inclusion of noise and biasness owing to the LLM generated dataset. approach to program comprehension from code comments, in: Advanced Computing and Systems for Security, Springer, 2020, pp. 29–42. [17] S. Majumdar, A. Bandyopadhyay, S. Chattopadhyay, P. P. Das, P. D. Clough, P. Majumder, Overview of the irse track at fire 2022: Information retrieval in software engineering, in: Forum for Information Retrieval Evaluation, ACM, 2022. [18] S. Majumdar, A. Bandyopadhyay, P. P. Das, P. Clough, S. Chattopadhyay, P. Majumder, Can we predict useful comments in source codes?-analysis of findings from information retrieval in software engineering track@ fire 2022, in: Proceedings of the 14th Annual Meeting of the Forum for Information Retrieval Evaluation, 2022, pp. 15–17. [19] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., Language models are few-shot learners, Advances in neural information processing systems 33 (2020) 1877–1901. [20] S. Majumdar, S. Paul, D. Paul, A. Bandyopadhyay, B. Dave, S. Chattopadhyay, P. P. Das, P. D.
Clough, P. Majumder, Generative ai for software metadata: Overview of the information retrieval in software engineering track at fire 2023, in: Forum for Information Retrieval Evaluation, ACM, 2023. [21] S. Majumdar, A. Varshney, P. P. Das, P. D. Clough, S. Chattopadhyay, An efective lowdimensional software code representation using bert and elmo, in: 2022 IEEE 22nd International Conference on Software Quality, Reliability and Security (QRS), IEEE, 2022, pp. 763–774.