In software development, code comments play a crucial role in enhancing code comprehension and collaboration. This research paper addresses the challenge of objectively classifying code comments as "Useful" or "Not Useful." We propose a novel solution that harnesses contextualized embeddings, particularly BERT, to automate this classification process. We address this task by incorporating generated code and comment pairs. The initial dataset comprised 9048 pairs of code and comments written in C, labeled as either Useful or Not Useful. To augment this dataset, we sourced an additional 739 lines of code-comment pairs and generated labels using a Large Language Model Architecture, specifically BERT. The primary objective was to build classification models that can efectively diferentiate between useful and not useful code comments. Various machine learning algorithms were employed, including Logistic Regression, Decision Tree, K-Nearest Neighbors (KNN), Support Vector Machine (SVM), Gradient Boosting, Random Forest, and a Neural Network. Each algorithm was evaluated using precision, recall, and F1-score metrics, both with the original seed dataset and the augmented dataset. This study showcases the potential of generative AI for enhancing binary code comment quality classification models, providing valuable insights for software developers and researchers in the field of natural language processing and software engineering.
Within the realm of software development, code comments assume a fundamental role as crucial
documentation artifacts [
Code comment classification, a subfield entrenched within natural language processing, has emerged as
a transformative methodology for impartially categorizing code comments as either "Useful" or "Not
Useful" [
The challenges inherent in comment classification are multifaceted [
The advent of LLMs signifies a pivotal transformation in the methodology of code comment analysis
and utilization [
This research embarks on an exhaustive exploration, elucidating the intricate relationship between
comment classification and LLMs [
The research encompasses an array of classification models, spanning traditional algorithms and neural
networks, subjected to comprehensive evaluation through an assortment of performance metrics [
The outcomes of this research illuminate the efectiveness of diverse models in code comment
classification, accentuating the transformative potential of LLMs in this domain [
In recent years, research in the field of software engineering has seen a growing interest in the
classification and evaluation of code comments to enhance code comprehensibility and maintenance.
Two papers, "Comment-Mine - Building a Knowledge Graph from Comments" [
Comprehensibility
[
The classification model in this research shares a common goal with [
In the realm of Natural Language Processing (NLP) and code-related tasks, the choice of word
embeddings plays a pivotal role in influencing the performance of machine learning models. "Contextualized
Word Representations for Code Search and Classification" [
In [
In the current research, we employ contextualized embeddings, inspired by the success of
contextualized word representations in NLP tasks, in the context of code comment classification. While [
The initial dataset comprised 9048 pairs of code and comments written in C, labeled as either Useful or Not Useful. The experiment begins with the acquisition of a diverse dataset of code comments and their associated code snippets. A corpus of code repositories is sampled from the GitHub platform, focusing on projects implemented in the C programming language. These repositories serve as the primary source of data for both the seed dataset and LLM-generated data. GitHub API calls are made to access code files and extract comments.
We sourced an additional 739 lines of code-comment pairs and generated labels using a Large Language Model Architecture, namely BERT.
Data preprocessing involves several steps to ensure the dataset’s quality and readiness for classification. The steps followed here include:
Data preprocessing played a crucial role in preparing our dataset for machine learning analysis. We started by addressing missing values in both the code-comment pairs and labels to ensure data
A traditional machine learning model used as a baseline.
A non-linear model capable of handling complex feature interactions.
A proximity-based model for instance-based learning.
A model known for its efectiveness in high-dimensional spaces.
An ensemble learning technique that combines multiple weak learners.
Another ensemble method leveraging decision trees.
Utilizing the BERT model, a state-of-the-art Large Language Model. completeness. Next, we focused on the essential content of code comments by removing punctuation, special characters, and code-specific syntax. Additionally, we converted all text to lowercase to ensure uniformity and reduce dimensionality.
To further enhance data quality, we performed outlier removal. This involved calculating Z-scores for the lengths of ’Comments’ and ’Surrounding Code Context’ strings. Z-scores were used to identify outliers, with a predefined threshold for what constitutes an outlier (e.g., z-score > 3 or < -3). Rows with z-scores beyond this threshold were filtered out.
Furthermore, we applied a function to the ’Surrounding Code Context’ column to remove preceding numbers, thereby enhancing the consistency and relevance of this text data.
Once the data preprocessing steps were completed, the next crucial step was vectorization. Vectorization is essential for converting text data into a numerical form that machine learning algorithms can work with. We employed two widely used techniques for text vectorization:
Bag of Words (BoW): We represented the text data as a sparse matrix, with each row corresponding to a code-comment pair and each column representing a unique word in the dataset’s vocabulary. The matrix values indicated the frequency of each word’s occurrence.
Term Frequency-Inverse Document Frequency (TF-IDF): This advanced vectorization technique considered the importance of words in each code comment relative to their significance in the entire dataset. It assigned higher weights to words that were frequent in a code comment but rare in the overall dataset, capturing their importance for classification.
These preprocessing and vectorization steps were essential in preparing the dataset for subsequent machine learning analysis.
These preprocessing and vectorization steps were crucial to ensure that our dataset was clean, structured, and ready for training classification models that could efectively diferentiate between useful and not useful code comments.
The experiment encompasses a range of classification models to evaluate their performance in code comment classification. These models include:
Each classification model undergoes a training phase using the training dataset. Hyperparameter tuning is performed to optimize model performance using random search strategy.
To ensure robustness and reliability, the experiment implements with k set to 5. For the neural network model, hyperparameters were fine-tuned, including the number of hidden units, activation functions (ReLU), a learning rate of 0.001, and a fixed training duration of 10 epochs.
The efectiveness of each model is evaluated using the standard classification metrics, namely, precision, recall, and F1 Score.
These metrics are computed for both the seed dataset and the seed dataset augmented with LLMgenerated data. Comparative analysis focuses on changes in these metrics, particularly improvements resulting from LLM data augmentation.
To assess the impact of LLM-generated data, the seed dataset is augmented with comments generated by LLMs, namely, BERT. LLM-generated comments are selected to be relevant to code snippets in the dataset. The experiment measures changes in model performance metrics when LLM-generated data is introduced, highlighting the potential benefits of this augmentation strategy.
The following table summarizes the results obtained from diferent classification models for code comment classification. The metrics evaluated include precision, recall, and F1 score for both the seed dataset and the seed dataset combined with Large Language Model (LLM)-generated data.
Serial #
Model
Precision with Recall with F1 Score with
Seed Data Seed Data Seed Data
The table illustrates the performance of diferent classification algorithms on code comment classification tasks. Notably, the results demonstrate variations in precision, recall, and F1 score across diferent algorithms. Further, the impact of combining seed data with LLM-generated data is evident in the improved performance metrics, particularly in terms of recall and F1 score.
Logistic Regression performs reasonably well with a relatively high recall, indicating that it correctly identifies a significant portion of "Useful" comments. However, precision is slightly lower, suggesting that it may occasionally misclassify comments as "Useful" when they are not. The introduction of LLM-generated data leads to a minor improvement in F1 score, indicating that this augmentation strategy contributes positively to the overall performance.
Decision Trees perform well in terms of precision, indicating that when they classify a comment as "Useful," they are often correct. However, the recall is slightly lower, suggesting that they may miss some "Useful" comments. The introduction of LLM-generated data leads to a minor improvement in F1 score, indicating that this augmentation strategy contributes positively to the overall performance, similar to Logistic Regression. 4.1.3. KNN 4.1.4. SVM KNN shows a balanced performance with relatively high precision and recall values for seed data. However, the introduction of LLM-generated data results in a significant drop in recall and, consequently, F1 score. This suggests that KNN may not handle the added LLM-generated data as efectively as some other algorithms, leading to decreased performance in identifying "Useful" comments. SVM performs well in terms of both precision and recall, indicating that it correctly identifies a significant portion of "Useful" comments while maintaining precision. The introduction of LLM-generated data results in a minor improvement in F1 score, indicating that SVM can efectively utilize this additional data for classification without compromising precision.
Random Forest demonstrates strong performance in precision, recall, and F1 score for both seed data and seed data augmented with LLM-generated data. This suggests that Random Forest efectively captures complex relationships within the data and benefits from the additional information provided by LLM-generated data. The F1 score for Random Forest is the highest among the models, indicating a balanced trade-of between precision and recall. Therefore, Random Forest is the preferred choice for code comment classification in this study.
The Neural Network model, with a binary cross-entropy loss function, ReLU activation, and 10 epochs, shows competitive performance. However, it falls slightly short of Random Forest in terms of F1 score. While Neural Networks have the potential to capture complex patterns in the data, the limited amount of data and training epochs may have afected its performance. Further experimentation with hyperparameters and more extensive training could potentially improve its results.
In summary, diferent algorithms exhibit varying strengths and weaknesses in classifying code comment pairs as "Useful" or "Not Useful." Logistic Regression and Decision Trees show reasonable performance, Models g n i t s o o B t n e i d a r G t s e r o F m o d n a R k r o w t e N l a r u e N with minor improvements when augmented with LLM-generated data. KNN exhibits a drop in performance with LLM-generated data, while SVM maintains a strong performance. The choice of algorithm for comment classification should consider the specific trade-ofs between precision and recall, as well as the efectiveness of LLM-generated data integration in improving F1 score.
The research presented in this paper addresses the challenge of objectively classifying code comments as "Useful" or "Not Useful" in the context of software development. It leverages contextualized embeddings, particularly BERT, to automate this classification process and provides precise and context-aware evaluations. The results of the experiment demonstrate the efectiveness of diferent classification models and highlight the potential benefits of incorporating LLM-generated data in improving classification performance.
This research contributes to the fusion of natural language processing and software engineering, promising improved code comprehensibility and maintainability. It opens avenues for further exploration of LLMs in code-related tasks and the development of more advanced models for code comment classification.
In the ever-evolving landscape of code comment assessment, this research elucidates a promising future wherein the symbiosis of comment classification and LLMs stands at the vanguard of innovation.
The authors would like to acknowledge the support and resources provided by the Department of Computer Science and Engineering at Sri Sivasubramaniya Nadar College of Engineering, Chennai, Tamil Nadu, India.