In the realm of software development, collaboration among development teams is vital, and comments play a pivotal role in maintaining and improving software quality. Comments serve diverse purposes, from elucidating complex code logic to aiding in debugging and ofering insights into design decisions. However, distinguishing between valuable and redundant comments can be a formidable challenge. This paper explores the potential of Language Model-based (LLM) approaches, particularly advanced models such as GPT-3, to automate the classification of comments and evaluate their efectiveness. By harnessing the contextual comprehension and generation capabilities of these models, this research postulates significant advancements in comment analysis. Through extensive experiments utilizing both real-world human-labeled comments and synthetic comments generated by ChatGPT, we demonstrate that LLMs can classify comments with remarkable accuracy, surpassing previous methods that rely on surface-level features. Additionally, this study critically examines factors like pre-training data, comment coverage, and model architectures, shedding light on their impact on comment analysis. In summary, this research makes several substantial contributions. It thoroughly explores the application of cutting-edge LLMs for comment classification across various contexts, provides a benchmark dataset of human-annotated comments, and highlights that LLMs can greatly enhance codebase documentation by automatically identifying low-quality comments. These techniques hold the potential for integration into Integrated Development Environments (IDEs) to provide developers with continuous feedback. Finally, this paper opens up new possibilities for leveraging advanced Natural Language Processing (NLP) in software engineering tasks that require deep code comprehension, despite lingering questions about model robustness and the nature of human-AI collaboration. This work underscores the enormous potential of LLMs in revolutionizing programming by mastering language understanding and generation in the context of software development.
In the landscape of modern software development, where large, collaborative teams construct intricate codebases, code comments serve as a lifeline for developers. Well-crafted comments offer insights into the reasoning behind the code, complex implementations, edge cases, and issues that may elude static analysis. This documentation proves invaluable for ongoing maintenance, facilitating the onboarding of new team members, debugging, and preserving institutional knowledge. While numerous studies have emphasized the benefits of comments in software development, sheer quantity alone does not guarantee improved code quality or comprehension. The real challenge lies in diferentiating between helpful comments and those that obfuscate understanding. Low-quality comments that merely restate the code, delve into trivial implementation details, or contain outdated information can introduce confusion and noise. The identification and management of comment quality pose significant challenges, particularly in large-scale projects. This paper delves into the possibility of automating the classification of code comments as either useful or non-useful using Language Model-based (LLM) approaches, such as GPT-3. These models excel in understanding and generating language, making them ideal candidates for assessing comments. Our research includes experiments involving two sources of comments: human-curated comments from real-world code and synthetic comments generated by the LLM ChatGPT. Through rigorous analysis and comparisons, our study demonstrates that LLMs can achieve accuracy rates exceeding 90% in comment classification, outperforming previous methods reliant on surface-level code features. These techniques hold the potential for seamless integration into developer workflows, aiding in the identification of unhelpful comments, enhancing documentation practices, and ultimately improving the maintainability of codebases in the long term.
Before delving into automatic comment classification, we first review the established literature
regarding the role of comments in program comprehension. Efective documentation is widely
acknowledged as crucial in software development and maintenance. Code tells you "what,"
while comments tell you "why." Comments clarify reasoning, capture design decisions, elucidate
complex sections, and prevent the loss of critical knowledge over time. Several seminal studies
have empirically demonstrated the advantages of high-quality comments. Tenny (1988) found
that comments significantly improved code understanding, even more so than identifier names
[
Recent advancements in Natural Language Processing (NLP) have yielded powerful Language Models (LMs) with remarkable capabilities. Models like GPT-3 exhibit proficiency in understanding, generating, and reasoning about natural language. While primarily designed for conversational tasks, LMs also demonstrate strengths in dealing with programming languages. Multiple studies have explored the potential applications of LMs in software engineering, including code search and retrieval, automated documentation generation, code summarization, bug detection, security vulnerability identification, and improved code completion. These applications underscore the potential of LMs to assist developers by leveraging their substantial knowledge of code and mastery of natural language for explaining it. We hypothesize that LMs can significantly enhance the analysis of code comments, given their strengths in both language and code understanding. Surprisingly, prior to our research, no comprehensive study had examined LMs for classifying comment utility. Our research conducts extensive experiments using powerful LMs like GPT-3, applied to both real and synthetic comments on a large scale.
Software metadata [
In the realm of mining code comments and assessing their quality, several authors have
conducted research. Steidl et al. [
In light of the emergence of large language models, such as GPT-3.5 or llama [
Early approaches relied on manually defined rules and heuristics to identify unhelpful comments. For instance, Ratzinger et al. (2007) specified rules such as overly lengthy comments, the presence of code tokens, or excessively short comments [21]. Tan et al. (2007) designed 197 regex patterns to match non-informative phrases [22]. de Souza et al. (2005) also defined rules based on comment length, special characters, and keywords [23]. These rule-based systems required extensive input from domain experts and sufered from limited generalizability across diferent contexts and dificulties in handling semantic variations. In contrast, our LM-based approach overcomes these challenges through automated inductive capabilities and contextual understanding.
More recent eforts have focused on extracting linguistic features to train traditional machine
learning classifiers. Steidl et al. (2013) computed lexical features such as comment length, terms
used, readability, and punctuation to train Support Vector Machines (SVMs) [
Several studies have explored neural networks for comment analysis, albeit with key diferences from our approach. For instance, Hu et al. (2018) employed Long Short-Term Memory (LSTM) networks on sequential comment text for classification [25]. Jiang et al. (2017) combined Recurrent Neural Networks (RNNs) for text with Convolutional Neural Networks (CNNs) for source code [26]. While promising, these models do not benefit from the extensive pre-training of large-scale LMs. Most closely related to our work, Prasetyo et al. (2020) fine-tuned BERT for comment quality assessment [27]. However, they only explored smaller BERT models on limited datasets. Our research conducts more extensive studies using powerful LMs like GPT-3, applied to both real and synthetic comments at scale.
In summary, our work represents the first comprehensive investigation into the application of state-of-the-art LLMs for comment classification. Through rigorous comparative experiments, we demonstrate their advantages over previous shallow learning and neural approaches. In the following sections, we detail our hypothesis, datasets, model architectures, training procedures, and evaluation methodology.
Our hypothesis posits that LLMs can accurately classify code comments as useful or not, leveraging their proficiency in understanding language semantics and programming concepts. We follow a structured approach that involves the curation of labeled datasets, the design of LLM-based classifiers tailored for this task, and extensive experiments to quantify performance while analyzing the factors influencing usefulness prediction.
Usefulness is inherently subjective. In this context, we consider comments that summarize intent, explain rationale, clarify edge cases, and capture critical knowledge as useful. Non-useful comments are those that are redundant, overly vague, or provide minimal value beyond what the code itself conveys.
Real comments provide us with ground-truth evaluation data derived from human-authored code. Synthetic comments, on the other hand, ofer us greater scalability and control over the dataset. For real comments, we sample approximately 10,000 comments from five Java projects and manually label them for usefulness. During the curation process, we balance the classes of useful and non-useful comments to ensure robust training. The selected projects span various domains, including databases, servers, compilers, and frameworks, to enhance diversity. For synthetic data, we utilize public GitHub repositories to extract 100,000 Java method bodies without accompanying comments. Each of these methods is presented to ChatGPT to generate a descriptive comment, which we treat as the ground-truth label. This process yields a diverse set of comments at scale in an automated manner.
Our classification model adheres to a standard LLM architecture. The input comment tokens undergo an initial text embedding layer. In our experiments, we explore both frozen and tunable embeddings. These embeddings are then fed into a multi-layer Transformer encoder model, similar to GPT-2/3, which contextualizes the representations through self-attention mechanisms. Finally, a linear output layer classifies the encoded comment as either useful or not. To prime the Transformer layers with programming language knowledge, we pretrain them on extensive corpora of public code sourced from GitHub. For smaller models, we subsequently fine-tune them end-to-end on our labeled comment datasets. In the case of larger models, we generate embeddings for the comments and train shallow classifiers.
Our model optimization process involves minimizing the cross-entropy loss between the predicted labels and the true labels indicating usefulness. We fine-tune hyperparameters, including Dataset Cassandra Elasticsearch Derby Solr Jetty batch size, learning rate, embeddings, and L2 regularization, through a systematic grid search on validation sets. For smaller models, we employ early stopping if the validation loss stabilizes. The evaluation of model performance is conducted on held-out test comments that were not part of the training dataset.
Furthermore, we conduct ablation studies to analyze various model variations and their impact on performance: • Pre-training data: Models trained on larger codebases generally outperformed those trained on smaller sets, although performance gains saturated beyond a certain dataset size. • Comment length: Short comments tended to pose greater dificulty than longer ones. Performance plateaued when comments exceeded a certain token length, as longer comments provided more contextual information. • Model choice: Transformer architectures consistently outperformed RNN and CNN models. The attention mechanisms within Transformers likely contribute to their ability to assess relationships and semantic meaning. • Embeddings: Contextual embeddings, such as ELMo, outperformed static embeddings like Word2Vec, highlighting the value of dynamic representations.
Our evaluation of LLM-based comment classifiers encompassed diverse settings, involving both real and synthetic datasets. The results indicate that our models significantly outperform previous approaches, underscoring the advantages of contextual language mastery. Below, we provide a summary of key findings:
Our models demonstrated robust usefulness classification across five real comment datasets:
The results in Table 1 showcase an impressive absolute accuracy improvement of approximately 20% over the prior state-of-the-art. Our models leverage the contextual understanding capabilities of LLMs, which were absent in previous feature-based methods. The consistent performance gains across diverse projects highlight the generalizability of our approach.
Model SVM baseline LSTM classifier DistilGPT GPT-2 Codex GPT-3
On the larger-scale synthetic test set, our models achieved even more remarkable performance:
Our analysis of various model variations revealed key factors influencing performance: • Pre-training data: Models trained on larger codebases generally outperformed those on smaller sets. However,performance gains saturated beyond a dataset size of 10 million samples. • Comment length: Short comments presented greater dificulty compared to longer ones.
Performance plateaued when comments exceeded 50 tokens, as longer comments provided more contextual information. • Model choice: Transformer architectures consistently outperformed RNN and CNN models. The attention mechanisms in Transformers likely play a crucial role in assessing relationships and semantic meaning. • Embeddings: Contextual embeddings, such as ELMo, outperformed static embeddings like Word2Vec, highlighting the value of dynamic representations.
• Subtle sarcasm or critique in comments for dysfunctional code. • Overly terse or condensed comments that require a high level of prior knowledge. • Comments that fall into a gray area between high-level explanations and necessary abstractions.
In general, distinguishing subjectivity and evaluating conceptual meaning proved to be the most challenging aspects. Integrating external knowledge could potentially address some of these challenges but remains a complex task for machines.
As an approximate upper bound on performance, three expert developers manually classified 1,000 held-out comments. Their aggregate accuracy reached 96.1%, indicating that LLMs can approach expert-level capabilities on this task. However, the presence of human subjective disagreement on certain borderline cases implies a potential performance ceiling.
This paper has presented a comprehensive study showcasing how advanced LLMs can enable the accurate classification of code comment utility. Through extensive experiments on both realworld and synthetic datasets, we have quantified significant improvements over the previous state-of-the-art. Our results indicate that the contextual mastery provided by LLMs leads to a deeper semantic understanding compared to previous surface-level feature extraction methods that were unable to capture conceptual usefulness.
The techniques proposed in this research have the potential to generalize across programming languages, given the broad knowledge base of LLMs. Our models can be seamlessly integrated into developer workflows to automatically identify unhelpful comments for removal or revision. Beyond the enhancement of documentation quality, this facilitates the concentration of programmer attention on meaningful explanations, thereby supporting long-term comprehension. In a broader sense, our work underscores the profound potential of LLMs in advancing software engineering tasks that require both code understanding and language proficiency.
However, certain limitations persist. The application of LLMs can impose a high computational cost, necessitating optimization. Evaluating subtler aspects of comment quality beyond binary usefulness could provide deeper insights. Additional real-world studies are warranted to assess robustness across various projects and programming languages. Opportunities exist for closer human-AI collaboration, combining automation with nuanced developer feedback. In conclusion, our research demonstrates that code comprehension stands as one of the domains where LLMs are poised to deliver immense practical value in the years to come. [20] S. Majumdar, A. Varshney, P. P. Das, P. D. Clough, S. Chattopadhyay, An efective lowdimensional software code representation using bert and elmo, in: 2022 IEEE 22nd International Conference on Software Quality, Reliability and Security (QRS), IEEE, 2022, pp. 763–774.