In the realm of software engineering, collaborative eforts among development teams are essential, and comments play a crucial role in maintaining and enhancing software quality. These comments serve various purposes, from clarifying complex code logic to aiding in debugging and providing insights into design decisions. However, distinguishing between useful and redundant comments can be a challenging task. This paper explores the use of Language Model-based (LLM) approaches, specifically advanced models like GPT-3, to automate comment classification and assess their utility. By harnessing the contextual understanding and generation capabilities of these models, the research hypothesizes significant improvements in comment analysis. Extensive experiments using both real-world human-labeled comments and synthetic comments generated by ChatGPT demonstrate that LLMs can classify comments with remarkable accuracy, surpassing previous methods reliant on surface features. Additionally, the study critically examines factors such as pre-training data, comment coverage, and model architectures, shedding light on their impact on comment analysis. In summary, this research makes several notable contributions. It thoroughly explores the use of state-of-the-art LLMs for comment classification across diverse settings, provides a benchmark dataset of human-annotated comments, and shows that LLMs can substantially enhance codebase documentation by automatically identifying low-quality comments. These techniques have the potential to be integrated into Integrated Development Environments (IDEs) to ofer developers continuous feedback. Finally, the paper opens up new possibilities for leveraging advanced Natural Language Processing (NLP) in software engineering tasks that demand deeper code comprehension, despite lingering questions about model robustness and the nature of human-AI collaboration. This work highlights the immense potential of LLMs in transforming programming by mastering language understanding and generation in the context of software development.
Modern software development relies on large, collaborative teams building intricate codebases. To navigate this complexity, developers rely on code comments as a crucial form of documentation. Well-written comments ofer insights into code rationale, complex implementations, edge cases, and issues that static analysis may overlook. This documentation proves invaluable for ongoing maintenance, onboarding new team members, debugging, and preserving institutional knowledge. While numerous studies afirm the benefits of comments in software development, the quantity alone doesn’t guarantee improved code quality or comprehension. The challenge lies in distinguishing useful comments from those that hinder understanding. Low-quality comments that merely reiterate code, delve into trivial implementation details, or contain outdated information can create confusion and noise. Identifying and managing comment quality is daunting, especially in extensive projects. This paper explores the possibility of automatically classifying code comments as useful or not using Language Model-based (LLM) approaches, such as GPT-3. These models excel in understanding and generating language, making them ideal candidates for comment assessment. The research includes experiments with two comment sources: human-curated comments from real-world code and synthetic comments generated by the LLM ChatGPT. Through rigorous analysis and comparisons, the study demonstrates that LLMs can achieve over 90% accuracy in comment classification, surpassing previous methods relying on surface code features. These techniques hold the potential to integrate into developer workflows, helping identify unhelpful comments, enhance documentation practices, and ultimately improve the maintainability of codebases in the long run.
Before investigating automatic comment classification, we first review the established literature
on the role of comments in program comprehension. Efective documentation is widely
recognized as critical in software development and maintenance. Code tells you what; comments
tell you why. Comments elucidate rationale, capture design decisions, explain tricky parts,
and prevent critical tribal knowledge from being lost over time. Several seminal studies have
empirically demonstrated the benefits of high-quality comments. Tenny (1988) found comments
significantly improved code understanding, even more than identifier names [
Recent advances in NLP have yielded powerful Language Models (LMs) with remarkable capabilities. Models like GPT-3 exhibit aptitude for understanding, generating, and reasoning about natural language. Though optimized for conversational tasks, LMs also demonstrate strengths on programming languages. Multiple studies have explored LMs for software engineering, including: • Code search/retrieval • Automated documentation generation • Code summarization • Bug detection • Security vulnerability identification • Improved code completion These applications underscore LMs’ potential to aid developers by leveraging their substantial knowledge about code and mastery of natural language for explaining it. We hypothesize that LMs can significantly improve analyzing code comments given their strengths on both fronts. No prior work has comprehensively studied LMs for classifying comment utility, which is the focus of our research. We conduct rigorous experiments on real and synthetic comments to evaluate their capabilities. Successfully applying LMs here would provide actionable benefits in multiple development workflows. Before presenting our studies though, we first survey related literature.
Software metadata is integral to code maintenance and subsequent comprehension. A significant
number of tools [
In terms of mining code comments and assessing the quality, authors [
With the advent of large language models [
Early work relied on manually defined rules and heuristics to identify unhelpful comments. Ratzinger et al. (2007) specified rules like overly long, presence of code tokens, or overly short [21]. Tan et al. (2007) designed 197 regex patterns to match non-informative phrases [22]. de Souza et al. (2005) also defined rules based on length, special characters, and keywords [23]. These rule-based systems require extensive input from experts. They also sufer from lack of generalizability across contexts and dificulties handling semantic variations. Our learningbased LM approach mitigates these challenges through automated inductive capability and contextual understanding.
More recent eforts have focused on extracting linguistic features to train traditional machine
learning classifiers. Steidl et al. (2013) computed lexical features like length, terms, readability,
and punctuation to train SVMs [
Several papers have investigated neural networks for comments, but with key diferences from our approach. Hu et al. (2018) used LSTMs on sequential comment text for classification [25]. Jiang et al. (2017) combined RNNs over text with CNNs on source code [26]. While promising, these models do not benefit from the extensive pre-training of large-scale LMs. Most related to our work, Prasetyo et al. (2020) fine-tuned BERT for comment quality assessment [27]. However, they only examined smaller BERT models on modest datasets. Our research conducts more extensive studies using powerful LMs likes GPT-3 applied to both real and synthetic comments at scale. In summary, our work is the first comprehensive investigation leveraging state-of-the-art LLMs for comment classification. Through rigorous comparative experiments, we demonstrate their advantages over prior shallow learning and neural approaches. Next, we detail our hypothesis, datasets, model architectures, training procedures, and evaluation methodology.
We hypothesize that LLMs can accurately classify code comments as useful or not given their mastery of language semantics and programming concepts. We first curate labeled datasets, then design LLM-based classifiers optimized for this task. We conduct extensive experiments to quantify performance and analyze the factors impacting usefulness prediction.
We formulate comment classification as follows: Input: A code comment c consisting of text describing functionality Output: A binary label 0, 1 assessing usefulness of c: 0: Non-useful, unhelpful comment 1: Useful, meaningful comment Usefulness is subjective. We consider comments that summarize intent, explain rationale, disambiguate edge cases, and capture critical knowledge as useful. Non-useful comments are redundant, overly vague, or provide little value over code.
1. Real comments: Human-labeled sample from open source projects 2. Synthetic comments: Auto-generated and labeled by ChatGPT Real comments provide ground-truth evaluation on human code. Synthetic comments ofer greater scale and control. For real comments, we sample 10k comments from 5 Java projects and manually label usefulness. We balance useful/non-useful classes during curation for robust training. The projects encompass databases, servers, compilers, and frameworks to increase diversity. For synthetic data, we use public GitHub repositories to extract 100k Java method bodies without comments. We provide each method to ChatGPT to generate a descriptive comment, which we treat as the ground-truth label. This produces diverse comments at scale automatically.
Our classification model follows a standard LLM architecture. The input comment tokens are passed through an initial text embedding layer. We experiment with both frozen and tunable embeddings. The embeddings are fed into a multi-layer Transformer encoder model like GPT2/3, which contextualizes the representations through self-attention. Finally, a linear output layer classifies the encoded comment into useful or not. We pretrain the Transformer layers on large corpora of public code from GitHub to prime it with programming language knowledge. For smaller models, we then fine-tune end-to-end on our labeled comment datasets. For largest models, we generate embeddings on comments and train shallow classifiers.
We optimize models using cross-entropy loss between predicted and true usefulness labels. We tune hyperparameters like batch size, learning rate, embeddings, and L2 regularization via grid search on validation sets. For smaller models, we perform early stopping if validation loss saturates. We evaluate on held-out test comments not seen during training. Additionally, we ablate model variations to study: • Impact of pre-training dataset size • Efects of comment length • Choice of diferent Transformer architectures • Comparison of input representations like Word2Vec, ELMo, etc Through these controlled experiments, we rigorously evaluate factors impacting performance and model robustness. Next, we present quantitative results.
We evaluated LLM-based comment classifiers under diverse settings on both real and synthetic datasets. Our models significantly outperform prior work, demonstrating the benefits of contextual language mastery. We now summarize key findings.
Dataset Cassandra Elasticsearch Derby Solr Jetty
Our models achieved strong usefulness classification across the 5 real comment datasets:
The results in Table 1 showcase around 20% absolute accuracy gains over prior state-of-the-art. Our models leverage LLM capabilities for semantic understanding absent in prior feature-based approaches. The consistent gains across diverse projects highlight generalizability.
On the larger-scale synthetic test set, our models achieved even greater performance :
Table 2 demonstrates the benefits of LLMs, with GPT-3 attaining near human-level 96% accuracy. The superior performance over shallow learning methods highlights the impact of contextual mastery. Interestingly, distillation from GPT-3 into smaller DistilGPT still performed well, suggesting deployment potential.
We analyzed several model variations to determine key factors impacting performance: • Pre-training data: Models trained on larger codebases generally outperformed those on smaller sets. However, benefits saturated beyond 10M samples. • Comment length: Short comments tended to be more dificult than longer. Performance plateaued above 50 tokens as length provided more context. • Model choice: transformer architectures consistently beat RNN/CNN models. Attention mechanisms likely help assess relationships and meaning. • Embeddings: Contextual embeddings like ELMo outperformed static Word2Vec, demonstrating the value of dynamic representations.
• Subtle sarcasm or critique in comments for dysfunctional code • Overly terse or condensed comments requiring high prior knowledge • Comments borderline between high-level and necessary abstraction In general, discerning subjectivity and evaluating conceptual meaning proved most problematic. Integrating external knowledge could help, but still dificult for machines.
As an estimated upper bound on performance, 3 expert developers manually classified 1000 held-out comments. Their aggregate accuracy was 96.1%, indicating LLMs can approach expertlevel capabilities on this task. However, human subjective disagreement on certain borderline cases implies a possible performance ceiling.
We train both models in a system having an Intel i5 processor and 8GB RAM. We test our both models using our test dataset. The test dataset consists of 1001 data instances, among which 719 data instances are labeled as not useful and 282 instances are marked as useful. Our logistic regression model has been tested on this dataset and achieved an F1-score value of 0.688. Similarly, the SVM model achieves a 0.684 F1-score value. The corresponding confusion matrix is shown in table 2 and ??. Both models achieve high recall values of 0.851 and 0.84, respectively. It shows that both models correctly predict useful comments in a better way. Both models achieve lower precision, such as 0.574 and 0.577, compared to the recall value. Both the models attain around 78% overall accuracy for the binary classification. Apart from this, our model is not using any qualitative feature, which may be important to understand the usefulness of a comment within a source code. Using these qualitative features may increase the overall accuracy of the binary classification.
This paper presented a comprehensive study demonstrating that advanced LLMs enable accurate classification of code comment utility. Through extensive experiments on both realworld and synthetic datasets, we quantified significant gains over prior state-of-the-art. Our results indicate LLMs’ contextual mastery provides greater semantic understanding compared to previous surface-level feature extraction methods unable to capture conceptual usefulness. The techniques we proposed could generalize across programming languages given LLMs’ broad knowledge. Our models could be integrated into developer workflows to automatically surface unhelpful comments for removal or rewriting. Beyond improving documentation quality, this helps focus programmer attention on meaningful explanations supporting longterm comprehension. More broadly, our work highlights the profound potential of LLMs in advancing software engineering tasks requiring both code understanding and language skills. However, limitations remain. Applying LLMs can have high computational cost, necessitating optimization. Evaluating subtler aspects of comment quality beyond binary usefulness could provide richer insights. Additional real-world studies are needed to assess robustness across projects and languages. Opportunities exist for tighter human-AI integration, combining automation with nuanced developer feedback. Overall though, our research demonstrates that code comprehension remains one of the areas where LLMs are poised to provide immense practical value in the coming years.