Comments are very useful to the flow of code development. With the increasing use of code in commonplace life, commenting the codes becomes a hassle for rookie coders, and often they do not even think commenting as a part of the development process. This in general causes the quality of comments to degrade, and a considerable amount of useless comments are found in such codes. In these experiments, the usefulness of C comments are evaluated using Support Vector Machine (SVM) and Naïve Bayes Classifier. The results of the experiments create a baseline for better results that can be found in the future through more research. Based on these findings, more complex and intricate machine learning models can be created that can improve the accuracy achieved in performing said task.
CEUR ceur-ws.org
Comments play an essential role in code development [
Various deep learning-based automatic commenting models can boost the quantity of
comments [
The author has explored a range of Machine Learning (ML) models in pursuit of solutions to
this problem. This paper aims to answer critical questions as part of the Information Retrieval
in Software Engineering (IRSE) shared task at the Forum for Information Retrieval Evaluation
(FIRE) 2022 [
© 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). • What level of complexity is necessary for a Machine Learning model to reliably distinguish useful comments from useless ones? • How do well-known general-purpose models like SVM and Naïve Bayes Classifier perform in this context, even if they are not built for this particular scenario?
This paper aims to demonstrate that models such as SVM or Naïve Bayes Classifier can serve as efective starting points for tackling this problem. More complex models can be built upon these foundations, while also considering the risk of overfitting.
In addition to the aforementioned objectives, this paper also addresses the crucial question of whether the dataset for the learning task can be augmented with data generated by large language models, such as GPT-3. This consideration plays a pivotal role in exploring the potential of leveraging artificial intelligence for improving comment quality assessment. By evaluating the impact of augmenting the dataset with AI-generated data, the study aims to shed light on the benefits and challenges of integrating advanced language models into the machine learning pipeline. This inquiry represents a significant aspect of the research, emphasizing the intersection of human-written and AI-generated content in the context of comment quality evaluation.
Software metadata [
In the realm of mining code comments and assessing their quality, several authors have conducted research. Steidl et al. [16] employ techniques such as Levenshtein distance and comment length to gauge the similarity of words in code-comment pairs, efectively filtering out trivial and non-informative comments. Rahman et al. [17] focus on distinguishing useful from non-useful code review comments within review portals, drawing insights from attributes identified in a survey conducted with Microsoft developers [ 18]. Majumdar et al. [19, 20, 21, 22] have introduced a framework for evaluating comments based on concepts crucial for code comprehension. Their approach involves the development of textual and code correlation features, utilizing a knowledge graph to semantically interpret the information within comments. These approaches employ both semantic and structural features to address the prediction problem of distinguishing useful from non-useful comments, ultimately contributing to the process of decluttering codebases
In light of the emergence of large language models, such as GPT-3.5 or llama [23], it becomes
crucial to assess the quality of code comments and compare them to human interpretation. The
IRSE track at FIRE 2023 [
In this section, a description of the task at hand and the dataset provided are given. The task at IRSE, FIRE 2023 was as follows: A binary code comment quality classification model needs to be augmented with generated code and comment pairs that can improve the accuracy of the model. The corresponding dataset was split into two: • The training dataset with 8048 entries, and • The testing dataset with 1000 entries.
The training dataset was shufled, and split into 70% for training the models, and 30% for cross-validation. The data was labelled as follows: • Useful: Comments that are useful for code comprehension • Not Useful: Comments that are not useful for code comprehension
The dataset augmentation process involved the integration of data generated by GPT-3, a powerful language model, to enrich the existing dataset. By leveraging GPT-3’s natural language generation capabilities, additional comment data was created to expand the diversity and scale of the dataset. This augmentation strategy aimed to introduce a broader spectrum of comments, encompassing a wide range of writing styles, structures, and content. The incorporation of GPT-3 generated data was carried out to assess its potential in enhancing the training of machine learning models for comment quality evaluation. This approach allowed for the exploration of how AI-generated content could complement human-written data, contributing to a more comprehensive and robust dataset for improved model performance.
All the links, punctuations, numbers and stop words have been removed. Then all words which have a POS tag other than Noun, Verb, Adverb and Adjective are removed. Lemmatization is used for grouping together the diferent forms of a word into a single word. NLTK wordnet [25] is used for lemmatization. Both training and testing datasets use same preprocessing steps.
TfidfVectorizer [ 26] is used for converting the text into numerical features. Tokenizer by Keras [27] library is used, along with TfidfVectorizer that was used from SciKit-Learn library.
Two runs have been submitted for the task: one using Support Vector Machine (SVM) model, and another with Naïve Bayes classifer model. We have used the SciKit-Learn library for both of the models, with the parameters for the SVM model as follows: • C: (regularization parameter) = 1 • kernel: (kernel type) = ’linear’
With these parameters set for the SVM model, the validation set gives a 77.26708074534162% accuracy score, along with an F1 score of 0.786464410735123.
Also, with the Naïve Bayes Classifier, the validation set gives a 60.993788819875775% accuracy score, along with an F1 score of 0.699233716475096.
With these parameters set for the SVM model, the validation set gives a 77.64842840512223% accuracy score, along with an F1 score of 0.7830939828521743.
Also, with the Naïve Bayes Classifier, the validation set gives a 64.0279394644936% accuracy score, along with an F1 score of 0.695431994330003.
The tasks were accomplished employing basic machine learning models like SVM and the Naïve Bayes Classifier. The outcomes obtained from the SVM classifier indicate that there is room for enhancement, enabling the development of more intricate models that align better with the problem statement and yield superior results. Notably, Srijoni Majumdar et al. [28] have already achieved superior results using neural networks, and the author anticipates continuous improvement in these results over time.
Thanks to the creators of IRSE FIRE for giving this wonderful opportunity to work on such a project, and their constant technical support throughout the timespan. from multi-threaded applications using pin, in: 2016 IEEE International Conference on Software Quality, Reliability and Security (QRS), IEEE, 2016, pp. 25–32. [11] S. Majumdar, N. Chatterjee, P. P. Das, A. Chakrabarti, A mathematical framework for design discovery from multi-threaded applications using neural sequence solvers, Innovations in Systems and Software Engineering 17 (2021) 289–307. [12] S. Majumdar, N. Chatterjee, P. Pratim Das, A. Chakrabarti, Dcube_ nn d cube nn: Tool for dynamic design discovery from multi-threaded applications using neural sequence models, Advanced Computing and Systems for Security: Volume 14 (2021) 75–92. [13] J. Siegmund, N. Peitek, C. Parnin, S. Apel, J. Hofmeister, C. Kästner, A. Begel, A. Bethmann, A. Brechmann, Measuring neural eficiency of program comprehension, in: Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, 2017, pp. 140–150. [14] Y. Wang, H. Le, A. D. Gotmare, N. D. Bui, J. Li, S. C. Hoi, Codet5+: Open code large language models for code understanding and generation, arXiv preprint arXiv:2305.07922 (2023). [15] J. L. Freitas, D. da Cruz, P. R. Henriques, A comment analysis approach for program comprehension, Annual Software Engineering Workshop (SEW), IEEE, 2012, pp. 11–20. [16] D. Steidl, B. Hummel, E. Juergens, Quality analysis of source code comments, International
Conference on Program Comprehension (ICPC), IEEE, 2013, pp. 83–92. [17] M. M. Rahman, C. K. Roy, R. G. Kula, Predicting usefulness of code review comments using textual features and developer experience, International Conference on Mining Software Repositories (MSR), IEEE, 2017, pp. 215–226. [18] A. Bosu, M. Greiler, C. Bird, Characteristics of useful code reviews: An empirical study at microsoft, Working Conference on Mining Software Repositories, IEEE, 2015, pp. 146–156. [19] S. Majumdar, A. Bansal, P. P. Das, P. D. Clough, K. Datta, S. K. Ghosh, Automated evaluation of comments to aid software maintenance, Journal of Software: Evolution and Process 34 (2022) e2463. [20] S. Majumdar, S. Papdeja, P. P. Das, S. K. Ghosh, Comment-mine—a semantic search approach to program comprehension from code comments, in: Advanced Computing and Systems for Security, Springer, 2020, pp. 29–42. [21] S. Majumdar, A. Bandyopadhyay, S. Chattopadhyay, P. P. Das, P. D. Clough, P. Majumder, Overview of the irse track at fire 2022: Information retrieval in software engineering, in: Forum for Information Retrieval Evaluation, ACM, 2022. [22] S. Majumdar, A. Bandyopadhyay, P. P. Das, P. Clough, S. Chattopadhyay, P. Majumder, Can we predict useful comments in source codes?-analysis of findings from information retrieval in software engineering track@ fire 2022, in: Proceedings of the 14th Annual Meeting of the Forum for Information Retrieval Evaluation, 2022, pp. 15–17. [23] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., Language models are few-shot learners, Advances in neural information processing systems 33 (2020) 1877–1901. [24] S. Majumdar, A. Varshney, P. P. Das, P. D. Clough, S. Chattopadhyay, An efective lowdimensional software code representation using bert and elmo, in: 2022 IEEE 22nd International Conference on Software Quality, Reliability and Security (QRS), IEEE, 2022, pp. 763–774. [25] E. Loper, S. Bird, Nltk: The natural language toolkit, 2002. URL: https://arxiv.org/abs/cs/ 0205028. doi:10.48550/ARXIV.CS/0205028. [26] V. Kumar, B. Subba, A tfidfvectorizer and svm based sentiment analysis framework for text data corpus, in: 2020 National Conference on Communications (NCC), 2020, pp. 1–6. doi:10.1109/NCC48643.2020.9056085. [27] N. Ketkar, Introduction to Keras, 2017, pp. 95–109. doi:10.1007/978-1-4842-2766-4_7. [28] S. Majumdar, A. Bansal, P. P. Das, P. D. Clough, K. Datta, S. K. Ghosh, Automated evaluation of comments to aid software maintenance, Journal of Software: Evolution and Process 34 (2022) e2463. URL: https://onlinelibrary. wiley.com/doi/abs/10.1002/smr.2463. doi:https://doi.org/10.1002/smr.2463. arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1002/smr.2463.