Irrelevant and Useless Comments is a prevalent problem that Programmers/Coders generally struggle with everyday.Though comments increases the size of the code file, but it is very useful for a new programmer to understand which work has already been done in the past. The programmer can easily use a function without understanding the logic with the help of comments. However there are some unnecessary or out-of-context comments that not only decreases the readability of the codebase but also, sometimes, confuses the programmer. This presents the need of automatic detection of such comments on a codebase to ease the programmer with understanding code context, adding new features or debugging the entire code for a bug. In this paper, we have presented the machine learning models that can detect Useless Comments as per the Surrounding code context. Specifically, we described the model submitted for the shared task on Comment Classification in C Language at FIRE 2022.The problem concentrates on binary classification of comments into two classes, namely: Useful and Useless Comments. Overall, our performance is good but it needs some improvement that can be done by some more data preprocessing techniques.Our scores are encouraging enough to work for better results in future.
This task can be considered as a problem requiring a suitable implementation of Text Classification Algorithms as, herein, comments/text are being put into binary classification. Several Works have been proposed that can be effectively used to classify text into two classes based on data set. Mukesh Zaveri[1] proposed anautomatic text classification model that was implemented over the content of Blog posts to classify them as structured or unstructured. Moreover, Kamel Alreshedy[2] proposed a research paper where he showed his team work on machine learning algorithms to classify Code Snippets into programming Languages they belong.
Also, Kriti Kumari[3] participated in HASOC-2019 task 1 i.e. Identification of Abusive Content in a text where they used Glove embedding and fastText embedding techniques to classify comments whether they are abusive or not. The problems that the references discussed above, were not the same as ours. However, these problems shared many similarities and hence were very useful to gain some intuition about the approach.
In this section, we have described the Comment Classification shared task and the dataset provided to the participants.
FIRE 2022 shared a task that basically aims to classify comments written in C Languageinto two classes, namely: Useful (U) and Not Useful (N). (shown in Table 1): 1. (U) Useful - This Comment is very useful as gives some vital information needed to be taken care of while running the code context. 2. (N) NotUseful - This Comment is not useful. It is unnecessarily provided or sometimes, even makes no sense in the codebase.
Alongwith the Comments, their respective Surrounding Code Context was also provided.On the basis of which the comments were required to be classified. We have used the dataset available at IRSE 2022. The dataset consists of 8,047 comments for training provided separately as a training dataset and 1,001 comments for testingprovided as a testing dataset with a balanced distribution of each class. There was no need of splitting the dataset for testing and training purposes.
We have removed all the punctuations, numbers and stop words from the Surrounding Code context. We also removed the starting two characters (“/*”) and ending two characters (“*/”) from the Comments as they are just basic syntax of writing comments in C Language. We then converted all alphabetic characters present into lowercase. We have also used lemmatization for grouping together the different forms of a word into a single word. NLTK wordnet is used for lemmatization. Both Train and Test data uses same preprocessing.
For Logistic regression,Multinomial Naive Bayes, and Support Vector Machine algorithms we have used TFIDF Vectorizer [4] from the Sci-kit learn library. Tf-IdfVectorizeris used for converting the text into numerical features. Pipeline 1 is used for doing Tf-IDFVectorizer and classification in a pipelined manner.
We have submitted runs based on three different algorithms, namely- Logistic Regression [5], Support VectorMachine [6],and Multinomial Naive Bayes [7].
We have used the Sci-kit-learn library for logisticregression-based models. We scored a maximum F1 score of 0.83 using SVM and Logistic Regression (both) for the task.
For SVM we have used a Linear kernel with a C=1.0 and gamma= ‘auto’ and degree=3. Proper value of c and gamma need to chosen for optimizing the performance of SVM Classifier.
The results of the task are represented in terms of Macro-F1,Macro Precision, Macro Recall, and Accuracy (shown in Table below). The best score is Macro-F1, we got a maximum Macro Precision of 0.79 from the implementation of the Logistic Regression Model.
Logistic Regression
Linear Support Vector Machine(SVM)
Multinomial Naive Bayes Classification
Macro F1 0.8 0.79 0.63 6. Conclusion and Future Work 0.8 0.8 0.7 0.79 0.79 0.67
83% 83% 63% We have completed the given task using various text classification algorithms and evaluated the performance of different algorithms for Comment Classification.
The task was very interesting and unique. We got to learn a lot from this task and we also got to test our knowledge of Machine Learning Algorithms.
We look forward to experimenting with different advancedalgorithms or neural network models. Also, fine-tuning the parameters of the algorithm can help in the improvement of the overall performance. And the results of more than one classification algorithm can be combined to generate an overall better score.We shall be exploring these tasks in the coming days. [1] Mukesh Zaveri, Mita K Dalal, Automatic Text Classification: A Technical Review: Sardar Vallabhai National Institute of Technology, Surat,India [2] Kamel Alreshedy, Dhanush Dharmaretnam, Daniel M. German, Venkatesh Srinivasan, T. Aaron Gulliver : SCC : Automatic Classification of Code Snippets: 21 September 2018 [3] Kriti Kumari, Jyotiprakash Singh : Deep Learning Approach for Classification of Abusive Text: National Institute of Technology, Patna: HASOC 2019 [4] V. Kumar, B. Subba, A TF-IDF vectorizer and SVM based sentiment analysis framework for text data corpus, in 2020 National Conference on Communications (NCC), 2020, pp. 1–6. DOI: 10.1109/NCC48643.2020.9056085. [5] Logistic regression (2010) 631–631. URL: https://doi.org/10.1007/978-0-387-30164-8_493DOI: 10.1007/978- 0- 387- 30164- 8_493 . [8]@inproceedings{majumdar2022overview,title={{Overview of the IRSE track at FIRE 2022: Information Retrieval in Software Engineering}},author={Majumdar,Srijoni and Bandyopadhyay, Ayan and Das, Partha Pratim and D Clough, Paul and Chattopadhyay, Samiran and Majumder,Prsenjit},booktitle={ForumforInformationRetrievalEvaluation},publisher={ACM},year={2022 },month={December}} [9] Majumdar, S., Papdeja, S., Das, P.P., Ghosh, S.K. (2020). COMMENT-MINE—A SemantiSearch Approach to Program Comprehension from Code Comments. In: Chaki, R., Cortesi, A., Saeed, K., Chaki, N. (eds) Advanced Computing and Systems for Security. Advances in Intelligent Systems and Computing, vol 1136. Springer, Singapore. https://doi.org/10.1007/978-981-15-29306_3