This paper presents the system submitted by the team from IIT(ISM) Dhanbad in FIRE IRSE 2023 shared task 1 on the automatic usefulness prediction of code-comment pairs as well as the impact of Large Language Model(LLM) generated data on original base data towards an associated source code. We have developed a framework where we train a machine learning-based model using the neural contextual representations of the comments and their corresponding codes to predict the usefulness of code-comments pair and performance analysis with LLM-generated data with base data. In the oficial assessment, our system achieves a 4% increase in F1-score from baseline and quality of generated data.
CEUR ceur-ws.org
In the rapidly developing world of software development, comments play a crucial role in
enhancing code readability and maintainability of the corresponding codes in the code bases [
However, sometimes the comments themselves may be incomplete, inconsistent, and dificult
to relate to the source code [
Thus understanding the relevance of a comment to a piece of code is crucial before actually CEUR Workshop Proceedings
/*deal with it later*/ -1. /*deal with it later*/1. /*switch on*/ -1.f(toggle) /*switch on*/1.else
for this comment.
using it to understand the purpose of the program. However, given the volume of source code in a standard software project, it is a laborious task to manually verify the usefulness of each comment to their corresponding code. Thus, a system that can automatically predict the usefulness of a comment to its related code snippet may significantly speed up the process of source code analysis. Furthermore, the comment may be rewritten to make it more relevant and informative in case the system predicts the comment to be unuseful.
Recently, artificial intelligence-based interactive systems, such as ChatGPT [
The Task-1 of FIRE 2023 IRSE shared task[
The second subtask is to study the impact of large language models in comments. In this
subtask, the participants are required to augment the base data provided for Subtask 1 with
additional data and to carry out a comparative study of the performance of the models trained
using the base data and augmented data. The additional data for augmentation is expected
to comprise the code-comment pairs obtained from diferent sources with their usefulness
labels predicted using large language models (LLMs)[
This paper reported the comprehensive explanation of the proposed system submitted to FIRE
IRSE2023 for task 1[
The remaining sections are arranged as follows. Section 2, presented the related works where we have done some literature surveys on previous work. In this section 3, we have presented a description of diferent types of LLM-generated datasets and the data made available for the shared task. We have also reported a brief description of data representation and system specification of the system submitted for the shared task. Section 4, presented a comprehensive analysis of the results on diferent runs with the dataset. In Section 5, we have concluded our work on shared tasks.
We have done some surveys on the usefulness of code-comment pairs as well as the impact of
ChatGPT[
Majumdar et al.[
This section presents a detailed discussion of the proposed system developed for automatically predicting the relevance of code-comment pairs and the experiments carried out on the diferent combinations of the data sets. In Subsection 3.1 we present the details of the prediction system. The details for the datasets used for the experiments are reported in Subsection 3.2.
Our prediction system is a supervised machine-learning-based system that consists of a support vector machine (SVM) [15] trained on the distributed representations of the code-comment pairs. It takes the representations of the code-comment pair as input and predicts whether the comment is relevant to the corresponding code snippet.
The distributed representations of code-comment pairs are obtained from a pre-trained ELMO-based model [16]. We have used the ELMO code1 provided by the Information Retrieval in Software Engineering (IRSE) team. For a given code comment pair, we separately pass the code and the comment to the ELMO model[16] as input as a sequence of tokens. For an input sequence, the ELMO model[16] generates 200-dimensional contextual embeddings for each space-separated token in the input sequence. The representation for an input sequence is then obtained by taking the mean of all the token representations. So, the representation of a given input sequence is a 200-dimensional embedding. The 200-dimensional representations of the code and the comment sequences are then concatenated into a joint 400-dimensional representation.
During training, we generate the representations for all the code-comment pairs in the training data and use them to train the support vector machine using the radial basis function (RBF) kernel[17]. During testing, the model saved during the training phase takes the representation of the code-comment pair as input and predicts the usefulness of the comment.
The details of the working of our prediction system is presented in Figure 1.
3.2.1. Original data
Here we present a description of diferent combinations of the used datasets for our experiments.
The original data for task 1[
For Subtask 2, we have manually collected a total of 510 code-comments pairs from diferent
data resources such as GitHub, stack overflow, computer vision, curl, etc., and then query the
ChatGPT[
For the sake of convenience, we referred to the original data, LLM-generated data, and extra-generated data as Data1, Data2, and Data3 respectively as shown in Table 3.
We have followed the following steps to split the original and the LLM-generated[
Thus, the training data comprised a combination of 80% of the code-comment pairs with ”Useful” labels and 80% of the code-comment pairs with ”Not Useful” labels. The selection of the 80% of the samples in both cases was done randomly. The test data consisted of the remaining 20% of the samples from both groups.
We followed the same splitting procedure for the ”LLM-generated data”[
We have created diferent combinations of datasets by combining diferent types of datasets.
The dataset description is given in this subsection ( 3.2.1, 3.2.2, and 3.2.3). Here, we have
combined the diferent data as shown in Table 4 with train test split sizes of data. The purpose
is to create new datasets to understand the impact of system performance on original data,
LLM-generated data[
In this section, we present a discussion of our results. We performed four diferent experiments with diferent combinations of test datasets as shown in Table 5, and Table 6.
We performed experiments with the original base data of size 11,452. Original data is split into train and test of sizes 9162 and 2290 in the ratio of 80:20 shown in Table 4. We used only the test data of the original base data (Dataset1) for usefulness prediction.
Our second experiment was carried out with original data and LLM-generated data[
Our third experiment is with the combination of original base data and extra generated data. A total of 500 extra-generated data are split in the ratio of 80:20 into train and test data sizes are 400 and 100 respectively. Now, the total sum of training data and test data of Dataset3 sizes are 9562 and 2390.
We did one more experiment with the combination of original data, LLM-generated data[
The overall accuracies corresponding to the experiments carried out for Run1 (Subsection 4.1, Run2 (Subsection 4.2), Run3 (Subsection 4.3) and Run4 (Subsection 4.4) are 92.18%, 92.76%, 90.696%, and 92.47% respectively. The results are summarized in Table 5. In Run3 (Dataset3), the accuracy value decreases, and in other Runs (with Dataset1, Dataset2, Dataset3), we are getting almost the same accuracies with slight variation in decimal fractions value.
To evaluate the performance of the system with respect to the ”Useful” class, we have used precision, recall, and F1-score as evaluation metrics. The results are summarized in Table 6. We have carried out diferent runs using their corresponding useful class dataset and evaluated the Useful precision, recall, and F1-score. In Run1 (Useful dataset size -1465) and Run4 (Useful dataset size- 1578), we are getting the same precision, recall, and F1 score. But, in the case of Run2 (Useful dataset size- 1542), it gets slightly higher recall than other Runs but other evaluation parameters remain the same and in Run2 (Useful dataset size- 1470), all evaluation parameters slightly decrease than other Runs.
In this paper, we presented our proposed system submitted for participating in task-1 shared by IRSE FIRE 2023. The first task of shared task-1 is to build a system that takes a code-comment pair as input to the encoder, which generates embedding that is passed to the classifier and the classifier classifies whether the comment that corresponds to the code is ”Useful” or ”Not Useful”. The second task is to make predictions on the augmentation of original seed data and LLM-generated data. We have also done impact analysis and model performance with an augmented dataset(original base data and LLM-generated data). All the performance evaluation metrics parameters are mentioned in Table 5 and Table 6. According to the declared result, our system achieves a 4% increase in F1-score from baseline and quality of data generated. real summaries through blinded reviewers and text classification algorithms, 2023. arXiv:2303.17650. [11] M. Naili, A. H. Chaibi, H. H. B. Ghezala, Comparative study of word embedding methods in topic segmentation, Procedia computer science 112 (2017) 340–349. [12] S. Majumdar, S. Papdeja, P. P. Das, S. K. Ghosh, Comment-mine—a semantic search approach to program comprehension from code comments, Advanced Computing and Systems for Security: Volume Twelve (2020) 29–42. [13] S. Majumdar, A. Bansal, P. P. Das, P. D. Clough, K. Datta, S. K. Ghosh, Automated evaluation of comments to aid software maintenance, Journal of Software: Evolution and Process 34 (2022) e2463. [14] S. Majumdar, A. Varshney, P. P. Das, P. D. Clough, S. Chattopadhyay, An efective lowdimensional software code representation using bert and elmo, in: 2022 IEEE 22nd International Conference on Software Quality, Reliability and Security (QRS), IEEE, 2022, pp. 763–774. [15] C. Cortes, V. Vapnik, Support-vector networks, Machine learning 20 (1995) 273–297. [16] M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, L. Zettlemoyer, Deep contextualized word representations, Association for Computational Linguistics (2018) 2227–2237. URL: https://aclanthology.org/N18-1202. doi:10.18653/v1/N18- 1202. [17] K. Thurnhofer-Hemsi, E. López-Rubio, M. A. Molina-Cabello, K. Najarian, Radial basis function kernel optimization for support vector machine classifiers, arXiv preprint arXiv:2007.08233 (2020).