Automating code documentation through explanatory text can prove highly beneficial in code understanding. Large Language Models (LLMs) have made remarkable strides in Natural Language Processing, especially within software engineering tasks such as code generation and code summarization. This study specifically delves into the task of generating natural-language summaries for code snippets, using various LLMs. The findings indicate that Code LLMs outperform their generic counterparts, and zero-shot methods yield superior results when dealing with datasets with dissimilar distributions between training and testing sets.
IRSE conala-train 1666 re.sub(’[^A-Z]’, ”, s)
Code explanation [
Large Language Models have been successfully employed in a wide variety of natural
language generation tasks [17]. The zero shot and few shot capabilities of these systems make
them highly adaptable to any NLP task. There are several general domain, open source LLMs
like LLama-2 [
Large Language Models have been used for Code explanation in a few shot setting [
In this work, we consider a dataset of 100 samples released at the Information Retrieval in
Software Engineering (IRSE) track at Forum for Information Retrieval Evaluation (FIRE) 2023 [
Table 1 shows a few examples from both the datasets. It can be observed that while the code snippets in are comparable in length (21 and 14 tokens respectively), the code explanations in the IRSE dataset are lengthier (mean length = 84 words) than the ones in the conala-train set (mean length = 15 words).
The model generated textual descriptions are evaluated with respect to the ground truth explanations using the following measures: (i) Token-based: BLEU [24] score combines precision scores of n-grams (typically up to 4grams) using weighted geometric mean, with higher weight given to shorter n-grams. BLEU-1, BLEU-2, and BLEU-N (for any integer N) extend the evaluation to unigrams, bigrams, and n-grams of varying lengths, respectively. (ii) Semantics-based: We use this measure to assess the semantic similarity between the model generated explanation () and the ground truth explanation (). We project both and in a continuous embedding space→,− an→d− respectively using the pretrained CodeBERT [25] model. We then take a cosine similarity between the embeddings →(−→,− ) to get the score.
We experiment with 5 LLMs (i) Generic LLM: Llama-2-70B-Chat model [
We load the CodeUp-13B-Chat model with 4-bit quantization using QLoRA [26] and bitsandbytes [27] methods. We then perform parameter-eficient finetuning (PEFT) [ 28] of the model using the above prepared dataset.
Table 3 shows the performance of the 5 diferent LLMs over three approaches – zero-shot, few-shot and zero-shot over the Instruction finetuned model. CodeLlama-13B-Instruct and Llama-2-Coder-7B have the best zero-shot performance over the other LLMs. Note that although the generic Llama2 model is the largest in size (70B), it has poor performance when compared to the smaller Code LLM models (13B, 7B). This shows that domain specific models perform better than generic ones.
While the few shot strategy is expected to give better performance than zero-shot, in this study we find that the performance is worse. This is mainly because the few shot examples had been selected from the conala-train set. As discussed in Section 3 and Table 1 the code explanation lengths in the IRSE dataset and the conala-train dataset vary hugely. Since the LLMs see few shot examples from the conala-train, it generates shorter length code explanations for input samples coming from the IRSE dataset. This train-test distribution mismatch causes the models to perform worse in the few shot scenario as compared to the zero-shot.
Similar arguments can be drawn for the Instruction finetuning+Zero shot approach, as the training data comes from the conala-train dataset which is diferent from the IRSE dataset.
In this work we explore the performance of 5 LLMs, both generic and code-specifc, for the task of code explanation. We use zero-shot, few shot and instruction finetuning approaches over the LLMs and assess their performance. We find that Code LLMs perform better than larger generic LLMs. Also, zero-shot prompting works well in the scenario where we do not have enough examples to prompt/finetune the model. Comprehension (ICPC), IEEE, 2013, pp. 13–22. [16] L. Moreno, J. Aponte, G. Sridhara, A. Marcus, L. Pollock, K. Vijay-Shanker, Automatic generation of natural language summaries for java classes, in: 2013 21st International conference on program comprehension (ICPC), IEEE, 2013, pp. 23–32. [17] J. Yang, H. Jin, R. Tang, X. Han, Q. Feng, H. Jiang, B. Yin, X. Hu, Harnessing the power of llms in practice: A survey on chatgpt and beyond, arXiv preprint arXiv:2304.13712 (2023). [18] R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, T. B. Hashimoto, Alpaca: A strong, replicable instruction-following model, Stanford Center for Research on Foundation Models. https://crfm. stanford. edu/2023/03/13/alpaca. html 3 (2023) 7. [19] G. Penedo, Q. Malartic, D. Hesslow, R. Cojocaru, A. Cappelli, H. Alobeidli, B. Pannier, E. Almazrouei, J. Launay, The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only, arXiv preprint arXiv:2306.01116 (2023). [20] B. Rozière, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y. Adi, J. Liu, T. Remez, J. Rapin, et al., Code llama: Open foundation models for code, arXiv preprint arXiv:2308.12950 (2023). URL: https://huggingface.co/codellama. [21] Manuel Romero, llama-2-coder-7b (revision d30d193), 2023. URL: https://huggingface.co/ mrm8488/llama-2-coder-7b. doi:10.57967/hf/0931. [22] M. Geng, S. Wang, D. Dong, H. Wang, G. Li, Z. Jin, X. Mao, X. Liao, Large language models are few-shot summarizers: Multi-intent comment generation via in-context learning (2024). [23] P. Yin, B. Deng, E. Chen, B. Vasilescu, G. Neubig, Learning to mine aligned code and natural language pairs from stack overflow, in: International Conference on Mining Software Repositories, ACM, 2018, pp. 476–486. URL: https://conala-corpus.github.io/. [24] K. Papineni, S. Roukos, T. Ward, W.-J. Zhu, Bleu: A method for automatic evaluation of machine translation, Association for Computational Linguistics, USA, 2002, p. 311–318. [25] Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, et al., Codebert: A pre-trained model for programming and natural languages, in: Findings of the Association for Computational Linguistics: EMNLP 2020, 2020, pp. 1536–1547. [26] T. Dettmers, A. Pagnoni, A. Holtzman, L. Zettlemoyer, Qlora: Eficient finetuning of quantized llms, 2023. arXiv:2305.14314. [27] T. Dettmers, M. Lewis, S. Shleifer, L. Zettlemoyer, 8-bit optimizers via block-wise quantization, 9th International Conference on Learning Representations, ICLR (2022). [28] S. Mangrulkar, S. Gugger, L. Debut, Y. Belkada, S. Paul, B. Bossan, Peft: State-of-the-art parameter-eficient fine-tuning methods, https://github.com/huggingface/peft, 2022.