We evaluate domain-adaptive pre-training of CodeBERT using masked language modeling (MLM) for binary codecomment relevance classification in the IRSE shared task. We compare CodeBERT+MLM against TF-IDF baselines and study the efect of synthetic and silver-standard augmentation. CodeBERT+MLM attains F 1=0.9193 on the original test split versus 0.7067 for the best TF-IDF baseline (logistic regression). Augmentation (combined data) produces a marginal change (F1=0.9166), highlighting that augmentation quality - not quantity - drives gains. We provide implementation details and practical recommendations for applying domain-adaptive pre-training in software engineering tasks.
Transformers adapted for code, such as CodeBERT, enable joint code–text representations useful for
search and summarization [
We use the IRSE/FIRE seed dataset (11,452 code–comment pairs extracted from C projects). After cleaning malformed or unlabeled entries we obtain 8,326 training samples (“Original”). To expand data we used two augmentation sources: 1. Rule-based synthetic: 500 examples generated by templated paraphrasing (synonym substitution, sentence restructuring) derived from existing comments to increase stylistic diversity while preserving labels. 2. HumanEval-derived silver standard: 1,640 examples extracted from human-eval.jsonl. We parsed each example to extract function docstrings and surrounding comments (mostly Python), then used an OpenRouter “fast-grok” free model as a preprocessing / normalization step to: • split function-level docstrings into candidate comment sentences, • canonicalize parameter and return descriptions, • filter out non-descriptive or placeholder docstrings (e.g., single-word comments). The normalized candidates were programmatically labeled using heuristics (presence of functionlevel description, param/return mentions, and overlap with code identifiers). These heuristics produce silver-standard labels (higher noise than human annotation but useful signal).
We then combined the C-based Original training set with Python-derived silver-standard and rulebased synthetic examples to form the Combined training set (8,826 samples). Table 1 summarizes composition.
Code and comment texts were concatenated with a separator token. Tokenization uses CodeBERT’s fast BPE tokenizer with a maximum sequence length of 512. For silver-standard examples we applied two lightweight filters: (1) minimum token length (3 words) and (2) presence of at least one verb or function-related identifier, to reduce trivial or metadata-only docstrings. The silver-standard labeling heuristics prioritize precision (favoring conservative "useful" labels) but inevitably inject label noise; we therefore treat these examples as complementary rather than authoritative.
We continued MLM from microsoft/codebert-base-mlm on the Original (C) training corpus prior to supervised fine-tuning (15% mask, batch size 16, lr 5e-5, max 7 epochs, AdamW). After DAPT we replace the MLM head with a classification layer (dropout 0.1) and fine-tune on the binary task (batch size 32, lr 2e-5, up to 10 epochs, linear warmup 10% steps). Baselines use TF–IDF features with: logistic regression (L2), linear SVM, multinomial NB, and random forest (100 trees). Baselines use 10-fold CV for hyperparameter tuning.
Metrics: accuracy, precision, recall, weighted F1. Experimental configurations: (A) train Original → test Original; (B) train Combined → test Combined; (C) Original → Combined; (D) Combined → Original. Results are reported on held-out test splits not used during validation.
Although silver examples derive from Python docstrings, the MLM DAPT step was performed primarily on C code and comments (Original corpus). Two reasons explain why DAPT remained efective after adding Python silver examples for classification: 1. Cross-language lexical overlap: many identifier names, API-style phrases, and comment conventions (e.g., “returns”, “params”, “compute”) are shared across languages; the C-trained MLM learns these patterns and applies them when encoding Python-derived comments. 2. Representation robustness: MLM adapts CodeBERT’s token and contextual embeddings to code/comment distributions (punctuation, identifier tokens, inline code snippets). These representational improvements transfer to mixed-language fine-tuning because the downstream classifier primarily relies on semantic patterns rather than language-specific syntax.
Empirically, mixing Python silver data produced only a slight drop in peak F1 (0.9193 → 0.9166), indicating that distributional mismatch is limited and quality/noise in silver labels is the larger contributor to performance change.
We compare our best performance against published IRSE-relevant works (table below uses provided summary metadata). This contextualizes gains from DAPT+MLM.
Our CodeBERT+MLM result (F1=0.9193) exceeds many classical/ML+LLM-augmented submissions while remaining below reported very large LLM one-of results (T7-9/T7-10) that use generative LMs as primary models. The comparison suggests DAPT is a practical and efective middle ground: much better than TF–IDF baselines and less resource-intensive than training or prompting very large LLMs end-to-end.
Domain-adaptive MLM significantly improves CodeBERT representations for comment relevance. Crossevaluation shows training on Original and testing on Combined drops F1 by only 0.52pp, indicating robust transfer. The small degradation with augmentation confirms that label quality and distribution alignment matter: silver-standard Python-derived examples add vocabulary and diversity, but their programmatic labels and stylistic diferences introduce noise that slightly ofsets gains.
Limitations: single-language (C) focus in DAPT, binary relevance label space, and the computational cost for continued pretraining. Future work: multi-task DAPT, curriculum strategies that gradually introduce silver examples, active human verification of synthetic examples, and interpretability analyses to surface tokens driving decisions.
We show CodeBERT with domain-adaptive MLM attains strong results on comment relevance classification, outperforming TF–IDF baselines by a wide margin. Silver-standard augmentation from HumanEval (processed via OpenRouter fast-grok + heuristics) provides additional data but must be quality controlled to benefit high-capacity transformers. DAPT is a practical, resource-eficient strategy to improve specialized SE NLP tasks.
In the course of preparing this manuscript, the author(s) employed the generative AI tool ChatGPT. Its use was limited to performing checks for grammar and spelling. Following this, the author(s) conducted a thorough review and revision of the text and assume full responsibility for the final published content. [13] Feng, S.Y., et al.: A survey of data augmentation approaches for NLP. In: ACL Findings, pp. 968–988 (2021). https://doi.org/10.18653/v1/2021.findings-acl.84 [14] Majumdar, S., et al.: Overview of the Information Retrieval in Software Engineering Track at FIRE 2023. In: CEUR Workshop Proceedings, vol. 3681 (2023). https://ceur-ws.org/Vol-3681/T7-1.pdf [15] Arumugam, R., Deborah, A.: Enhancing binary code comment quality classification: Integrating generative AI for improved accuracy. In: CEUR Workshop Proceedings, vol. 3681 (2023). https: //ceur-ws.org/Vol-3681/T7-6.pdf [16] Binary classification of source code comments using machine learning. In: CEUR Workshop
Proceedings, vol. 3681 (2023). https://ceur-ws.org/Vol-3681/T7-7.pdf [17] Majumdar, S., Deshpande, A., Das, P.P., Chakrabarti, P.P.: Comprehending C Codes with LLMs: Efective Comment Generation through Retrieval and Reasoning. Pattern Recognition Letters (2025). Elsevier. [18] Paul, S., Majumdar, S., Shah, R., Das, S., Ghosh, M., Ganguly, D., Calikli, G., Sanyal, D., Das, P.P., Clough, P.D., et al.: Overview of the “Information Retrieval in Software Engineering”(IRSE) track at Forum for Information Retrieval 2024. In: Proceedings of the 16th Annual Meeting of the Forum for Information Retrieval Evaluation, pp. 18–21 (2024). [19] Paul, S., Majumdar, S., Bandyopadhyay, A., Dave, B., Chattopadhyay, S., Das, P., Clough, P.D., Majumder, P.: Eficiency of Large Language Models to scale up Ground Truth: Overview of the IRSE Track at Forum for Information Retrieval 2023. In: Proceedings of the 15th Annual Meeting of the Forum for Information Retrieval Evaluation, pp. 16–18 (2023). [20] A. Deshpande, A. Maji, D. Mondol, P. P. Das, P. D. Clough, and S. Majumdar, “The Code–LLM Handshake: Smarter Maintenance Through AI,” in Proceedings of the 17th Annual Meeting of the Forum for Information Retrieval Evaluation, pp. 9–12, 2025. [21] A. Mitra, S. Majumdar, A. Mukhopadhyay, P. P. Das, P. D. Clough, and P. P. Chakrabarti, “Operationalizing Large Language Models with Design-Aware Contexts for Code Comment Generation,” arXiv preprint arXiv:2510.22338, 2025. [22] S. Majumdar and P. P. Das, “Smart Knowledge Transfer using Google-like Search,” arXiv preprint arXiv:2308.06653, 2023.