This work addresses the challenge of fine-grained misinformation detection in Large Language Model (LLM) generated summaries for Indian languages. We focus on Subtask 2: Detect Misinformation in LLM Output, which requires classifying LLM-generated summaries into specific categories of factual incorrectness. The task is particularly dificult for regional languages like Tamil, Telugu, Malayalam, and Kannada due to limited annotated resources. To tackle this, we fine-tune IndicBERTv2-MLM-only, a multilingual transformer model pre-trained on Indian languages. Our methodology uses article-summary pairs with stratified sampling and optimizes for macro-F1 score across four distinct classes: misrepresentation, fabrication, false attribution and incorrect quantities. We fine-tune separate IndicBERTv2-MLM-only models for each target language using identical architectures and hyperparameters. The models achieve competitive results given the complexity of the fine-grained classification task, with cross-lingual averages of 71.69% accuracy and 46.69% macro-F1, demonstrating better handling of multilingual complexity.
Information plays a crucial role in shaping public opinion, especially during sensitive periods such as elections, conflicts, and pandemics. When false or misleading content spreads unchecked, it can amplify misunderstandings, trigger public panic, and drive impulsive decisions that may escalate civil unrest or threaten national stability. Efectively identifying and managing misinformation is therefore essential, both to enable targeted remediation and to protect the credibility of news outlets and online platforms.
This work is part of the PROMID Shared Task at FIRE 2025, which focuses on Misinformation
Detection and Prompt Recovery [
Previous approaches have largely relied on traditional machine learning models such as Support
Vector Machines and Logistic Regression trained on English datasets, which do not generalize well
to multilingual or low-resource settings. More recent multilingual transformer-based models (e.g.,
BERT variants) [
The key gap we address is the lack of multilingual support for South Indian (Dravidian) languages
in fine-grained misinformation detection. In particular, existing systems rarely (i) target regional
languages such as Tamil, Telugu, Kannada, and Malayalam, and (ii) jointly use both the news article and
its LLM-generated summary for multi-class classification across distinct misinformation categories. To
bridge this gap, we fine-tune IndicBERTv2-MLM-only [
Our experimental results show that the proposed language-specific approach achieves promising performance across the four languages: Kannada (F1: 52.75%, Accuracy: 75%), Telugu (F1: 48.00%, Accuracy: 72.25%), Malayalam (F1: 47.00%, Accuracy: 71.50%), and Tamil (F1: 39.00%, Accuracy: 68.00%). On average, the models attain 71.69% accuracy and 46.69% macro-F1 across languages, demonstrating the efectiveness of our language-isolated fine-tuning strategy in handling multilingual complexity and LLM-induced summarization errors in low-resource South Indian language settings.
Raja et al. introduce one of the first systematic frameworks for fake news detection in Dravidian languages using transfer learning, fine-tuning multilingual encoders such as mBERT and XLM-R on Tamil, Malayalam, and Kannada news datasets [7]. Their work clearly demonstrates that transformer-based models substantially outperform classical baselines and confirms the value of pre-trained multilingual representations for low-resource Indian languages. However, the task formulation is binary (fake vs. real) and each instance is treated as a single news headline or article. The models do not consider the interaction between an original article and a derived text (e.g., a summary), nor do they distinguish between diferent types of factual errors such as misrepresentation or incorrect quantities. As a result, they cannot explain how a piece of content is wrong, only that it is likely fake.
Subsequent shared-task systems at DravidianLangTech extend this line of work by exploring a broader set of architectures, including monolingual BERT variants (e.g., Malayalam-BERT) [8] and ensembles that combine multiple multilingual transformers in shared-task systems at DravidianLangTech [9] and team reports such as CIC-NLP [10]. These systems achieve strong accuracy on fake-news benchmarks, but they inherit similar conceptual limitations: they focus on binary or coarse multi-class labels, and their inputs are almost always standalone texts (tweets, posts, headlines, or full articles). As a consequence, they are not exposed to paired inputs where one text (an LLM-generated summary) may selectively distort another (the source article), and they are not designed to detect hallucinations or summarisation-specific distortions introduced by LLMs.
Most misinformation detection systems are built for English or high-resource languages, often using machine translation for others [11], which introduces noise and fails to capture language-specific phenomena in South Indian languages [12]. These systems also typically treat misinformation as binary, lacking fine-grained categories. In contrast, we target four South Indian languages—Tamil, Telugu, Kannada, and Malayalam—using article-summary pairs to detect inconsistencies in LLMgenerated summaries. We fine-tune separate IndicBERTv2-MLM-only models for each language with identical training protocols, performing four-way classification across misrepresentation, fabrication, false attribution, and incorrect quantities, evaluated with macro-F1 for consistent multilingual comparison.
Given a piece of LLM-generated text containing misinformation, the objective is to categorize each datapoint into one of four specific categories—Misrepresentation, Fabrication, False Attribution, and Incorrect Quantities—based on the nature and type of factual incorrectness present in the summaries. This task addresses the growing challenge of detecting nuanced factual errors in LLM outputs, where traditional binary classification fails to capture the diverse ways large language models can introduce inaccuracies.
The PROMID dataset reflects the current scenario in LLM-generated misinformation research, where models increasingly produce plausible but factually flawed content across low-resource South Indian languages (Tamil, Telugu, Kannada, Malayalam). With 989 samples per language showing moderate class imbalance, the dataset captures real-world distribution patterns while enabling fine-grained analysis of diferent error types prevalent in LLM summaries
The dataset for Subtask 2 on misinformation detection in LLM-generated text [13], provided by the task organizers, contains 989 paired samples for each of the four South Indian languages—Tamil, Telugu, Kannada, and Malayalam—where each sample pairs a large language model-generated summary with its corresponding original article, meticulously annotated into four specific misinformation categories: Misrepresentation, Fabrication, False Attribution, and Incorrect Quantities.Table 1 presents the distribution of samples across these categories.
Misinformation Category
Samples
Percentage Misrepresentation Fabrication False Attribution Incorrect Quantities Total
Prior to model training, the dataset was loaded and preprocessed to ensure consistency and reliability. Column names were standardized, and samples with missing summaries or misinformation labels were removed. The incorrect summaries and their corresponding misinformation categories were retained for further analysis. Each category—Misrepresentation, False Attribution, Incorrect Quantities, and Fabrication—was encoded into numerical labels using a fixed and reproducible label mapping.
For Subtask 2: Misinformation Detection in LLM-Generated Text, our approach fine-tunes separate IndicBERTv2-MLM-only models for each target language (Tamil, Telugu, Kannada, Malayalam) using the HuggingFace Transformers library. Each model follows the standard BERT-base configuration with 12 transformer layers, 12 attention heads, and 768-dimensional embeddings. For classification, we use the [CLS] token representation passed through a linear projection layer to produce four output units corresponding to the misinformation categories. Input text is formatted as “Article: {text} Summary: {text}”, tokenized with a maximum length of 384 tokens. We implement an 80/20 stratified train-validation split to preserve class distribution and use macro-F1 scoring for model selection to address class imbalance. The training hyperparameters are shown in Table 2.
The IndicBERTv2-MLM-only model shows consistent convergence with training loss dropping steadily from ∼ 1.39 (Epoch 1) to < 0.80 (Epoch 8). Validation F1 peaks mid-training (Epochs 4-6) before stabilizing.
Overall performance (macro-averaged) shows precision of 0.80, recall of 0.49, F1-score of 0.53, and accuracy of 0.75, reflecting a model that is generally precise in its predictions but still tends to underrecall certain under-represented classes.
Overall, the model achieves a macro-averaged precision of 0.6725, recall of 0.4275, F1-score of 0.47, and accuracy of 0.715. These results show that the classifier is relatively precise in its predictions but still under-recognizes several true positive instances, especially for the fabrication class.
Overall, the model achieves a macro-averaged precision of 0.5225, recall of 0.3625, F1-score of 0.39, and accuracy of 0.68. These results suggest that while predictions are moderately precise, the model still under-detects several true positives, particularly in the less frequent misinformation categories.
Overall, the Telugu model attains a macro-averaged precision of 0.63, recall of 0.4325, F1-score of 0.48, and accuracy of 0.7225. This indicates reasonably precise predictions, but with room for improvement in recall, especially for fabrication and incorrect_quantities, where many true instances are still missed.
This research presents a comprehensive framework for fine-grained misinformation detection in South Indian languages. Our approach leverages IndicBERTv2, a transformer model pre-trained on major Indian languages, adapting it through transfer learning to distinguish between four distinct categories of misinformation: fabrication,false attribution,incorrect quantities,misrepresentation.
We address several critical challenges in multilingual misinformation analysis. First, we implement a training strategy with macro-F1 optimization to mitigate class imbalance, ensuring balanced performance across all categories regardless of their frequency in the dataset. Second, we establish an evaluation framework that prioritizes macro-F1 scores alongside accuracy, providing a more comprehensive assessment of model performance. Third, we design a practical two-stage inference pipeline that filters pre-verified content to optimize computational eficiency during deployment, making the system suitable for real-world applications.
The technical implementation involves fine-tuning for eight epochs with a learning rate of 1 × 10 −5 , AdamW optimization with weight decay, linear warmup scheduling, and automatic selection of the best model checkpoint based on validation performance. Our methodology demonstrates that transfer learning from multilingual pre-trained models efectively addresses the linguistic diversity and data scarcity challenges inherent in South Indian language processing.
This work contributes to the field of computational linguistics for social good by providing a scalable, nuanced approach to misinformation detection that moves beyond binary classification. The system ofers practical utility for content moderation, fact-checking organizations, and platform operators working with South Indian language content, while establishing methodological foundations for future research in low-resource multilingual NLP applications.
During the preparation of this work, we used OpenAI’s ChatGPT model for language fluency improvement and technical documentation assistance. All experimental work, data analysis, results, and scientific conclusions are our own. We have reviewed and refined all AI-assisted content and take full responsibility for the published work. [7] E. Raja, B. Soni, S. Bhat, Fake news detection in dravidian languages using transfer learning,
Engineering Applications of Artificial Intelligence 121 (2023) 106877. [8] F. Chakraborty, Others, Malayalam-bert based transformer models for fake news detection, in:
Proceedings of DravidianLangTech 2025, Association for Computational Linguistics, 2025. [9] F. Shanmugavadivel, Others, Overview of the shared task on fake news detection in dravidian languages, in: Proceedings of the Workshop on Speech and Language Technologies for Dravidian Languages (DravidianLangTech 2025), Association for Computational Linguistics, Turin, Italy, 2025. [10] CIC-NLP, Cic-nlp@dravidianlangtech 2025: Fake news detection in dravidian languages using multilingual transformers, in: Proceedings of DravidianLangTech 2025, Association for Computational Linguistics, 2025. [11] X. Wang, Others, Monolingual and multilingual misinformation detection for low-resource languages: A comprehensive survey, arXiv preprint arXiv:2410.18390 (2024). [12] A. Goyal, A. Basu, D. Sharma, Challenges in neural machine translation for dravidian languages: Morphology, script variation, and data noise, in: Proceedings of the Workshop on Indian Language Data: Resources and Evaluation (WILDRE), Association for Computational Linguistics, European Language Resources Association (ELRA), Dublin, Ireland, 2022. URL: https://aclanthology.org/ 2022.wildre-1.8. [13] S. Satapara, P. Mehta, D. Ganguly, S. Modha, Fighting fire with fire: Adversarial prompting to generate a misinformation detection dataset, CoRR abs/2401.04481 (2024). URL: https://doi.org/10. 48550/arXiv.2401.04481. doi:10.48550/ARXIV.2401.04481. arXiv:2401.04481.