Ofensive Language Identification (OLI) has become an important task, particularly in the context of social media, where harmful content such as hate speech, cyberbullying, and toxic discourse can spread rapidly and widely. The challenge becomes even more complex in low-resource, code-mixed settings across Dravidian languages such as Tamil, Malayalam, Kannada, and Tulu. To address these challenges, Ofensive Language Identification in Dravidian Code-Mixed Languages shared task at FIRE 2025 released gold-standard annotated datasets of comments collected from online platforms in these four languages, with the goal of developing automated systems that can classify user comments as either ofensive or non-ofensive. We - team MUCS, participated in the shared task on OLI by developing two complementary pipelines: i) Of_ML - an ensemble of traditional Machine Learning (ML) estimators (Logistic Regression (LR), Support Vector Machines (SVM), Random Forest (RF), and Naïve Bayes (NB) classifiers) with hard and weighted voting, trained with Term Frequency-Inverse Document Frequency (TF-IDF) of word ngrams in the range (1-3) as features, and ii) Of_DL - a hybrid Deep Learning (DL) architecture that combines Convolutional Neural Network (CNN) with Long Short-Term Memory (LSTM). These pipelines highlight the comparative strengths of feature-based ML models and representation-driven DL architectures, while emphasizing the benefits of ensembling for multilingual ofensive language detection. Experimental results demonstrated that the classical ML pipeline consistently outperformed the DL architecture in all four languages. For Tamil, Of_ML model with hard voting obtained a macro-averaged F1-score of 0.452 (Rank 3) compared to that of DL's 0.350 (Rank 8). For Malayalam, Of_ML model with weighted voting reached 0.712 macro-averaged F1-score (Rank 4), while that of DL model achieved only 0.350 (Rank 8). For Kannada, Of_ML model with hard voting obtained a macro-averaged F1-score of 0.421 (Rank 5), whereas DL model managed to get 0.344 macro-averaged F1-score (Rank 8). Finally, for Tulu, Of_ML model with weighted voting delivered its strongest performance with 0.790 macro-averaged F1-score (Rank 2), while DL model recorded that of 0.610 (Rank 7). These results highlight that, despite the increasing prominence of neural methods, traditional ML models with careful feature engineering remain highly competitive and, in several cases, superior for OLI in low-resource, code-mixed Dravidian languages such as Tamil, Malayalam, Kannada, and Tulu.
OLI is an important application in Natural Language Processing (NLP), particularly in the context of
social media platforms where these platforms serve as primary spaces for public discourse. Online
platforms often sufer from toxic interactions, hate speech, cyber bullying, and abusive content, making
the detection of ofensive language a necessary step toward ensuring safe and inclusive communication.
Unlike monolingual formal text, user-generated social media content tends to be short, noisy, and
highly informal, presenting significant challenges for automated text processing systems [
Dravidian languages such as Tamil, Malayalam, Kannada, and Tulu are frequently mixed with English
in online discourse, producing a unique set of challenges. Earlier studies of OLI in Dravidian languages
[
To address the challenges of OLI in code-mixed Dravidian languages, Ofensive Language Identification
in Dravidian Code-Mixed Languages [
We - team MUCS, participated in this shared task with two distinct pipelines, aiming to systematically compare traditional feature-based ML models with modern DL architectures. The first pipeline - Of_ML, relies on TF–IDF features to train the ensemble of ML estimators (LR, SVM, RF, and NB) with hard and weighted voting. The second pipeline - Of_DL, explores a hybrid CNN–LSTM model to capture semantic and contextual information more efectively. Our code is available on GitHub 1. Our experimental results reveal a clear performance gap between the two approaches. For Tamil, the ML pipeline with hard voting achieved a macro-averaged F1-score of 0.452 (Rank 3), while the DL pipeline settled with that of 0.350 (Rank 8). For Malayalam, ML with weighted voting delivered macro-averaged F1-score 0.712 1https://github.com/rachanabn20/Ofensive-Language-Identification-in-Dravidian-Code-Mixed-Languages---DravidianCodeMix-FIRE-2025 (Rank 4), compared to that of DL’s 0.350 (Rank 8). For Kannada, ML model with hard voting secured 0.421 macro-averaged F1-score (Rank 5), while DL reached that of 0.344 (Rank 8). Finally, for Tulu, ML with weighted voting recorded its strongest result with 0.790 macro-averaged F1-score (Rank 2), whereas DL achieved its best overall performance with 0.610 macro-averaged F1-score (Rank 7). These ifndings emphasize that, despite the increasing popularity of DL approaches, traditional ML methods with careful feature engineering remain highly efective in low-resource and code-mixed language scenarios. Moreover, while DL models showed some promise in Tulu, their overall performance was inconsistent, highlighting the need for further research in adapting neural architectures for complex multilingual and code-mixed environments.
The subsequent sections of this paper details the related works (Section 2), methodology (Section 3), experiments, results, and implications of our approach (Section 4) followed by conclusion and future works (Section 5).
The automatic detection of ofensive and abusive language has been widely studied in recent years to
advance research and develop robust systems for automatic OLI, particularly in Indian languages and
code-mixed social media texts. The TRAC shared task [
Early surveys by Schmidt and Wiegand [
Saumya et al. [13] compared traditional and neural approaches for OLI, reinforcing the importance
of character- and word-level representations in noisy code-mixed data. Chakravarthi et al. [
Prajnashree et al. [14] proposed traditional ML models (SVM, RF, Passive Aggressive Classifier) and Siamese LSTM for OLI in low-resource Indian languages on HASOC 2023 datasets. The results illustrated that SVM achieved strong results for Sinhala (macro F1 = 0.78, Rank 11) and Siamese LSTM for Gujarati (macro F1 = 0.72, Rank 12). Although efective in low-resource conditions, these models lagged behind transformer-based approaches due to limited contextual understanding. Fazlourrahma et al. [15] introduced COOLI, a system for code-mixed OLI in Tamil-English (Ta–En), Malayalam-English (Ma–En), and Kannada-English (Kn–En). They proposed two models: (i) COOLI-Ensemble (MLP, XGBoost, and LR in a voting setup) and (ii) COOLI-Keras (dense neural network). COOLI-Ensemble achieved the best performance, ranking 1st for Ma–En (F1 = 0.97), 4th for Ta–En (F1 = 0.75), and 6th for Kn–En (F1 = 0.69). Despite these strong results, dataset imbalance and inconsistent Romanization limited cross-lingual generalization.
Overall, prior work underscores the dual challenge of OLI and code-mixing, motivating the need for specialized approaches that balance classical feature-based ML approaches with DL models tailored for low-resource, code-mixed settings.
Our approach to OLI in Dravidian code-mixed texts consists of classical ML and DL models. The methodology includes text pre-processing, feature extraction, model training, and evaluation. We developed two independent pipelines, namely Of_ML and Of_DL, to systematically analyze performance trade-ofs between traditional ML models and neural methods for OLI in four Dravidian languages. The steps involved in the methodology are given below:
The dataset comprised user-generated code-mixed texts in four Dravidian languages: Kannada, Malayalam, Tulu, and Tamil. These texts contained multiple classes, including few types of ofensive categories and a non-ofensive category. Due to the noisy nature of user-generated content, the following preprocessing steps are applied to clean and prepare the text for further processing: • Normalization: Repeated characters are reduced (e.g., soooo → soo) to handle exaggerated expressions. • Noise Removal: URLs, user mentions, emojis, hashtags, numbers, and non-alphanumeric symbols are removed. • Stopwords: Minimal stopword removal was performed to avoid loosing functional words that may contribute to ofensive tone. • Transliteration Variants: Transliterated forms (e.g., phonetic spellings in Roman script such as maga (Roman script) vs. maga (native Kannada script) are retained as it is without exlicit normalization.
Feature representations difered significantly across the two pipelines: • Features for Of_ML Models: Word-level -grams in the range (1–3) (unigrams, bigrams, and trigrams) are extracted to capture explicit lexical signals pertaining to ofensive expressions and handle spelling variations. These features are then vectorized using the TfidfVectorizer2 with the vocabulary size restricted to 15,000 features. Even though the feature set is highdimensional and sparse, it is highly efective for linear classifiers. However, this work does not include the analysis of TF–IDF features contributing to classification performance, nor does it provide inspection of the most discriminative -grams. Further, no explicit linguistic features are incorporated to address the challenges arising from code-mixed or morphologically rich text, which are common in multilingual social media data. • Features for Of_DL Models: Each word token in the dataset is mapped to dense vector representation using an embedding layer with randomly initialized weights. 2https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html
The two pipelines are trained with diferent features and frameworks of these pipelines are shown in Figures 1 and 2.
• Of_ML Model: The following estimators are used in the ensemble model to enhance the performance of the classifier: – Logistic Regression3: is a linear classifier efective in high-dimensional sparse spaces. – Support Vector Machine4: is a margin-based classifier that maximizes class separation. – Random Forest5: is an ensemble of decision trees that improves robustness to feature noise.
– Naïve Bayes6: is a probabilistic classifier well-suited for text data.
These estimators are ensembled with hard voting (predictions from individual classifiers are combined based on majority voting) for Kannada and Tamil, and weighted voting (classifiers with higher validation performance are assigned larger weights) for Tulu and Malayalam. • Of_DL Model: The Hybrid CNN–LSTM model combines CNN layers for local feature extraction with LSTM layers for sequential modeling, balancing short-span and long-range contexts: – Convolutional Neural Network7: learns phrase-level features directly from embeddings. – Long Short-Term Memory8: captures discourse-level and sequential patterns across tokens. 3https://en.wikipedia.org/wiki/Logistic_regression 4https://en.wikipedia.org/wiki/Support_vector_machine 5https://en.wikipedia.org/wiki/Random_forest 6https://en.wikipedia.org/wiki/Naive_Bayes_classifier 7https://en.wikipedia.org/wiki/Convolutional_neural_network 8https://en.wikipedia.org/wiki/Long_short-term_memory The final representations are fed to a fully connected layer to output probabilities across the ofensive language categories.
Table 2 summarizes the hyperparameters used for both pipelines. While classical ML models are tuned via grid search9, DL models used validation-based tuning.
The proposed pipelines are built and evaluated using the Dravidian languages: Kannada, Malayalam,
and Tamil datasets provided by the organizers of the shared task [
The performance of the models is reported in terms of macro-averaged Precision (mPrecision), macro averaged Recall (mRecall), and macro averaged F1-score (mF1-score) across all the classes. This allows us to capture performance across all the classes in a better way rather than being dominated by the majority class. The participating teams are ranked based on the performances of the models in terms of mF1-score. Tables 4 and 5 present the results of our traditional ML ensemble approaches on Validation and Test sets respectively, while Tables 6 and 7 report the performance of the DL pipelines on Validation and Test sets respectively.
From the results, it is clear that the Of_ML models consistently outperformed the Of_DL models across all languages on the Test sets. For Kannada, Of_ML model achieved mF1-score of 0.42 with 5th rank, compared to Of_DL model’s mF1-score of 0.34 with 8 th rank. For Malayalam, Of_ML model obtained mF1-score of 0.71 (Rank 4), while Of_DL model reached only 0.35 mF1-score (Rank 8). Tulu showed the strongest Of_ML model performance with a mF1-score of 0.79 (Rank 2), substantially better than Of_DL model’s 0.61 mF1-score (Rank 8). Tamil also followed this trend, where Of_ML model scored a mF1-score of 0.45 (Rank 3) but Of_DL model lagged at 0.35 mF1-score (Rank 8). Though Of_DL models captured contextual patterns, TF–IDF based Of_ML models proved significantly more competitive in the shared task.
Figures 3a, 3b, 3c, and 3d illustrate the leaderboard rankings based on mF1-score for Tulu, Tamil, Malayalam, and Kannada respectively, highlighting the relative performance of our systems against other participating teams.
The confusion matrices provide further insight into classification performance by highlighting the misclassification. Figures 4a, 4b, 4c, and 4d illustrate the performances of Of_ML models across Kannada, Malayalam, Tamil, and Tulu datasets, respectively. Confusion matrices for Kannada, Tulu, Malayalam, and Tamil, obtained using Of_DL models are provided in Figures 5a, 5b , 5c and 5d, respectively, to show the contrast with ML pipelines. The low mF1-scores of Of_DL models for Kannada and Tamil, indicate that minority ofensive categories are under-predicted. This mirrors the behavior observed in Of_ML models, though Of_DL pipelines are better at balancing recall across the major and mid-frequency classes.
While the overall results demonstrate that both Of_ML and Of_DL models are capable of handling OLI in multiple langauges, a closer look at the performance across languages reveals important insights: (a) Leaderboard Rankings for Tulu (b) Leaderboard Rankings for Tamil (c) Leaderboard Rankings for Malayalam (d) Leaderboard Rankings for Kannada • Class imbalance has a significant impact in the performances of the classifiers. Minority categories such as Ofensive_Targeted_Insult_Other or Ofensive_Untargeted consistently showed very low recall, often close to zero in Kannada and Tamil. This suggests that both ML and DL systems leaned heavily toward predicting the more frequent Not_Ofensive class, leading to missed detections of subtler ofensive expressions. • Linguistically, both Ofensive_Targeted_Group and Ofensive_Targeted_Individual categories rely on similar cues — such as insults or derogatory expressions — making it dificult for the models to distinguish whether the ofense was aimed at a group or a single person. (c) Tamil (d) Tulu • Language-specific diferences are striking. Malayalam and Tulu achieved higher weighted F1scores, indicating relatively robust detection. These datasets appear cleaner, with fewer ambiguous cases, which may explain the performance boost. In contrast, Kannada and Tamil sufered from noisier data and heavier code-mixing, which made ofensive intent harder to capture. • Code-mixing itself remains one of the most challenging factors. Many social media posts combine English with native Dravidian scripts, producing hybrid structures that traditional ML and DL models struggled with. For example, a sentence might switch languages mid-phrase, obscuring both semantic and syntactic cues that are crucial for ofensive language detection.
These observations highlight that while the models perform reasonably well for some languages, particularly Malayalam and Tulu, further work is needed to improve robustness in noisier and more code-mixed contexts. Addressing class imbalance and exploring advanced contextual embeddings or data augmentation techniques could help bridge this gap in the future.
Table 8 presents representative misclassified samples across languages. Several factors contribute to the mis-classification of Test samples as given below: (c) Tamil (d) Tulu • Code-switching and Mixed Scripts: The classifier struggles when users mix English with regional languages or emojis. For example: “Enjoyed too much. Superb” was labeled as Not_Language but predicted as Not_offensive. • Sarcasm and Context Dependence: Subtle insults or sarcastic remarks are dificult for the model to interpret. Example: “so poor dialogue delivery From Mohanal...” was a targeted insult but predicted as untargeted. • Ambiguity in Ofense Type: The model may detect ofensive tone but fail to decide whether it is targeted or untargeted. Example: “Rashmika ide iro avtava” (targeted insult at an individual) was predicted as untargeted. • Dialect and Spelling Variations: Regional slang and non-standard spellings confuse the classiifer. Example: “Bvc yrt comment manpva ha yavu” (untargeted insult) was predicted as a targeted insult. • Named Entities and Cultural References: The system fails to recognize organizations or cultural groups as insult targets. Example: “RIP kannada film chamber. No proper plannings” was an insult towards a group but predicted as non-language.
• Polite vs. Non-language Confusion: Positive or polite expressions are sometimes misclassified as Not_Language. Example: “Sir dasara habbada shubashayagalu” was actually non-ofensive but predicted as Not_Language. • Entity Type Confusion (Individual vs. Group vs. Other): The classifier struggles to distinguish whether an insult is directed at an individual, a group, or an abstract concept. Example: “Padam pottum... Charithram akanam kett marakkar varanam” was a group insult but predicted as non-ofensive.
The above mis-classification samples illustrate the challenges of noisy code-mixed content which includes inconsistent language mixing, informal spellings, and limited annotated data.
In this work, we - team MUCS explored both ensemble of traditional ML models (Of_ML) and DL Of_DL pipelines for OLI in four Dravidian languages: Kannada, Malayalam, Tulu, and Tamil. The overall results showed that Of_ML models are more efective achieving strong leaderboard rankings — 2nd place for Tulu, 3rd place for Tamil, 4th place for Malayalam, and 5th place for Kannada. In contrast, Of_DL models struggled, consistently ranking 8 th across all languages despite reasonable validation performance. Our analysis highlighted the key challenges: i) Minority ofensive categories such as Ofensive_Targeted_Insult_Other and Ofensive_Untargeted, are severely under-predicted, leading to low mF1-scores (0.32–0.35 for Kannada and Tamil using Of_DL models), ii) Frequent confusions also occurred between related classes such as Ofensive_Targeted_Group and Ofensive_Targeted_Individual, and iii) Malayalam and Tulu benefited from relatively cleaner datasets, while Kannada and Tamil are impacted by noisy, code-mixed inputs that make ofensive intent harder to capture. Looking ahead, there are several promising directions. Leveraging multilingual transformers (e.g., XLM-R, mBERT) or instruction-tuned Large Language Models (LLMs) may enhance the ability of the models to capture subtle cues of ofensiveness. Text augmentation, re-sampling, and synthetic text generation, could help alleviate class imbalance and improve recall for minority classes. Code-mixing aware models and transliteration-based pre-processing are also worth exploring to handle hybrid language scenarios more efectively. Finally, extending these experiments to additional Dravidian and low-resource languages will provide broader insights into the generalizability of ofensive language detection systems. In summary, while we achieved competitive leaderboard performance through ML ensembles, further refinements are required for DL pipelines. Addressing class imbalance, code-mixing, and fine-grained class distinctions remains crucial for developing robust multilingual ofensive language detection systems.
Generative Artificial Intelligence (AI) tools are used in the preparation of this work for language refinement, grammar improvement, and formatting suggestions. The research content, analysis, results, and conclusions are developed independently, and AI tools are not used to generate original research ifndings. [11] B. R. Chakravarthi, R. Priyadharshini, N. Jose, A. Kumar M, T. Mandl, P. K. Kumaresan, R. Ponnusamy, H. R L, J. P. McCrae, E. Sherly, Findings of the Shared Task on Ofensive Language Identification in Tamil, Malayalam, and Kannada, in: B. R. Chakravarthi, R. Priyadharshini, A. Kumar M, P. Krishnamurthy, E. Sherly (Eds.), Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages, Association for Computational Linguistics, Kyiv, 2021, pp. 133–145. URL: https://aclanthology.org/2021.dravidianlangtech-1.17/. [12] P. Patwa, G. Aguilar, S. Kar, T. Solorio, SemEval-2020 Task 9: Overview of Sentiment Analysis of Code-Mixed Tweets, in: Proceedings of the 14th International Workshop on Semantic Evaluation, 2020. [13] S. K. Saumya, D. Khurana, B. R. Chakravarthi, Ofensive Language Identification in Dravidian Code-Mixed Text, in: Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages, 2021. [14] P. M, R. K, A. Hegde, K. G, S. Coelho, H. L. Shashirekha, Taming Toxicity: Learning Models for Hate Speech and Ofensive Language Detection in Social Media Text, in: Working Notes of FIRE 2023 Forum for Information Retrieval Evaluation, 2023. URL: https://ceur-ws.org/Vol-3681/T6-22.pdf. [15] F. Balouchzahi, A. B. K, H. L. Shashirekha, MUCS@DravidianLangTech-EACL2021: Cooli - CodeMixing Ofensive Language Identification, in: Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages, Association for Computational Linguistics, 2021, pp. 323–329. [16] A. M. D, D. Vikram, B. R. Chakravarthi, P. R. Hegde, Overcoming Low-Resource Barriers in Tulu: Neural Models and Corpus Creation for Ofensive Language Identification, 2025. URL: https://arxiv.org/abs/2508.11166. arXiv:arXiv:2508.11166.