This paper presents the system submitted by the team from IIT Dhanbad in the FIRE IRSE HASOC-2025 shared task on automatically identifying hateful and ofensive memes in four Indian languages-Gujarati, Hindi, Bengali, and Bodo. Here, we developed a multimodal deep learning framework for five subtasks of Sentiment, Sarcasm, Vulgarity, Abuse, and Target Communities to automatically predict the hateful and ofensive memes. we have used a task-specific strategy in which a diferent multimodal model handles each label. In addition to methods like gated attention and weighted losses, we employed a variety of text encoders (XLM-R, mBERT, MuRIL, BanglaBERT) and image encoders (EficientNet, DenseNet, VGG19, ResNet). Extensive experiments show that task-specific pipelines consistently provide better performance when combined using ensemble methods. These results demonstrate the value of modular multimodal models in addressing the complex, varied, and frequently implicit character of hate in memes. In the oficial assessment, the final leader board's best macro F1-scores on the Kaggle platform for Hindi, Gujarati, Bangla, and Bodo were 0.5741 (6th rank), 0.5887 (8th rank), 0.5720 (8th rank), and 0.5818 (7th rank), respectively.
A meme is a joke, cultural concept, or trend that is conveyed through text, images, or videos and quickly
gains traction on social media. Over the past ten years, social media’s influence has fundamentally
changed how people express themselves, share their thoughts, and participate in public discussions.
Memes have emerged as one of the most widely used and influential online communication tools. They
combine images with just enough text to tell a story or make a joke; they are brief, visually appealing,
and frequently humorous. But more often than not, the same format is also being twisted into a platform
for hate speech, sarcasm, targeted abuse, and vulgarity[
If platforms are to safeguard users and stop the spread of harmful content, they must be able to
lfag hateful memes with speed and accuracy. However, the task is much more dificult than it seems.
Because they combine text and images to convey meaning, memes are by nature multimodal. Optical
Character Recognition (OCR) is frequently used to extract text from images; however, in practice, this
results in outputs that are noisy, incomplete, or distorted because of stylized fonts or poor image quality
[
CEUR Workshop
ISSN1613-0073
Previous studies have demonstrated the efectiveness of Transformer-based models for handling text
[
In this study, we investigate three methods for classifying memes in five diferent subtasks: target
communities, sarcasm, vulgarity, abuse, and sentiment. In the first, shared encoders try to learn
everything together in a unified multimodal pipeline. The second, a hierarchical transformer fusion
model (HIT-FAME), presents attention-based mechanisms that facilitate better alignment of text and
image features [
This paper makes two contributions. First, we demonstrate that ensemble strategies in conjunction
with task-specific modular pipelines consistently outperform unified models, especially when dealing
with multilingual meme languages and noisy OCR text. In order to shed light on the ways preprocessing
decisions, encoder selection, and fusion strategies afect performance, we secondly present a benchmark
framework for multimodal hate speech detection in Indic settings [
Three interconnected insights from recent studies on multimodal hateful, abusive, and subjective
content—particularly memes that combine images with text overlayed or associated with them—directly
inform the HASOC 2025 challenge[
Overall, previous work ofers both useful toolkits (OCR pipelines, multimodal transformers, CLIP-style
visual embeddings, and instruction-tuned multimodal LLMs) and warnings (cultural bias, unimodal
shortcuts, and annotation subjectivity). The literature recommends that for HASOC 2025 [
This section provides a thorough explanation of the suggested systems created for predicting hate speech detection experiments that have been conducted across the various Indo-Aryan languages, including Bengali, Hindi, Gujarati, and Bodo, to solve the four distinct tasks. The proposed framework has been shown in the figure 3. We have given more detail about our methodology, including word embeddings and neural transformer architecture training, in the upcoming section.
FIRE HASOC-2025 organizers provided the datasets1, which were primarily two training and testing datasets with images (JPEG) of memes gathered as depicted in Figure 1, arranged by language, and a.csv file with the image id. As illustrated in Figure 2, represents the overall true labels with noisy and uncleaned OCR data extracted from the images they provided, including languages such as Bengali, Hindi, Bodo, and Gujarati. 1https://hasocfire.github.io/hasoc/2025/dataset.html
There were emojis, profanities, informal structures, and noise in the meme OCR text. The following procedures were used to apply a strong cleaning function consistently across all languages: • Emoji Replacement: To translate commonly used emojis into textual representations, a carefully chosen emoji-to-word mapping was used. • Lower casing and Numeric Normalization: To maintain semantic intent, all English characters were lowercased, and a <num> placeholder was used in place of numerical values. • Removal of Unwanted Tokens: Regex patterns were used to eliminate URLs, hashtags, mentions, and mixed-script artifacts. • Mixed-Script Filtering:To get rid of OCR-related errors, words with both Latin and Indic script characters were removed. • Stop Word Removal: Both native and Romanized tokens were included in the unique list of stop words used by each language. • Profanity Masking: For every language (Roman + native script), a longer list of ofensive terms was kept up to date, and matches were concealed by using the <PROFANITY> token2. The list of ofensive words for each language, which is prepared for profane words: Bengali= ’chod’, ’chodon’, ’chudi’, ’gandu’, ’bal’, ’randi’, ’khanki’, ’madarchod’, ’banchod’, ’bokachoda’, ’shala’, ’shali’, ’haramzada’ Hindi = ’bhenchod’, ’madarchod’, ’gaandu’, ’gandu’, ’chutiya’, ’chutiye’, ’bhosdike’, ’bhosdiwale’, ’bsdk’, ’mc’, ’bc’, ’randi’, ’kutta’, ’kamine’, ’kamina’, ’harami’, ’haramkhor’, ’lund’, ’loda’, ’lauda’, ’jhaant’, ’gote’, ’tatte’, ’choot’, ’behenchod’, ’maaki’, ’gaand’, ’bhadwa’, ’lavde’ Gujarati = ’Chodhru’, ’chodkanya’, ’Chodu’, ’Chodu Bhagat’, ’Choidi Rand’, ’Choot Marina’, ’Gaand Ma Ghal’, ’gaandina’, ’gandina’, ’gand ma ghali de’, ’Gand Maraav’, ’Gand nu kaanu’, ’gandi chodina’, ’Gandi Gaan’, ’gando’, ’Gando salo’, ’Ghel Sappo’, ’Ghelchoydi’, ’Halki Raand na’, ’hopa’, ’Jhaanth’, ’Jhadhino!’, ’Lukkha Loda’, ’luli’, ’Lund Bhagat’, ’mota jijah’, ’Moti Gaand’, ’nago choido’, ’Nakhodiya’, ’taari gaand ma dando ghalu’ Bodo = ’abu gidir’, ’Bazari’, ’Bima khoigra’, ’Bima khoygra’, ’Bimani fishai’, ’Bwitali’, ’bwithala’, ’Cfa swlangra’, ’Hinjao maogra (khoigra)’, ’khilama’, ’Khoynai’, ’Lwdwi’, ’Nwma shifa’, ’Nwmani abu’, ’Sifa gudung’, ’Sifa jagra’, ’sifa jagra’, ’Sifa swlagra’, ’Sikwmwn’, ’swima ni pisa’, ’tapa’ • Whitespace and Punctuation Normalization: Excessive punctuation was eliminated, and one space was used in place of several. This procedure made sure that every sample had clear, comprehensible text devoid of harmful or unnecessary tokens.
Particularly for Indic memes, where the textual component may be embedded in noisy, stylized, or codemixed OCR outputs, textual comprehension is an essential part of multimodal hate speech detection. We experimented with and used both pretrained Transformer-based encoders and a custom static embedding approach (HIT-BPE) to eficiently process such content across multiple languages. Taskwise performance was used to select or combine each encoder. Before being fused with the visual modality, all encoders convert the final cleaned and pre-processed text into a fixed-length vector representation. mbert, XLM, and MuRIL have all been employed as text encoders.
We employed deep convolutional neural networks (CNNs) for image encoding in order to supplement the textual stream. These use visual cues such as facial expressions, symbols, gestures, or embedded text to extract high-level semantic and spatial features from meme images that may suggest hate, sarcasm, 2Warning: This paper contains certain ofensive or potentially upsetting content, which is included only due to the nature of the work and could not be avoided
xlm-roberta-base mBERT Google MuRIL BanglaBERT 768 768 768 768 vulgar, sarcasm sentiment, target comm. vulgar, abuse abuse, sentiment
Robust multilingual, cross-lingual
Transferable & widely used, multilingual Indian language focused,
eficient NLU Bengali optimized, high accuracy sentiment, or abusive context. A resized and normalized RGB image is converted by each image encoder into a fixed dimensional feature vector, usually 512, which is then projected and fused with the textual representation in the multimodal fusion module. These models have been used, including DenseNet-161, ResNet-152, EficientNet-B7, and VGG19-BN (Batch Normalized VGG19).
VGG19-BN DenseNet-161 ResNet-152 EficientNet-B7
All five classification tasks can be powered by our fusion strategy, which combines the two modality streams—text and image—into a single joint representation. There are three stages to it: 1. Modality Concatenation • A single 1024-dimensional vector is created by concatenating the 512-dimensional text and 512-dimensional image embeddings. • By preserving all of the data from both encoders, this raw concatenation enables the model to focus on the features that are most predictive for each sample. 2. Gated Fusion • A sigmoid activation is performed after a single linear transformation to compute a learnable ”gate” vector of the same size (1024). • The concatenated features are multiplied element-wise by this gate, which functions similarly to a soft mask. • In practice, the gate learns to up- or down-weight particular joint features, such as highlighting image cues when the image itself is ofensive or text cues when the meme’s humor is text-centric. 3. Joint MLP and Task Heads • A three-layer MLP (1024→512→256→128) with ReLU activations and 30% dropout at each layer is used to process the gated 1024-dim vector. • Higher-order interactions between the two modalities are learned by this MLP. • The 128-dim representation is then projected to the task’s label space by a specialized linear head for each of the five tasks (2 classes for Abuse/Vulgarity/Sarcasm, 3 classes for Sentiment, and 8 classes for Target communities).
• The multi-class heads use softmax, while the binary heads use sigmoid activations at inference.
We present the results of the FIRE HASOC-2025 task, which entails evaluating multimodal data (text and images) in order to identify targeted communities, detect abuse, evaluate sarcasm and vulgarity, and assign sentiment labels.
The FIRE HASOC-2025 organizers initially provided training datasets containing meme images (JPEG) and corresponding CSV files with image IDs. These four individual datasets covered the languages—Bengali, Hindi, Bodo, and Gujarati—and included noisy, uncleaned OCR text extracted from the images. We split the training data into 80:20 ratios for training and testing to evaluate the best-performing model for each language across the diferent tasks described in the Table 3 4.2. Tasks
In Bengali (BanglaBERT+VGG19, F1 = 0.7938) and Gujarati (Muril+VGG19, F1 = 0.7973) perform the best, as indicated in Tables 6 and 4. According to tables 5 and 7, Hindi and Bodo perform somewhat worse (about 0.74–0.76 F1). The efectiveness of the pretrained models (BanglaBERT for Bengali and Muril for Gujarati/Hindi) in conjunction with CNN backbones emphasizes the significance of language-adapted models for abuse detection.
In Bengali has the highest score (XLM-R + DenseNet161, F1 = 0.8496), as indicated in tables 6 and 4. Gujarati comes in second (0.8416), as indicated in tables 4. As indicated in tables 5 and 7, respectively, Hindi and Bodo continue to fall within the mid-0.73–0.76 range. Here, as illustrated in figure 4, we discovered that the robust cross-lingual models, such as XLM-R, when combined with deeper CNNs (DenseNet, VGG19), perform exceptionally well at vulgar detection, indicating generalizability across languages.
As indicated in Tables 6 and 7, respectively, Bengali once again demonstrates promising results with BanglaBERT+VGG19 (F1 = 0.8229), while Gujarati (0.6793) and Hindi (0.6801) trail behind. When available, language-specific models perform better than multilingual ones, but Sarcasm is still a dificult task in low-resource environments.
The F1-scores for all languages are generally lower, ranging from 0.41 for Bodo to 0.67 for Bengali, as indicated in 7 and 6, respectively. Bengali benefits from BanglaBERT (0.6720), while Gujarati and Hindi both struggle ( 0.49–0.64). Here, we discovered that multimodality and subtle cues make sentiment detection in memes extremely dificult; results are marginally improved by language-specific embeddings.
In this task, the extremely low F1-scores (0.23–0.42), this is the most dificult task in all languages. Bengali (mBERT+EficientNetB7, F1 = 0.4215) was the best result, as indicated in table 6, and Hindi (0.2319) performs the worst among the other languages. Here, we found that the lack of data, implicit hate speech, and the current models’ poor contextual awareness make it dificult to identify target communities.
The performance of various models on a range of classification tasks is shown in the figure 4. Here, the BanglaBERT + VGG19 and XLM-R + VGG19 models obtain the highest F1 scores, especially for the Abuse and Sarcasm tasks. On the other hand, several models display F1 scores below 0.5, suggesting that tasks such as Target Comm. are extremely dificult.
A direct comparison of the F1 scores for each task across the languages is shown in Figure 5. With the highest F1 score for the Sarcasm and Sentiment tasks, this figure demonstrates Bengali’s consistently strong performance. The Sentiment and Target Comm. tasks have the lowest F1 scores for the Bodo language, despite its strong performance on other tasks.
Figure 5 and 6 illustrates the diferences in performance for each language on various tasks. For the Abuse and Vulgar tasks, all four languages—Gujarati, Hindi, Bengali, and Bodo—display high F1 scores. However, F1 scores for the Sentiment and Target Comm. tasks are significantly lower than those for the other languages, indicating a significant decline in performance for all languages.
Here, we have used diferent transformer models because each subtask required distinct linguistic understanding. For example, XLM-R was used for Vulgarity and Sarcasm due to its strong cross-lingual and contextual capabilities. mBERT was used for Sentiment and Target Community tasks as it handles general multilingual contexts well. This task-specific approach improved accuracy since no single model worked best for all subtasks. In short, the Selection of transformer model and their combination was solely done based on their performance on various specific tasks (Vulgar, Sarcasm, Sentiment, Abuse) and thereby used to boost the overall performance and prediction of the Memes
When language-specific models (BanglaBERT ([ 29]), Muril ([
In this paper, we presented a multimodal deep learning framework for five subtasks of Sentiment, Sarcasm, Vulgarity, Abuse, and Target Communities to automatically predict the hateful and ofensive memes across the Indo-Aryan languages such as Hindi, Gujarati, Bengali, and Bodo. This work addressed the complex problem of detecting hateful and ofensive content in multimodal memes across four Indian languages by leveraging a range of text and image encoders, preprocessing strategies, and multimodal fusion techniques. Our findings show that task-specific modular pipelines, when combined with ensemble strategies, consistently outperform unified multimodal models, particularly in handling noisy OCR text, multilingual variation, and implicit signals of hate. The results highlight that no single model is universally efective across all subtasks, and instead, modularity and flexibility are crucial for robust meme classification. Importantly, our framework not only sets a benchmark for multimodal hate speech detection in Indic contexts but also underscores the need for richer, balanced datasets and context-aware architectures that can capture the subtle interplay of humor, sarcasm, and coded language.
In the future, we will explore integrating external knowledge sources, advanced cross-lingual embeddings, and continual learning approaches to further improve performance in low-resource settings and ensure safer digital environments.
The authors acknowledge the Science and Engineering Research Board (SERB), Department of Science and Technology (DST), Government of India, for supporting the hardware used in this research under the Start-up Research Grant (SRG) scheme.
No generative AI tools were used in the preparation of this paper, including in writing, data analysis, ifgure generation, or any other part of the research process, in accordance with the CEUR Policy. Mukherjee and Sandip Modha and Debasis Ganguly and Utpal Garain and Sylvia Jaki and Thomas Mandl, Overview of the HASOC Track at FIRE 2025: Abusive Meme Identification — Shadows Behind the Laughter, in: FIRE 25: Proceedings of the 17th Annual Meeting of the Forum for Information Retrieval Evaluation. December 17-20, Varanasi, India, Association for Computing Machinery (ACM), New York, NY, USA, 2025. [21] A. Kumar, et al., Semeval-2020 task 8: Memotion analysis, in: Proceedings of the 14th International Workshop on Semantic Evaluation (SemEval-2020), 2020. URL: https://aclanthology.org/2020. semeval-1.99.pdf. [22] S. Mishra, S. Suryavardan, P. Patwa, M. Chakraborty, et al., Memotion 3: Dataset on sentiment and emotion analysis of code-mixed hinglish memes, arXiv preprint arXiv:2309.06517 (2023). URL: https://arxiv.org/abs/2309.06517. [23] M. Das, S. Banerjee, A. Mukherjee, Data bootstrapping approaches to improve low resource abusive language detection for indic languages, in: Proceedings of the 33rd ACM conference on hypertext and social media, 2022, pp. 32–42. [24] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, I. Sutskever, Learning transferable visual models from natural language supervision, arXiv preprint arXiv:2103.00020 (2021). URL: https://arxiv.org/abs/2103.00020. [25] J. Li, D. Li, S. Savarese, S. Hoi, Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, Proceedings of Machine Learning Research (2023).
URL: https://arxiv.org/abs/2301.12597. [26] J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, et al., Flamingo: a visual language model for few-shot learning, in: NeurIPS, 2022. URL: https://arxiv.org/abs/2204.14198. [27] T. Mandl, S. Modha, P. Majumder, D. Patel, M. Dave, C. Mandlia, A. Patel, Overview of the hasoc track at fire 2019: Hate speech and ofensive content identification in indo-european languages, in: Proceedings of the 11th Annual Meeting of the Forum for Information Retrieval Evaluation, FIRE ’19, Association for Computing Machinery, New York, NY, USA, 2019, p. 14–17.
URL: https://doi.org/10.1145/3368567.3368584. doi:10.1145/3368567.3368584. [28] b. Koyel Ghosh and Mithun Das and Mwnthai Narzary and Saptarshi Saha and Shubhankar Barman and Animesh Mukherjee and Sandip Modha and Debasis Ganguly and Utpal Garain and Sylvia Jaki and Thomas Mandl, Overview of the HASOC Track at FIRE 2025: Abusive Meme Identification — Shadows Behind the Laughter, CEUR-WS.org, 2025. [29] A. Bhattacharjee, M. T. Rahman, M. S. I. Azam, S. Rahman, S. Joty, M. S. F. Islamb, M. T. Karim, Banglabert: Language model pretraining and benchmarks for low-resource language understanding, in: Findings of the Association for Computational Linguistics: ACL 2022, 2022, pp. 1054–1069.