The growth of internet technologies has given rise to various social media platforms such as Twitter, Facebook, Instagram, and many more. People all over the world, irrespective of their age, have access to these platforms and can use them to upload their day to day events. Everyone has the power to voice their opinions on various items/posts of other people as well. Although this helps to bring people together, sometimes this can lead to issues within communities and groups. The vast majority of people tend to post ofensive/hate content on the internet. This has become a serious global concern. To solve this problem, a method needs to be introduced which can analyze huge amounts of data (text and images) over the internet and filter out the ofensive content. Our paper introduces a multimodal comparative analysis for detecting content that is, sarcastic, abusive, and vulgar. In addition to that, the sentiment of any content are also captured. The models are applied on four diferent datasets consisting of diferent languages: Hindi, Bodo, Gujarati, and Bengali. Experiments are performed on various models, of which VisualBert was found to give the best performance. This model is most efective for detecting hateful and ofensive content in multilingual datasets. Within the HASOC 2025-Meme task, our Team DeepSemantics ranked 11th, 14th, 11th, and 17th in Bodo, Gujarati, Hindi and Bangla datasets respectively. We achieved F1-scores of 0.5604 on Bodo, 0.4203 on Gujarati, 0.5498 on Hindi, and 0.4821 on Bangla datasets.
Social media platforms have become an integral part of modern communication, enabling users to share
content and express their opinions on a global scale. While these platforms were originally intended
to foster open dialogue and the exchange of ideas, they have also become channels for the spread of
negativity, hate speech and ofensive content [
Hate speech refers to derogatory expressions or terms directed at individuals or groups with the
intention to cause harm. Such expressions are typically based on characteristics such as ethnicity, gender,
religion, disability, sexual orientation, or nationality. Hate speech can have harmful consequences at
both individual and societal levels, fostering discrimination, hostility, and social division, and therefore
requires efective mechanisms for detection and removal [
Nowadays, memes have become a frequent mode of online communication, combining text and images to convey humorous or satirical messages. However, some memes may also contain hateful or ofensive content. The multimodal nature of memes makes them dificult to analyze, which often causes traditional text-based hate speech detection models to fail. Detecting harmful content in memes therefore requires approaches that consider both textual and visual modalities, posing a significant global challenge.
The Hate Speech and Ofensive Content Identification (HASOC) is a shared task series which is
organized anually as part of the Forum for Information Retrieval Evaluation (FIRE) since 2019 [
The 2025 edition of HASOC significantly extends previous years’ eforts by introducing multi-task, multilingual, and multi-dimensional classification challenges, reflecting the complex nature of harmful communication on social media platforms. Specifically, HASOC 2025 addresses the classification of social media posts (memes) across four diferent languages and for four distinct application domains: • Sentiment analysis: Identifying the polarity of online expressions to capture efective attitudes conveyed in user-generated content. • Sarcasm Detection: Detecting sarcastic expressions, which pose unique challenges for computational models due to their reliance on contextual cues, implicit meaning and irony. • Abuse Detection: Classifying abusive and derogatory language that targets individuals or groups, thereby aiming to protect users from direct harm and online harassment. • Vulgarity Identification: Identifying the use of vulgar, profane, or explicit language, whichalthough not always hateful contributes to ofensive and harmful communication.
HASOC encourages the development of generalizable and robust NLP models capable of handling diverse manifestations of ofensive content. By framing the task across multiple languages and content categories, HASOC 2025 provides a comprehensive benchmark to evaluate multilingual, cross-domain and multimodal approaches in ofensive content detection.
The datasets provided for HASOC 2025 are sourced from real-world social media platforms, ensuring both practical relevance and linguistic diversity. This shared task ofers researchers an opportunity to develop models that are not only accurate but also sensitive to cultural and linguistic nuances.
The remaining sections of the paper are structured as follows. In Section 2, a survey of techniques for classification tasks is provided. The dataset description is presented in Section 3. The proposed architecture has been discussed in Section 4. Results and discussions are presented in Section 5, and the conclusion is provided in Section 6
Due to the rise of multimedia content on the internet, multimodal (text and images) analysis has emerged as an important and critical area of research. The various classification tasks mentioned in previous section not only need textual data but also images for better understanding of contexts. For example, a comment may convey a positive sentiment but its image may represent a sarcastic tone. Also, abusive and vulgar content may rely on visual representations which might not be detected from text alone.
Hate speech and abusive content detection in Indic languages has been an active research area, with
multiple shared tasks organized under HASOC in recent years to support benchmark development and
multilingual evaluation [
Chanda et al. (2021) fine-tuned pre-trained transformer models across English, Hindi, Marathi and English-Hindi code-mixed data within the HASOC 2021 shared task framework, which illustrated their versatility in cross-lingual and mixed-language scenarios [11]. At SemEval-2020, the IRlab@IIT-BHU team applied an SVM classifier with TF-IDF features to detect ofensive language in multiple languages [12]. The “Crossing Borders: Multilingual Hate Speech Detection” paper examines sentence-level hate/ofensive detection in less-resourced languages such as Gujarati and Sinhala, emphasizing the challenges inherent in adapting models across diverse linguistic landscapes [13]. In HASOC 2022, IRLab@IITBHU submitted a model based on fine-tuning of existing models like XLM Roberta and German BERT for classification of tweets in Marathi, Hinglish Codemix and German language [ 14] . A deep learning approach to Hindi-English code-mixed hate speech detection explores sequential modeling architectures tailored to capture the nuances of mixed-language text in social media contexts [15]. In addition, Hate Content Identification in Code-Mixed Social Media Data by Chanda and Pal (2022) investigates Hindi-English code-mixed conversations, showing that deep learning architectures can efectively capture linguistic variation and identify hate content in noisy real-world contexts [16]. Collectively, these studies demonstrate that transformer-based models, particularly when adapted for multilingual and code-mixed data, ofer notable advantages in accuracy and adaptability over traditional machine learning classifiers.
Two major approaches exist for multimodal classification tasks. One is by making use of convolutional neural networks (CNNs) [17] and the other is through transformer based models [18]. CNNs are capable of extracting high quality visual features and transformer architectures are good for joint reasoning on image-text pairs.
Traditional CNN models such as AlexNet [19], VGGNet [20] and GoogLeNet [21] had a common issue known as the vanishing/exploding gradient problem. Also, it was observed that the accuracy of such models would remain saturated or even drop if more layers were introduced. To solve these issues, the ResNet (Residual Network) architecture [22] was introduced which implemented skip connections. These would prove efective in training deep networks consisting of multiple layers. Instead of directly learning a mapping () from input x to output, ResNet learns a residual mapping () = () − . The final output is computed as = () + , where x is propagated through the network via an identity shortcut connection. Common ResNet variants include ResNet-18, ResNet-34 and ResNet-50. These difer primarily in depth and the type of residual blocks used.
A joint understanding of images and text is necessary in multimodal domain. A powerful transformer model called BERT (Bidirectional Encoder Representations from Transformers) [23] revolutionized the domain of Natural Language Processing. However, it was designed to handle only text data. VisualBERT [24] is a model that injects image features into the BERT’s transformer architecture along with the textual features. This enables reasoning over both text and image modalities.
The HASOC 2025 shared task comprises datasets in four diferent languages: Hindi, Bengali, Bodo and Gujarati. For each language, the dataset is designed to support four distinct classification tasks: Sentiment Analysis, Sarcasm Detection, Vulgarity Detection and Abusive Content Classification. This multi-task, multilingual setup provides a comprehensive benchmark for evaluating natural language processing models in low-resource and high-resource language contexts. The detailed statistics of each dataset is presented in Table 1. For the Sentiment and Vulgarity classification tasks, the labels are represented as -1 (Negative), 0 (Neutral), and 1 (Positive). In contrast, for the Sarcasm and Abuse detection tasks, a binary labeling scheme is adopted, where 0 denotes Non-Sarcastic/Non-Abusive and 1 denotes Sarcastic/Abusive.
As described in previous section, the datasets consist of images as well as the extracted text. In this paper we utilize ResNet and VisualBERT to perform multimodal classification across all languages.
In the multimodal architecture using ResNet, a pre-trained ResNet backbone (ResNet-18/34/50) is used for visual modality. This network processes images and outputs vector embeddings in the form of 512-dimensional feature vectors. These embeddings capture both low-level patterns and high-level semantic features from the image domain.
To extract the features of textual data, the OCR extracted texts are encoded using BERT. It converts the textual input into 768-dimensional embedding vectors. The resulting vector efectively represents the semantic information contained in the text, preserving both word-level and contextual dependencies.
The visual and textual embeddings are concatenated to form a unified 1280-dimensional multimodal embedding vectors. This representation is then passed into a classifier (Multilayer Perceptron). The classifier consists of a linear transformation, followed by a ReLU activation and dropout to prevent overfitting. The output is then passed through a final linear layer to produce logits corresponding to target classes. A softmax function is applied to the logits to generate the final class probabilities.
The entire model is trained end-to-end, enabling the joint optimization of both modalities and ensuring that the fused representation captures complementary visual and textual cues for robust classification. The proposed architecture using ResNet is shown in figure 1.
In the multimodal architecture utilizing VisualBERT, Faster R-CNN (Regions with Convolutional Neural Networks) is used as an object detector to capture semantically meaningful regions of interest (ROIs). This model generates a set of visual embeddings for an input image and each embedding is represented as a 2048-dimensional vector. The top 36 detected regions are used for computational eficiency and the output becomes a matrix of shape (36,2048).
For the textual embeddings, the extracted text from the images are encoded using a pretrained BERT model. This model maps tokenized sequences into 768-dimensional embedding vectors.
After obtaining the image and textual embeddings, a pretrained VisualBERT is utilized to align the 36 visual embeddings with the text embeddings in a 768-dimensional feature space. This is achieved Language Gujarati Bengali Bodo Hindi
Model ResNet-18 ResNet-34 ResNet-50 VisualBERT ResNet-18 ResNet-34 ResNet-50 VisualBERT ResNet-18 ResNet-34 ResNet-50 VisualBERT ResNet-18 ResNet-34 ResNet-50 VisualBERT
Sentiment Sarcasm by linearly projecting the visual features to match the BERT embedding size before fusion. Both modalities are then concatenated and processed by the transformer layers of VisualBERT, which learns the cross-modal interactions. The [CLS] token of the fused sequence is used as a compact multimodal representation.
The final part of the architecture constitutes a classifier. The pooled [CLS] representation is passed through a fully connected linear layer, producing logits over three target classes. A softmax layer is then applied to obtain the final label distribution. The entire model is fine-tuned end-to-end, enabling both the visual and textual encoders to adapt to the downstream task. The entire architecture is presented in Figure 2.
Since only one model can be deployed for HASOC 2025, a comparative study has been conducted between ResNet (18,34,50) and VisualBERT. The initial datasets released by the HASOC team for four languages (containing labels) was divided into training and test datasets for evaluation in the ratio 80:20. To maintain fairness, all models were trained with the same hyperparameters: 10 epochs and a learning rate of 2e-5. The results obtained for all tasks have been shown in table 2. • Gujarati: VisualBERT outperforms all ResNet models in sarcasm and abuse detection. Although ResNet-18 achieves slightly higher accuracy in sentiment and vulgarity detection, VisualBERT maintains competitive performance while excelling in more context-dependent tasks. • Bengali: VisualBERT achieves the best sentiment and sarcasm detection, outperforming all ResNet variants. For vulgarity detection, VisualBERT ties with the best ResNet model. Abuse detection shows mixed results, but the gap is small. • Bodo: VisualBERT achieves the highest accuracy in abuse and vulgarity detection. Although ResNet-18 and ResNet-34 outperform VisualBERT in sentiment, they all give the same result for sarcasm. • Hindi: While ResNet models achieve slightly higher results in vulgarity and sentiment classification, VisualBERT clearly dominates in sarcasm and abuse detection.
Across all languages, VisualBERT consistently demonstrates superior performance compared to the ResNet variants (10 out of 16 tasks) , particularly in tasks that require deeper semantic understanding such as sarcasm and abuse detection. Its performance advantage over ResNet highlights the importance of transformer-based multimodal architectures, which can efectively capture nuanced relationships between textual and visual features in memes.
The variation in performance across languages for the same task is expected and can be attributed to multiple dataset and model-related factors. The datasets we have evaluated on are highly imbalanced, with Bodo and Gujarati being very small in size. This directly impacts model generalization due to lack of labeled data. Secondly, meme structure and language of the datasets vary significantly with diferent levels of code-mixing and script style. VisualBERT relies heavily on informative textual signals, (which are stronger in Bengali and Hindi) leading to higher scores. Also, all models were trained with the same hyperparameters to maintain fairness, without language specific tuning. While this ensures proper comparability, it may not fully optimize performance for every language dataset, resulting in imbalance of the scores.
Our run submission consists of the results obtained from the VisualBERT architecture. Table 3, Table 4, Table 5 and Table 6 shows the overall leaderboard results of our model compared to other performing teams of HASOC 2025 in Gujarati, Bodo, Bangla and Hindi datasets respectively.
In this paper, we investigated multimodal classification approaches for detecting sentiment, sarcasm, abuse, and vulgarity in online memes. Through extensive experiments on four diferent language datasets, our results demonstrate that transformer-based VisualBERT outperforms traditional convolutional models such as ResNet in efectively capturing multimodal representations.
This work provides a comparative analysis between two multimodal architectures, highlighting the strengths of transformer-based fusion in handling both textual and visual information. Beyond the image and textual feature extraction methods employed in this study, several alternative strategies for feature extraction and multimodal fusion remain unexplored. Future research may focus on incorporating advanced vision-language models, attention-based fusion strategies, or contrastive learning approaches to further enhance performance in multilingual meme classification tasks.
During the preparation of this work, the author(s) used ChatGPT, Grammarly in order to: Grammar and spelling check, Paraphrase and reword. After using this tool/service, the author(s) reviewed and edited the content as needed and take(s) full responsibility for the publication’s content. [8] M. Das, S. Banerjee, A. Mukherjee, Data bootstrapping approaches to improve low resource abusive language detection for indic languages, in: Proceedings of the 33rd ACM conference on hypertext and social media, 2022, pp. 32–42. [9] K. Ghosh, M. Das, M. Narzary, S. Saha, S. Barman, A. Mukherjee, S. Modha, D. Ganguly, U. Garain, S. Jaki, T. Mandl, Overview of the hasoc track at fire 2025: Abusive meme identification — shadows behind the laughter, in: K. Ghosh, T. Mandl, S. Pal (Eds.), Forum for Information Retrieval Evaluation (Working Notes) (FIRE 2025) December 17-20, Varanasi , India, CEUR-WS.org, 2025. [10] K. Ghosh, M. Das, S. Patel, N. Bhandary, A. Das, A. Mukherjee, S. Modha, D. Ganguly, U. Garain, S. Jaki, T. Mandl, Overview of the hasoc track at fire 2025: Abusive meme identification — shadows behind the laughter, in: FIRE ’25: Proceedings of the 17th Annual Meeting of the Forum for Information Retrieval Evaluation. December 17-20, Varanasi , India, Association for Computing Machinery (ACM), New York, NY, USA, 2025. [11] S. Chanda, S. Ujjwal, S. Das, S. Pal, Fine-tuning pre-trained transformer based model for hate speech and ofensive content identification in english indo-aryan and code-mixed (english-hindi) languages, in: Working Notes of FIRE 2021 - Forum for Information Retrieval Evaluation, Gandhinagar, India, December 13-17, 2021, volume 3159 of CEUR Workshop Proceedings, 2021, pp. 446–458.
URL: https://ceur-ws.org/Vol-3159/T1-44.pdf. [12] A. Saroj, S. Chanda, S. Pal, IRlab@IITV at SemEval-2020 task 12: Multilingual ofensive language identification in social media using SVM, in: Proceedings of the Fourteenth Workshop on Semantic Evaluation, 2020, pp. 2012–2016. URL: https://aclanthology.org/2020.semeval-1.265/. [13] S. Chanda, A. Dhaka, S. Pal, Crossing borders: Multilingual hate speech detection, in: FIRE (Working Notes), 2023, pp. 486–500. URL: https://ceur-ws.org/Vol-3681/T6-15.pdf. [14] S. Chanda, S. D. Sheth, S. Pal, Coarse and fine-grained conversational hate speech and ofensive content identification in code-mixed languages using fine-tuned multilingual embedding, in: FIRE (Working Notes), 2022, pp. 502–512. URL: https://ceur-ws.org/Vol-3395/T7-3.pdf. [15] S. Chanda, A. Dhaka, S. Pal, Towards safer online spaces: Deep learning for hate speech detection in code-mixed social media conversations, in: Companion Publication of the 16th ACM Web Science Conference, Association for Computing Machinery, 2024, pp. 103–109. doi:10.1145/ 3630744.3663610. [16] S. Chanda, S. Pal, Hate content identification in code-mixed social media data, in: Text and Social Media Analytics for Fake News and Hate Speech Detection, Chapman and Hall/CRC, 2024, pp. 225–247. doi:10.1201/9781003409519-13. [17] Y. Lecun, L. Bottou, Y. Bengio, P. Hafner, Gradient-based learning applied to document recognition,
Proceedings of the IEEE 86 (1998) 2278–2324. doi:10.1109/5.726791. [18] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, I. Polosukhin, Attention is all you need, Advances in neural information processing systems 30 (2017). URL: https://proceedings.neurips.cc/paper_files/paper/2017/file/ 3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf. [19] A. Krizhevsky, I. Sutskever, G. E. Hinton, Imagenet classification with deep convolutional neural networks, Advances in neural information processing systems 25 (2012). doi:https://doi.org/ 10.1145/3065386. [20] K. Simonyan, A. Zisserman, Very deep convolutional networks for large-scale image recognition, CoRR abs/1409.1556 (2014). doi:https://api.semanticscholar.org/CorpusID: 14124313. [21] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, A. Rabinovich, Going deeper with convolutions , in: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, pp. 1–9. doi:https://doi.ieeecomputersociety.org/10.1109/ CVPR.2015.7298594. [22] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015) 770–778. URL: https: //api.semanticscholar.org/CorpusID:206594692. [23] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, in: North American Chapter of the Association for Computational Linguistics, 2019. URL: https://api.semanticscholar.org/CorpusID:52967399. [24] L. H. Li, M. Yatskar, D. Yin, C.-J. Hsieh, K.-W. Chang, Visualbert: A simple and performant baseline for vision and language, ArXiv abs/1908.03557 (2019). URL: https://api.semanticscholar. org/CorpusID:199528533.