Sexism persists as a pervasive issue in society, particularly evident on social media platforms. This phenomenon encompasses a spectrum of expressions, ranging from subtle biases to explicit misogyny, posing unique challenges for detection and analysis. While previous research has predominantly focused on textual analysis, the dynamic nature of some social networks demands a more comprehensive approach. Multimodal analysis surpasses text-only methods, particularly in understanding sexism. Our results demonstrate that adding BLIP-generated image captions to OCR text raises F1-Macro from 0.6367 to 0.7298 (+9.3 points), and further including a GPT-4o description boosts it to 0.8114 (+8.2 points). The ViT+RoBERTa fusion model achieves the best overall performance (F1-Macro = 0.8308, +19.4 points over text-only), confirming that joint visual-text representations substantially enhance sexism detection. A similar pattern holds under soft-label training and across downstream tasks of intent classification and category categorization. These findings underscore the value of multimodal integration for robust, real-world sexism identification on social media.
Hate Speech (HS) is generally described as any form of communication that belittles a person or a group
based on attributes such as race, ethnicity, gender, sexual orientation, nationality, religion, among others
[
The field of NLP has increasingly focused on detecting hate speech and sexism, driven by their growing societal impacts, especially on social platforms. Notable eforts include SemEval-2019 Task 5, which targeted hate speech against immigrants and women [4], and SemEval-2023 Task 10, which developed a hierarchical taxonomy of sexist content and a dataset of 20,000 social media comments to enhance detection explainability [5]
Since 2021, the EXIST task addresses the problem of sexism identification in social networks [ 6, 7, 8]. Recent advancements in multimodal analysis have significantly enhanced the detection of hate speech in memes and images. The Multimodal Hate Speech Event Detection task organized in 2023 explored binary and target-specific detection strategies in text-embedded images, demonstrating the efectiveness of multimodal approaches in identifying hate speech [9]. A novel method introduced in 2023 utilizes pre-trained vision-language models (PVLMs) for hateful meme detection [10]. Additionally, a study conducted in 2018 demonstrated the superiority of a multimodal approach over unimodal methods in detecting sexist content in advertisements [11]. The Multimedia Automatic Misogyny Identification (MAMI) task at SemEval-2022 focused on identifying misogynous content in memes [12]. In [13], the authors investigated both unimodal and multimodal approaches for recognizing misogynous memes and proposed a bias estimation and mitigation strategy—based on Bayesian Optimization—that corrected model predictions toward the true class in up to 61.43% of cases.
3.1. Tasks Following EXIST [14], our aim is to address sexism identification in the following three tasks: 1. Sexism Detection. Determine if the meme contain sexist content. This is a binary classification: • Not Sexist. Memes not focusing on gender-related themes.
• Sexist. Memes discussing or portraying gender-related stereotypes or issues. 2. Source Intention Classification. Categorize sexist memes based on the creation intent: • Judgmental Sexist. Memes sharing experiences of encountering sexism.
• Direct Sexist. Memes explicitly promoting sexist beliefs. 3. Sexism Categorization. Classify sexist memes by the aspect of sexism they exhibit: • Ideological and Inequality. Memes undermining women’s rights or contributions. • Role Stereotyping and Dominance. Memes perpetuating gender role stereotypes. • Objectification. Memes portraying women solely as objects of desire. • Sexual Violence. Memes containing or promoting sexual harassment or assault. • Misogyny and Non-sexual Violence. Memes expressing hostility or violence towards women.
In this work, we partitioned the dataset into a 90% training split and a 10% test split. All models are trained on the training split and evaluated on the held-out test split.
To systematically evaluate how diferent levels of multimodal information afect sexism detection in memes, we experiment with four progressively richer model variants. We start with a strong text-only baseline built on RoBERTa and OCR-extracted meme text, then incrementally add visual cues: first by appending BLIP-generated captions to the text, next by incorporating a GPT-4o–derived high-level description, and finally by fusing raw image embeddings from ViT with text embeddings from RoBERTa. Below we describe each variant in turn, along with the data augmentations and training details designed to improve robustness and capture complementary visual–textual signals.
• Text-only baseline: RoBERTa [15] fine-tuned on OCR-detected meme text. • Text + BLIP [16]: OCR text concatenated with BLIP-generated image captions. • Text + BLIP + GPT Description: additionally include a GPT-4o explanation. Inputs: OCR text, BLIP captions, GPT-4o description.
GPT-4o Prompt Describe the meme, identify whether it addresses sexist topics, and explain the intent (humor, critique, normalization, etc.). Justify your analysis. • ViT + RoBERTa [17]: Fuse ViT image embeddings with RoBERTa text embeddings. For this variant, the following data augmentations were applied: – Images: ∗ Random horizontal and vertical flips. ∗ Random rotations up to 30°. ∗ Random perspective distortions.
∗ Random adjustments of brightness, contrast, saturation, and hue. – Text:
∗ Random token masking with a 10% probability per token.
These augmentations aim to improve model robustness by exposing it to various visual perturbations and textual variations during training. 4.2. Labeling Schemes • Task 1 (Detection).
uncertainty).
6 – Soft labels: set = – Hard labels: set = 1 if at least three annotators marked the meme sexist, otherwise = 0. #sexist votes
, e.g. 3/6 = 0.5. (Used only in Task 1 to capture annotator • Task 2 (Intention). Models struggled when using soft labels, so we train only on those memes with hard label = 1. • Task 3 (Category Union). For each category , merge the six annotator labels by =
max =1,...,6 ,, i.e. category is positive if any annotator flagged it.
In Table 2 we observe a clear progression from the text-only OCR baseline (F1-Macro = 0.6367) through multimodal enhancements. Adding BLIP captions raises F1-Macro by +0.0931 (to 0.7298), and the GPT-4o description contributes another +0.0816 (to 0.8114). The ViT+RoBERTa fusion achieves the highest F1-Macro of 0.8308, representing a +0.1941 gain over the baseline and +0.0194 over Text+BLIP+GPT. Across languages, English posts consistently outperform Spanish (e.g. ViT+RoBERTa: EN 0.8657 vs. ES 0.7897), indicating stronger model alignment with English meme content.
5.1. Task 1 5.1.1. Hard Labels 5.1.2. Soft Labels When we switch to soft labels (Table 3), the OCR baseline improves slightly by +1.00 point to 0.6466; Text+BLIP gains +5.78 points over that soft baseline (to 0.7044); Text+BLIP+GPT adds +9.49 points (to 0.7993); and ViT+RoBERTa attains +11.07 points (to 0.7573) relative to the soft baseline. Comparing hard vs. soft labels for ViT+RoBERTa, F1-Macro drops from 0.8308 to 0.7573 (–7.35 points). This shows that soft labeling takes into account diverse annotator perspectives, but comes at the cost of some discriminative strength in our top multimodal model. Table 5 confirms that multi-label categorization benefits from multimodal fusion. The baseline starts at F1-Macro = 0.6597; Text+BLIP adds +0.0350 (to 0.6947); Text+BLIP+GPT adds +0.0783 (to 0.7380); 5.2. Task 2 5.3. Task 3 slightly outperform English here (0.7670 vs. 0.7562 for ViT+RoBERTa), demonstrating robust crosslingual generalization in category detection. Within the ViT+RoBERTa model, the Stereotyping category is predicted best (F1 = 0.8517), followed by Objectification (0.8072) and Ideological-Inequality (0.8023), whereas Sexual Violence (0.6437) and Misogyny (0.6763) remain the most challenging categories. Task 3 (Category Union) — Complete results by model and language
Participation in the EXIST 2025 shared task was registered under the team name ArcosGPT. According to the oficial task overview [ 18], third place was achieved in the hard–hard evaluations of all three subtasks, using the full multimodal fusion model (ViT+RoBERTa) in each case. Tables 6–8 present the performance alongside the top five systems in each hard–hard evaluation.
This study demonstrates the efectiveness of multimodal approaches combining textual and visual information. By integrating OCR-extracted text, BLIP-generated captions, GPT-4o contextual descriptions, and visual embeddings from ViT models, we achieved significant performance improvements across all tasks, particularly in fine-grained classification scenarios. The ViT+RoBERTa fusion model achieved the highest performance, validating the strength of joint visual-textual representations.
Despite these advancements, several avenues remain open for further research. First, dataset expansion is critical to improve model generalizability, especially in capturing underrepresented forms of sexism and ensuring cultural diversity.
We also plan to investigate new multimodal features, including sociolinguistic cues, and user interaction patterns such as comments and reactions, to enhance contextual understanding. In addition, an analysis of content difusion may help reveal the mechanisms of propagation and ofer insights for efective mitigation strategies. Finally, adopting advanced model architectures—such as cross-modal attention mechanisms—could further enhance the synergy between modalities and elevate performance across all subtasks.
This work was done in the framework of Malicious Actors Profiling and Detection in Online Social Networks Through Artificial Intelligence (MARTINI) research project, funded by MCIN/AEI/10.13039/501100011033 and by NextGenerationEU/PRTR (Grant PCI2022-135008-2).
During the preparation of this work, the authors used OpenAI’s ChatGPT and GPT-4o models in order to: (i) generate multimodal descriptions of memes used as additional model input features, (ii) assist with grammar and spelling checking, and (iii) rephrase and improve the clarity of some sentences.
After using these tools, the authors carefully reviewed and edited the content as needed and take full responsibility for the final text. No generative AI tool was used for developing the scientific insights, analysis, or conclusions of this paper. [4] V. Basile, C. Bosco, E. Fersini, D. Nozza, V. Patti, F. Rangel Pardo, P. Rosso, M. Sanguinetti, SemEval2019 Task 5: Multilingual Detection of Hate Speech Against Immigrants and Women in Twitter, 2019, pp. 54–63. [5] H. Kirk, W. Yin, B. Vidgen, P. Röttger, SemEval-2023 task 10: Explainable detection of online sexism, in: A. K. Ojha, A. S. Doğruöz, G. Da San Martino, H. Tayyar Madabushi, R. Kumar, E. Sartori (Eds.), Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023), Association for Computational Linguistics, Toronto, Canada, 2023, pp. 2193–2210. [6] F. Rodríguez-Sánchez, et al., Overview of EXIST 2021: Sexism identification in social networks,
Procesamiento del Lenguaje Natural 67 (2021) 195–207. [7] F. Rodríguez-Sánchez, et al., Overview of EXIST 2022: Sexism identification in social networks,
Procesamiento del Lenguaje Natural 69 (2022) 229–240. [8] L. Plaza, J. Carrillo-de Albornoz, R. Morante, E. Amigó, J. Gonzalo, D. Spina, P. Rosso, Overview of exist 2023 – learning with disagreement for sexism identification and characterization, in: Experimental IR Meets Multilinguality, Multimodality, and Interaction, Springer Nature Switzerland, Cham, 2023, pp. 316–342. [9] S. Thapa, F. A. Jafri, A. Hürriyetoğlu, F. Vargas, R. K. W. Lee, U. Naseem, Multimodal Hate Speech Event Detection - shared task 4, case 2023, in: Proceedings of the 6th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text, Incoma Ltd, 2023, pp. 151–159. [10] R. Cao, M. S. Hee, A. Kuek, W.-H. Chong, R. K.-W. Lee, J. Jiang, Pro-Cap: Leveraging a Frozen Vision-Language Model for Hateful Meme Detection, in: Proceedings of the 31st ACM International Conference on Multimedia (MM ’23), 2023, pp. 5244–5252. doi:10.1145/3581783.3612498. [11] F. Gasparini, I. Erba, E. Fersini, S. Corchs, Multimodal Classification of Sexist Advertisements, in: Proceedings of the 15th International Joint Conference on e-Business and Telecommunications (ICETE), 2018, pp. 565–572. doi:10.5220/0006859405650572. [12] E. Fersini, F. Gasparini, G. Rizzi, A. Saibene, B. Chulvi, P. Rosso, A. Lees, J. Sorensen, Semeval-2022 task 5: Multimedia automatic misogyny identification, in: Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), Association for Computational Linguistics, 2022, pp. 533–549. [13] G. Rizzi, F. Gasparini, A. Saibene, P. Rosso, E. Fersini, Recognizing misogynous memes: Biased models and tricky archetypes, Information Processing Management 60 (2023) 103474. doi:10. 1016/j.ipm.2023.103474. [14] L. Plaza, J. Carrillo-de Albornoz, I. Arcos, P. Rosso, D. Spina, E. Amigó, J. Gonzalo, R. Morante, EXIST 2025: Sexism Identification in Social Networks, http://nlp.uned.es/exist2025/, 2025. Accessed: 2025-06-05. [15] Facebook AI, FacebookAI/xlm-roberta-large, https://huggingface.co/FacebookAI/ xlm-roberta-large, 2021. Accessed: 2025-06-05. [16] Salesforce AI Research, salesforce/blip-image-captioning-base, https://huggingface.co/salesforce/ blip-image-captioning-base, 2022. Accessed: 2025-06-05. [17] Google Research, google/vit-base-patch16-224, https://huggingface.co/google/ vit-base-patch16-224, 2022. Accessed: 2025-06-05. [18] L. Plaza, J. Carrillo-de Albornoz, I. Arcos, P. Rosso, D. Spina, E. Amigó, J. Gonzalo, R. Morante, Overview of EXIST 2025: Learning with Disagreement for Sexism Identification and Characterization in Tweets, Memes, and TikTok Videos (Extended Overview), in: G. Faggioli, N. Ferro, P. Rosso, D. Spina (Eds.), Working Notes of CLEF 2025 – Conference and Labs of the Evaluation Forum, CEUR Workshop Proceedings, 2025.