Content moderation at scale warrants automated systems that are capable of understanding nuance from the text and images being posted online. Transformer-based models have been shown to perform with these preconditions in mind, but the additional complexities originating from multimodality and multilinguality mandate the need for better tuned systems that can capture more enriched representations of the context. This can essentially translate to downstream tasks such as sexism identification in online content, which is the forefront of the EXIST 2024 shared tasks. This paper, as part of the EXIST challenge at CLEF 2024, investigates an attention-based approach to improve performance over baseline multimodal models by assigning separate importance to the textual and visual representations. The proposal is evaluated against CLIP and ViLT, two established multimodal models, while achieving state-of-the-art performance in multi-label classification tasks in the hard-hard evaluation context. The study is further augmented by the inclusion of diferent forms of ablations, involving confusion metrics for the applicable tasks.
The proliferation of internet users throughout the globe has caused an upsurge in the amount of content being generated and consumed on a daily basis. This huge volume of content includes a wide range of information, personal opinions and entertainment, which reflect the diverse perspective of its global audience. Naturally, given the volume, a large portion of the content on the internet enforces harmful and problematic behaviors [1]. One of the most concerning issues among them is sexism. Memes have recently entered the cultural zeitgeist where a piece of visual or audio-visual media may be accompanied by a humorous piece of text [2]. However, as with any form of media, memes too can be weaponized to difuse problematic views and ideologies. Consequently, a good portion of memes on the internet enforce discriminatory behavior, such as sexism. The influence of such content is multifaceted, ranging from creating further polarization and radicalization of impressionable people, to causing psychological discomfort to its victims.
In order to combat such problematic trends and create a safer online experience regardless of gender, it has become quite important for automated systems to screen and flag for potentially inimical content at scale. The simple rule-based approaches may work to a certain extent for filtering textual content, but these approaches tend to fail when multiple modalities, such as images along with text, are involved.
This is due to the fact that nuance and context cannot be inferred easily from a set of rules. To take this into account, transformer-based architectures have recently gained prevalence in hateful content detection due to their ability to better understand context.
The role of a comprehensive dataset is quintessential in order to tackle the challenge of training these transformer-based models for the purpose of employing them to downstream tasks. The Hateful Memes Challenge dataset [3] is a popular pick, containing over 10,000 multimodal samples of binary-labeled data. However, datasets that cover hateful behaviour at a broad scale fail to capture the specifics of sexism found in online content. Moreover, while not a low-resource language, there is a lack of Spanish datasets for sexism detection that are adequately labeled. The SemEval-2019 dataset [4] does address sexism in English and Spanish textual content specifically, but the binary labels lack resolution when it comes to understanding intent, and it does not address sexism found in images. Similarly, the Automatic Misogyny Identification (AMI) text dataset [ 5] ofers labeled data for sexist text classification with some granularity, but the classifications are exclusive, with each sample being assigned to a single category only. In order to address multiple dimensions at once, the EXIST 2024 [6, 7] dataset is the only multimodal dataset for both Spanish and English content, where the labels are assigned with varying levels of granularity, ranging from binary classification, to multi-class and even multi-label classification where multiple labels can co-exist for the same sample.
The multimodal nature of the EXIST 2024 memes dataset imposes a diferent set of challenges as opposed to its text-only counterpart. The multilinguality also adds to the complexity since most pretrained models tend to be trained on datasets that are predominantly in English. To address all these issues, we have proposed an attention-enhanced approach that uses both the textual and the visual context to get a more enriched representation of the sample and then use it for the downstream task of sexism classification, the architecture of which is discussed in finer detail within this paper. We have also evaluated our approach empirically over existing multimodal models, such as CLIP and ViLT, along with the error analysis that delineates the shortcomings of our system. As per the experiments, our approach yields superior performance in multi-class and multi-label classification problems when considering the hard-hard evaluation context (ICM-Hard), improving the performance by 7%-12% and 6%-13% respectively.
The EXIST 2024 memes dataset includes a total of 5,044 image-text pairs in English in Spanish. The training split contains 4,044 memes, each labeled by 6 diferent annotators for their respective tasks. The test split provides 1,053 unlabeled samples. Since the memes dataset does not provide a separate split for validation, an 80-20 split has been performed on the original training split, resulting in 3,235 samples for training and 809 samples for validation. A detailed language-wise breakdown of the dataset splits can be visualized in Table 1.
The dataset also provides additional features pertaining to the annotators, such as their genders, ages, ethnicities, education level, and countries.
The tasks for the memes dataset seek to classify sexism in memes to varying degrees. A more detailed overview of the hard label distribution of each of these tasks can be found in Table 2. The soft labels for each of the tasks assign a numeric value to each category in the Learning With Disagreement (LeWiDi) format. The soft values for tasks 4 and 5 add up to 1, as the labels are unique for each sample. For task 6, they do not necessarily add up to 1, as it is a multi-label problem.
• Task 4: Given an image and its caption, the subtask is to classify whether meme in question contains any references to sexism or not. This includes any form of sexist content, describing situations involving discrimination towards women, and even contexts where sexism is criticised.
It is essentially a binary classification problem. • Task 5: This subtask is a multi-class classification problem where the intention behind the meme needs to be identified. Given that a meme is sexist, this can either be classiefid as “DIRECT” or “JUDGEMENTAL”. If it is not sexist to begin with, then the identification should also indicate that. – DIRECT: The meme itself enforces sexist ideology without any ironic or satirical aspect to it.
– JUDGEMENTAL: The meme condemns sexist behavior, either directly or satirically. • Task 6: The image can be classified into multiple labels — “IDEOLOGICAL-INEQUALITY”, “STEREOTYPING-DOMINANCE”, “OBJECTIFICATION”, “SEXUAL-VIOLENCE” or “MISOGYNYNON-SEXUAL-VIOLENCE”. Multiple labels can coexist for a single image at the same time and similar to the previous tasks, there should be a separate label if the meme is not sexist. – IDEOLOGICAL-INEQUALITY: Ideological discrediting refers to all memes that discredit the feminist movement with the intention of devaluing, belittling and defaming the plight of women. On the other hand, “Inequality” refers to the memes that establish a narrative that no gender discrimination exists in the current society, or the flipped narrative where men are presented as the victims. – STEREOTYPING-DOMINANCE: Memes that impose the idea of specific roles being better suited for women fall under stereotyping. Dominance is characterized by the positioning of men above women in various standings. – OBJECTIFICATION: Memes dehumanizing or treating women as objects or commodities count under this label. These may also include the exertion of beauty or societal standards. – SEXUAL-VIOLENCE: These memes in question call for sexual suggestions, sexual favors or sexual abuse. – MISOGYNY-NON-SEXUAL-VIOLENCE: Expressions of physical violence and hatred towards women fall under this label.
In this section, we will provide a brief overview of the methodology employed to address the tasks at hand. Specifically, we were provided with memes sourced from the internet, structured in imagecaption pairs denoted as (, ). For solving the tasks, the model architecture we have used has five diferent components i) Pretrained Vision-Language Model, ii) Semantics from Pooled Representations, iii) Attention Enhanced Context Vector for each Modality, iv) Modality Fusion, and v) Classification Head
Our experiment uses a pre-trained ViLT model. Each input meme sample = (, ), comprising the image content and its caption , is processed individually. The text processor tokenizes the caption into its constituent tokens = 1, 2, . . . , . Meanwhile, the image processor divides the input image ∈ R× × into patches, which are then flattened to ∈ R× ( 2· ), where (, ) represents the patch resolution, and = 2 . Specifically, ViLT employs the BERTTokenizer as the text processor and the ViT (Vision Transformer) processor as the image processor.
The tokens and the image patches are then fed into the pre-trained ViLT model to enhance the comprehensive understanding of image-text pairs through multimodal fusion. From the model, we get image-aware text representations = {ℎ1 , ℎ2 , . . . , ℎ } and text-aware image representations = {ℎ1 , ℎ2 , . . . , ℎ } where ℎ is the last layer hidden representation for − th token and ℎ is last layer hidden representation for − th image-patch
The ViLT model also gives a pooled representation of the whole multimodal input. ViLT incorporates a special [CLS] token at the beginning of the multimodal input. We also extract the last layer representation of this token. A linear projection is applied to that representation to extract the pooled representation ℎ = · ℎ[CLS] +
Each token and patch holds its unique representation, with some being more crucial for prediction than others. To efectively combine these representations based on their significance, we utilize an additional attention network within each modality, ultimately determining a context vector. • Text Context Vector: The text representations = {ℎ1 , ℎ2 , . . . , ℎ } obtained from ViLT are passed into an additional attention layer to compute learnable attention scores for each token in . The calculation is as follows:
= softmax( · ℎ + ), After finding attention scores for each token, we find the context vector for the text modality by multiplying the ℎ of token with its attention score . • Vision Context Vector:
Similarly, the vision representations = {ℎ1 , ℎ2 , . . . , ℎ } are also fed into another additional attention layer to get the attention weights: = softmax( · ℎ + ),
= 1, 2, . . . , Having calculated the attention scores for each patch, we determine the context vector for the vision modality by multiplying the representation ℎ of each patch with its corresponding attention score .
= 1, 2, . . . ,
= ∑︁ · ℎ =1 = ∑︁ · ℎ
=1 = concat[ , ] = MLP()
′ = · +
The context vectors and are summed to the pooled representation ℎ to get more enhanced representations ′ and ′ where ′ = ℎ + ′ and ′ = ℎ + ′. Finally, we fuse both vision and text modality concatenating ′ and ′ and pass them into a Multi Layer Perceptron (MLP) to get the modality fused feature. (1) (2) (3)
After finding the modality fused feature representation , it is fed into a classification layer. The representation is the logits is employed for the classification process by the following:
Finally, we calculate the Cross-Entropy (CE) loss based on ′ with the ground truth.
All experiments were conducted using Python (version 3.12) and PyTorch, leveraging the free NVIDIA Tesla P100 GPU provided by Kaggle. For the pretrained vision language models, we use HuggingFace transformers library (version 4.40.1). The models used for tasks 4 and 5 have been trained for 15 epochs, and the models for task 6 have been trained for 50 epochs from their pretrained checkpoints. We have oitenA h p 2
+ oitenA + edlicous
olking A oLgits used the schedule-free AdamW optimizer [8] which requires no explicit hyperparameters for the optimization stopping step due to not using scheduling altogether. The initial learning rate used is 10− 4. The epsilon value of the optimizer is 10− 6, and the coeficients for computing the running average of the gradient and its square are 0.9 and 0.999 respectively. The seed or random state value used across all operations is 42. The batch size used in all the instances is 64.
For the classification head, the dimension of the hidden layer is 256 and the dropout layer sets 10% of the input values to zero. To do better modeling and include diferent explainability results, we adopt diferent training and explainable experiment settings from EDAL, ITPT, HateXplain [ 9, 10, 11] papers.
Our evaluation metrics for validation follow the same scoring system selected for the EXIST test rankings. “ICM-Hard” is the primary metric in the hard-hard evaluation context and “ICM-Soft” serves the same purpose in the soft-soft evaluation context. Additionally, the “Macro F1” values have also been listed as a more traditional metric that does not consider hierarchical classifications. The PyEvALL library [12] has been used to calculate the scores throughout all the experiments.
• ICM-Hard: The Information Contrast Model-Hard (ICM-Hard) metric [13] is designed to evaluate unbalanced hierarchical multi-label classification problems by incorporating hierarchical relationships and category specificity. ICM-Hard operates by considering the information content (IC) of categories and their intersections. The IC of a category represents the probability of items to appear in the category or any of its descendants, providing a measure of category specificity. Given two feature sets A and B, the metric is defined as: (, ) = 1() + 2() − ( ∪ )
() = − ( ())
The values selected for 1, 2 and for evaluating these experiments are 2, 2 and 3 respectively. • ICM-Soft: The Information Contrast Model-Soft (ICM-Soft) metric is an extension of ICM-Hard, designed to handle hierarchical multi-label classification problems in a learning with disagreement (LeWiDi) scenario. The metric is defined similarly using information content but it accommodates for soft ground truth assignments and soft system outputs. Given a category with an agreement to a given item, the IC is defined as —
({⟨, ⟩}) = − log2( ({ ∈ : () ≥ })) A recursive function is applied to calculate the ICM-Soft over a set of assignments: ︃( ⋃︁{⟨, ⟩}
)︃ =1 = (⟨1, 1⟩) + ︃( )︃ ⋃︁{⟨, ⟩} − ︃( ⋃︁{⟨(1, ), min(1, )⟩} )︃ =2 =2 ({⟨, ⟩}) = − log2( ({ ∈ : () ≥ }))
Where (, ) is defined as the lowest of common ancestor of categories and . • Macro F1: The F1 score is the harmonic mean of precision and recall for a class. The macro F1 metric is the average of the F1 scores of each class. All classes are given equal weight in this metric, which is desirable for underrepresented classes in highly imbalanced datasets.
Macro-F1 = =
1 ∑︁
=1 2 · · +
For our experiments, we have chosen CLIP [14] and ViLT [15] as our baseline models due to their multimodal capabilities in image-text pairs. For standard classification using CLIP, we concatenated the representations obtained from both vision and language inputs and passed them into an MLP for classification. For ViLT, we extracted the representations of the CLS token and fed them into an MLP for standard classification. As ViLT outperformed CLIP in most scenarios, we selected ViLT as our final model for experiments with the proposed architecture.
Table 4 displays the performance comparison of three models — CLIP, ViLT, and our approach on the task 4 test dataset. Overall, ViLT outperforms both CLIP and our approach, exhibiting the lowest error rates and achieving the highest macro F1 score of 0.6742. CLIP performs moderately well, with a macro F1 score of 0.6524, while our approach shows the least favorable performance with a score of 0.6101. This trend is consistent across subsets, with ViLT consistently outperforming the other models.
Particularly, in the English subset, ViLT achieves the highest macro F1 score of 68.37%, followed by CLIP at 67.13%, emphasizing ViLT’s efectiveness in this context. However, in the Spanish subset, although ViLT still leads with a score of 66.57%, the performance gap between ViLT and CLIP is narrower. Unfortunately, our proposal demonstrates the highest error rates across all subsets, suggesting potential areas for improvement.
Table 5 illustrates the performance of the models for Task 5 test dataset. In this task, CLIP achieves a macro F1 score of 38.56%, with values of -0.6546 for ICM-Hard and -5.3096 for ICM-Soft instances. ViLT performs marginally worse than CLIP with a macro F1 score of 38.41%, having -0.7089 for ICM-Hard and the value of -5.9832 for ICM-Soft. Our approach shows the best performance in both the ICM metrics with the scores being -0.6123 and -5.2668 respectively. The macro F1 performance of 38.41% is comparable to ViLT.
On the Spanish tweet subset, CLIP performs similarly to its overall performance, with a macro F1 score of 36.94%, while ViLT slightly outperforms CLIP with a score of 37.33% and the lowest error rate for ICM-Soft instances at -5.7680. Our approach achieves the most promising results on the Spanish subset, with the highest values for both ICM-Hard and ICM-Soft instances being -0.6144 and -5.4547 respectively. For the English subset, CLIP performs slightly better with a macro F1 score of 40.19%, whereas ViLT shows a slight decrease with a score of 38.35%. Our approach remains competitive with the highest score for ICM-Hard instances and a macro F1 score of 39.49%.
Key observations indicate that performance varies across diferent subsets and models, with CLIP and ViLT performing consistently and our approach obtaining the highest scores in the ICM-Hard metric and largely having better performance in the ICM-Soft metric as well, while maintaining competitive macro F1 scores.
In Table 6, we reported the performance of models on Task 6 in test dataset. For the overall dataset, our approach outperforms both CLIP and ViLT, achieving the best ICM-Hard score of -1.3631 and the highest macro F1 score of 33.56%, though it shows a slightly lower ICM-Soft score of -13.2556 compared to ViLT’s best score of -11.2593. In the Spanish subset, Our approach again demonstrates superior performance with the highest macro F1 score of 31.92% and the best ICM-Hard score of -1.6610, although it has a worse ICM-Soft score of -15.5081 compared to CLIP’s -12.4430. For the English subset, our approach achieves the highest overall performance, leading with a macro F1 score of 34.87% and the highest ICM-Hard score of -1.1027, while ViLT shows the best ICM-Soft score of -10.3388.
Across all subsets, our approach excels consistently in macro F1 and ICM-Hard metrics, indicating its robustness in handling hard instances for multi-label tasks, while ViLT often provides the best scores for ICM-Soft instances, demonstrating its efectiveness in handling soft instances. Overall, this table highlights the strengths of each model in diferent aspects of Task 6, with our approach showing the most balanced and highest performance in key metrics across various subsets.
From the confusion matrices of task 4 in figure 2 we can see that the CLIP model excels in identifying the 300 true positive cases, which is 69% of the total. But it has high Type-I (49% false positives) and Type-II (31% false negatives) error, resulting in significant misclassifications.
ViLT performs better than CLIP yielding 341 true positive cases, which accounts for 79% of the total. But it has a higher Type-I (55% false positives) and moderate Type-II (21% false negatives) error, resulting lower accuracy to detect the true negative cases which only account for 45% of the total. Our approach, on the other hand, performs consistently on both the true positives and the true negatives, with both covering 63% of the total. To be more specific, the true negative cases have been handled T IEC 0.43 R D l a tcu NO 0.32 A LETANUDG EM 0.39 J DIRECT better than both CLIP and ViLT. It also has consistent rates for Type-I and Type-II errors, both clocking at 37%.
Using the confusion matrices for task 5 from figure 3, we observe that the CLIP model performs well on the ‘NO’ class with 148 correct predictions but sufers from high misclassification rates for ‘DIRECT’ and ‘JUDGEMENTAL’ classes. The ViLT model shows a similar trend. It performs better than CLIP on both the ‘NO’ and ‘DIRECT’ classes with 58% and 30% correct predictions respectively. But it still shows a tendency towards wrongly considering other classes to be ‘JUDGEMENTAL’, despite it having very low representation in the dataset. Our approach, on the other hand, performs really well in predicting the true ’DIRECT’ samples with the highest amount of 57%. It has comparable performance for the ‘NO’ class, but struggles heavily with ‘JUDGEMENTAL’ samples, showing signs of being afected by the dataset distribution.
In this study, we addressed the pressing issue of sexism in online content, focusing on the complexity posed by multimodal and multilingual (English and Spanish) data. Our proposed attention-enhanced approach efectively integrates textual and visual contexts, providing enriched representations for sexism classification. By leveraging the EXIST 2024 dataset, which ofers a thoroughly outlined look into the variants of sexism present in online content, we demonstrated improvements over existing models such as CLIP and ViLT. Our empirical evaluations and subsequent error analysis highlighted both the strengths and areas for improvement in our system. The findings underscore the importance of comprehensive datasets and advanced model architectures in tackling online discrimination. Future work will explore further enhancements in multimodal learning and expand the approach to other forms of harmful content beyond sexism, contributing to a safer and more inclusive online environment.
This project has been sponsored by Penta Global Limited, Bangladesh. We would like to express our deepest gratitude to Penta Global for for their financial support.
[1] J. B. Walther, Social media and online hate, Current Opinion in Psychology 45 (2022) 101298. [2] B. Kostadinovska-Stojchevska, E. Shalevska, Internet memes and their socio-linguistic features,
English Language and Linguistics 2 (2018). doi:10.5281/zenodo.1460989. [3] D. Kiela, H. Firooz, A. Mohan, V. Goswami, A. Singh, P. Ringshia, D. Testuggine, The hateful memes challenge: Detecting hate speech in multimodal memes, 2021. arXiv:2005.04790. [4] H. R. Kirk, W. Yin, B. Vidgen, P. Röttger, SemEval-2023 Task 10: Explainable Detection of Online Sexism, in: Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval2023), Association for Computational Linguistics, 2023. URL: http://arxiv.org/abs/2303.04222. doi:10.48550/arXiv.2303.04222. [5] E. Fersini, D. Nozza, P. Rosso, Overview of the evalita 2018 task on automatic misogyny identification (ami), in: EVALITA@CLiC-it, 2018. [6] L. Plaza, J. Carrillo-de-Albornoz, V. Ruiz, A. Maeso, B. Chulvi, P. Rosso, E. Amigó, J. Gonzalo, R. Morante, D. Spina, Overview of EXIST 2024 – Learning with Disagreement for Sexism Identification and Characterization in Social Networks and Memes, in: Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Fifteenth International Conference of the CLEF Association (CLEF 2024), 2024. [7] L. Plaza, J. Carrillo-de-Albornoz, V. Ruiz, A. Maeso, B. Chulvi, P. Rosso, E. Amigó, J. Gonzalo, R. Morante, D. Spina, Overview of EXIST 2024 – Learning with Disagreement for Sexism Identification and Characterization in Social Networks and Memes (Extended Overview), in: G. Faggioli, N. Ferro, P. Galuščáková, A. G. S. de Herrera (Eds.), Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum, 2024. [8] A. Defazio, Xingyu, Yang, H. Mehta, K. Mishchenko, A. Khaled, A. Cutkosky, The road less scheduled, 2024. arXiv:2405.15682. [9] M. Fahim, M. S. Shahriar, M. R. Amin, Hatexplain space model: Fusing robustness with explainability in hate speech analysis (2023). [10] M. Fahim, Aambela at blp-2023 task 2: Enhancing banglabert performance for bangla sentiment analysis task with in task pretraining and adversarial weight perturbation, in: Proceedings of the First Workshop on Bangla Language Processing (BLP-2023), 2023, pp. 317–323. [11] M. Fahim, A. A. Ali, M. A. Amin, A. M. Rahman, Edal: Entropy based dynamic attention loss for hatespeech classification, in: Proceedings of the 37th Pacific Asia Conference on Language, Information and Computation, 2023, pp. 775–785. [12] UNEDLENAR, Pyevall, https://github.com/UNEDLENAR/PyEvALL, 2024. [13] E. Amigo, A. Delgado, Evaluating extreme hierarchical multi-label classification, in: S. Muresan, P. Nakov, A. Villavicencio (Eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Dublin, Ireland, 2022, pp. 5809–5819. URL: https://aclanthology.org/2022.acl-long.399. doi:10. 18653/v1/2022.acl-long.399. [14] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al., Learning transferable visual models from natural language supervision, in: International conference on machine learning, PMLR, 2021, pp. 8748–8763. [15] W. Kim, B. Son, I. Kim, Vilt: Vision-and-language transformer without convolution or region supervision, in: International conference on machine learning, PMLR, 2021, pp. 5583–5594.