Sexism has pervasive negative efects on both individuals and society. This paper presents a generalizable BERT-based approach to identifying and classifying the source intent of sexism across diferent social network channels. This approach focuses on individual models trained on the text of tweets and then applied to both Meme (image) and Video data using OCR and annotations respectively. The identification model performed well across all channels and the classification model performed well on both Tweets and Memes. This research suggests that a single model, fine-tuned on one media type can be efectively applied to multiple media types with minimal data preprocessing required.
For all three data sets there were accompanying metadata provided. These included data about the annotators, the labels, as well as the text or transcripts of speech present in the media.
The data concerning the annotators was not used in this research, and the conclusions reached by the annotators were assumed to be correct for labeling the data.
The labels provided for each individual Tweet, Meme or Video are shown in Table 2. All labels shown are present in the Tweets dataset, and all labels except “Reported” are present in the Memes and Videos datasets.
As seen in Table 2, the provided data included 10034 tweets, split across training, dev (which, for this paper, was used as validation), and test datasets. Each of these datasets were split into approximately 53% Spanish language, and 47% English language tweets. The training and dev datasets were labeled with annotations from 6 annotators, with YES indicating the annotator’s belief that the tweet was sexist and NO indicating the belief that it was not sexist. Demographic data for the annotators was also provided, but was not used in training of the models.
No preprocessing was applied to this data, all elements such as hashtags and emojis were retained. No attempts were made to correct spelling or grammar, and no data augmentation techniques were used. 2.2. Memes The Memes dataset was comprised of 5097 images, along with accompanying metadata and labels. These data were split across training and test data. With a split of approximately 51% Spanish language, and 49% English language. These were composed of 4703 JPEGs and 394 PNGs of varying dimensions. These images were supplied with a provided copy of their text content generated by Optical Character Recognition (OCR).
For further analysis on the images, all were converted to PNG files using a standard sRGB color palette.
A secondary description of the image was generated using Gemma 3 [
The Video dataset was also split into train and test datasets, with 2525 training videos and 674 test videos. All videos were mp4 files of varying duration, with accompanying transcriptions.
A Whisper speech-to-text model [8] was used to annotate the videos, generating an additional transcript to the one provided in the video metadata. Ultimately, this secondary transcript was not used in the final method due to time constraints.
For the competition submission two methods were used, they varied on the method used to create the outputs and how the data was loaded for training.
The first method (labeled bergro_1) used a pair of pretrained BERT models, fine-tuned on the tweets in the training data to output a single soft classification label. This label was used directly for the soft evaluation then mapped to YES/NO (with NO for all values less than 0.5, otherwise YES) for the hard evaluation. The data for this model was loaded with each tweet appearing once per epoch, with the label created by dividing the number of annotators indicating YES divided by the total number of annotators. Therefore, for a tweet which three annotators labeled the tweet YES (sexist) and three labeled it NO (not sexist) the label would be 0.5.
The second method used a pair of pretrained BERT models, fine-tuned on the tweets in the training data, but this time as a classifier with YES/NO outputs. The data for this model was loaded, with each tweet appearing once per annotator, with that annotator’s label. As all tweets in the training and dev data provided were labeled by six annotators, the models were trained on the tweets six times per epoch, often with conflicting labels, reflecting the disagreement between annotators.
For the final competition, the models were trained using both the training and dev data sets to increase the amount of training data for the model.
The approach to tackle this problem for both submission methods was to fine-tune a version of a BERT
model. This approach was taken as these models work well for classification tasks and are fine-tuneable
of consumer grade GPUs. Multilingual BERT variants exist [
For processing the English language tweets, DistilBERT [9], a smaller, more eficient version of the BERT
(Bidirectional Encoder Representations from Transformers) language model [
The key innovation behind DistilBERT is a technique called knowledge distillation. During its pretraining phase, a smaller student model (DistilBERT) is trained to mimic the full output distribution of a larger, pre-trained teacher model (BERT). This process transfers the rich dark knowledge from the teacher to the student.
For the bergro_1 approach the model was adapted to a single output label, to allow for the regression style approach, then fine-tuned on the training data using the dev data as validation data to optimise the hyperparameters.
For the Spanish language model BETO: Spanish BERT [10] was selected, this model is larger than the DistilBERT model used for the classification of English language tweets. This model contains approximately 110 million parameters, and is structured like the original BERT model, although the pre-training has some adaptations taken from later BERT derivatives such as RoBERTa [11]. The key diference, however, is the training corpora used which included 3 billion tokens of Spanish language content [12].
To augment the OCR generated content for the memes, the images were fed to an multimodal LLM to
provide a description. The model used was Gemma3 [
The prompts used for this purpose were: • English: “In 250 words or less describe the image. Do this without any preamble.” • Spanish: “En 250 palabras o menos, describa la imagen. Haga esto sin preámbulos.” (this was machine translated from the English prompt).
For two Spanish memes the model ignored the requested word count and the text was cropped to 512 tokens to fit within the input size used in the BERT-based models.
Training the bergro_1 model, which is a regressor model with a single output, used the Mean Squared Error (MSE) loss, which is the squared diference between the prediction probability distribution of the annotators and the actual distribution.
For training the bergro_2 model, which is a binary (YES/NO) classifier, accuracy was used as a metric. This measures the overall proportion of correct predictions.
For the final results the metrics were provided by the EXIST 2025 organizers. As in previous iterations of EXIST, the organizers used both a soft evaluation and a hard evaluation. The soft evaluation was intended to measure the model’s ability to capture disagreements. It achieves this by comparing the probability distribution of predicted labels with that of the labels provided by the annotators. In contrast the hard evaluation requires the system to predict a single label for every tweet and is compared against an absolute value calculated by a majority vote of the annotators’ responses. For those instances for which there is no majority class (i.e. for subtask 1 there are 3 YES labels and 3 NO labels) were removed from this evaluation scheme. These are then combined into three metrics per evaluation type (soft/hard). 3.4.1. Hard evaluation • Information Contrast Model (ICM): How well the predicted outcomes align with group diferences within the data [13]. • Normalized Information Contrast Model (ICM Norm): A normalized version of ICM. • F1: A balanced accuracy measure taking into account false positives and false negatives [14].
• Information Contrast Model-Soft (ICM-Soft): A modification of the original ICM metric in which the Information Contrast is estimated from both the soft ground truth values and the soft predictions [15]. • Information Contrast Model-Soft Norm (ICM-Soft Norm): A normalized version of ICM
Soft. • Cross Entropy: The diference between the predicted probability distribution and the ground truth distribution.
For training the Source Intention model, only the Tweets dataset was used. The model was trained on the proportion values for each class, as shown under intention in table 1. Two separate models were trained, as for the sexism identicfiation. One model for the English content; DistilBERT, and one model for the Spanish content; BETO: Spanish BERT. Cross validation was performed in order to tune the hyperparameters, with the final hyperparameters shown in Table 3. The model was then used to predict tweets from a Soft likelihood estimates for each class from a previously unseen dataset. This model was also used to predict the Soft estimates for both Videos and Memes. As both of these datasets did not include the “Reported” label, this was removed from the predictions with the remaining values being normalized to once again sum to one.
Hard values were inferred from the soft values by selecting the class with the maximum predicted probability (argmax) for each instance.
As seen in Table 4 the best results for task 1.1 were achieved with the bergro_1 model, using a regression approach to tweet classification. This outperformed bergro_2 in which a hard label was directly generated. Table 4 shows the runs submitted to the EXIST lab organizers.
As seen in Table 6, the models trained on source intention from the Tweets training data performed well on both an unseen Tweets dataset as well as on the Memes and Video data. For each dataset the Majority score was also included, this shows the metrics for selecting the most common labels in each dataset. The BERT model performed well on both Tweets and Memes, notably outperforming the majority selection, suggesting much better than random inference. This applied to both the hard and soft labels. The performance was less good on Videos, slightly outperforming the majority selection on hard labeling, and almost matching it on soft labeling.
The focus of the EXIST lab as a whole was to build models tuned to each data stream, Tweet, Meme and Video. This paper chose a slightly diferent path, namely to focus on a single generalized model that could be applied across channels. Using a generalized model reduced training time and resources required. Furthermore, the tweets set was the largest, and simplest to process.
For the trained BERT models, predicting a soft likelihood estimation and then mapping that estimation to a hard-label (bergro_1) outperformed the direct generation of a hard-label (bergro_2). This trained model performed strongly on tweets, and without further training also outperformed, the majority-class baseline for both memes (on both OCR generated text and LLM generated descriptions) and videos (on the provided transcripts). Using a multimodal LLM to provide a description outperformed using the OCR generated text context. This may be due to the spatial and non-textual information the multimodal LLM was able to provide.
The trained BERT model showed promise in predicting labels across diferent social media formats. Our results show that a BERT model trained only on Tweets data can notably outperform a majority baseline for both Tweets and Memes. This applied for Soft likelihood estimation as well as the Hard classification.
The model performed less well on the Videos dataset, outperforming the majority baseline in Hard classification, and roughly matching it in Soft likelihood estimation.
This suggests that a pre-trained BERT model can be tuned to identify both sexism and intent across multiple media channels using only a single channel for tuning. It also shows that this may not work universally, with certain similarities required between the channels.
Amongst the challenges with detecting sexism, particularly direct sexism is that there are often eforts made to obfuscate the intent of a post, hashtags and slang terms used euphemistically to indicate ideological context [16, 17]. These methodologies change rapidly meaning that any system seeking to detect them also needs to dynamically update over time. This holds some importance within the context of this paper, as well as more broadly.
The data used in this paper spans from 2015 through to 2024. As such, models are also impacted by the evolution of language use through this time period. Language usage has been shown to change rapidly within social networks [18], with word and phrase meanings shifting over time.
This paper has shown some promise in using BERT models trained on one media channel to classify sexism in another channel. This could be further explored to develop systems that could quickly be deployed to new and emerging media channels with minimal re-tuning required.
This research did not utilize all available information from the provided datasets. This includes the information about the annotators, and further information contained within the media, such as imagery in the videos. All this information is likely to hold value, and exploiting this in future research could be the key to unlocking improved accuracy and precision, whilst gaining deeper insights.
The authors have not employed any Generative AI tools in writing this paper. [8] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, I. Sutskever, Robust speech recognition via large-scale weak supervision, in: International conference on machine learning, PMLR, 2023, pp. 28492–28518. [9] V. Sanh, L. Debut, J. Chaumond, T. Wolf, Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, ArXiv abs/1910.01108 (2019). [10] J. Cañete, G. Chaperon, R. Fuentes, J.-H. Ho, H. Kang, J. Pérez, Spanish pre-trained bert model and evaluation data, in: PML4DC at ICLR 2020, 2020. [11] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, V. Stoyanov, Roberta: A robustly optimized BERT pretraining approach, CoRR abs/1907.11692 (2019). URL: http://arxiv.org/abs/1907.11692. arXiv:1907.11692. [12] J. Cañete, Compilation of large spanish unannotated corpora, 2019. URL: https://doi.org/10.5281/ zenodo.3247731. doi:10.5281/zenodo.3247731. [13] E. Amigo, A. Delgado, Evaluating extreme hierarchical multi-label classification, in: S. Muresan, P. Nakov, A. Villavicencio (Eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Dublin, Ireland, 2022, pp. 5809–5819. URL: https://aclanthology.org/2022.acl-long.399/. doi:10. 18653/v1/2022.acl-long.399. [14] N. Chinchor, MUC-4 evaluation metrics, in: Fourth Message Understanding Conference (MUC4): Proceedings of a Conference Held in McLean, Virginia, June 16-18, 1992, 1992. URL: https: //aclanthology.org/M92-1002/. [15] L. Plaza, J. Carrillo-de Albornoz, I. Arcos, P. Rosso, D. Spina, E. Amigó, J. Gonzalo, R. Morante, Overview of exist 2025: Learning with disagreement for sexism identification and characterization in tweets, memes, and tiktok videos (extended overview), in: G. Faggioli, N. Ferro, P. Rosso, D. Spina (Eds.), CLEF 2025 Working Notes, 2025. [16] A. Perliger, C. Stevens, E. Leidig, Mapping the ideological landscape of extreme misogyny, International Centre for Counter-Terrorism, 2022. [17] W. Zhu, H. Gong, R. Bansal, Z. Weinberg, N. Christin, G. Fanti, S. Bhat, Self-supervised euphemism detection and identification for content moderation, in: 2021 IEEE Symposium on Security and Privacy (SP), 2021, pp. 229–246. doi:10.1109/SP40001.2021.00075. [18] T. Dembe, The impact of social media on language evolution, European Journal of Linguistics 3 (2024) 1–14.