This paper presents our participation in the EXIST (sEXism Identification in Social neTworks) challenge at CLEF 2025, focusing on the classification of tweets and memes. We participated in all the tasks for tweets and memes, including both hard and soft classifications for tweets and hard classification for memes. For tweet classification, we propose a multi-task headed BERT model enriched with relevant information surrounding the tweet, helping the model achieve a full understanding of the tweet and its context. For memes, the paper explores the use of a Vision-Language Model (VLM)-based application to detect and categorise sexism in diferent scenarios, leveraging the ability of such models to understand the relationship between images and text in situations where sexist ideas are often expressed subtly. Our solutions achieved excellent performance, ranking first in all soft-soft tweet classification tasks and second in all hard-hard meme classification tasks. Content Warning: This paper includes examples of hateful, explicit and sexist language presented for illustrative purposes.
Sexism, in the form of pre-judges or hateful comments, is a prevalent form of digital violence that
must be addressed in a context where social networks and digital platforms are ubiquitous. In 2024,
81% of French women reported experiencing sexist comments on these platforms [
Therefore, automatic identification of sexist content on social media becomes a crucial task. To foster
such initiatives, the EXIST 2025 challenge [
The categories of sexism used in this study are defined on the EXIST 2025 challenge website. Given the complexity and the need for comprehensive detection tools, we decided to tackle both tweet-based subtasks (1.1, 1.2, 1.3) and meme-based subtasks (2.1, 2.2, 2.3) in our work. To address this challenge, we evaluated and compared state-of-the-art techniques, incorporating our insights to propose two tailored solutions: one for textual classification and another for meme classification.
The remainder of the paper is organised as follows. In Section 2, we provide a brief overview of approaches used for automatic detection of sexist content. We then describe the dataset and the evaluation metrics in Section 3. We describe our proposed solutions for tweets and memes in Section 4. We report the results of our experiments in Section 5. Section 6 concludes the paper and outlines the directions for future work.
In this section, we present the diferent approaches used to detect online sexism. These methods fall
into four broad categories: traditional approaches, Deep Learning-based approaches, transform-based
approaches (BERT and LLM) [
Before the emergence of deep architectures, a number of studies used classic machine learning methods
- such as Logistic Regression, SVMs or Random Forests. These methods were generally combined with
feature extraction techniques (N-Grams, TF–IDF, Static Word Embeddings) [
Deep Learning models have made it possible to capture complex patterns using specialised architectures.
CNN-BiLSTM architectures, combining convolutional neural networks (CNNs) to detect local patterns
(e.g. ofensive N-Grams) and BiLSTMs to model long-term contextual dependencies, marked a significant
advance [
The advent of transformers has revolutionised the detection of sexism thanks to their ability to encode
the overall context of text:
• BERT and derivatives: Models such as RoBERTa [
Existing datasets have played a crucial role in advancing the field of online sexism detection. Notable
examples include:
• Sexist Stereotype Classification (SSC) [
These datasets have contributed significantly to our understanding of online sexism, enabling researchers
to develop more accurate and robust detection methods. With the rise of visual social media platforms,
sexism is increasingly conveyed through multimodal forms such as memes, which blend text and images
to encode prejudice in subtle, culturally loaded ways. This shift has spurred research into models
capable of understanding both modalities simultaneously. Several key datasets support this area:
• MAMI (SemEval-2022) [
The correlation between textual and visual elements in memes make VLMs (Vision Language Models),
architectures built by combining large language models and vision encoding, suitable for the task.
Several studies have been done on applying VLMs to social media memes for semantic understanding
[
Despite notable advances, several challenges persist:
1. Knowledge obsolescence: Pre-trained models possess frozen knowledge that may not always
capture recent language and usage developments in tweets, limiting their relevance in current
contexts. Valavi et al. [
In this paper, we tackle the aforementioned limitations by proposing two novel approaches that enhance the performance and robustness of sexism detection models in social media, as well as in tweets and in memes. The details of our methodology are presented in Section 4, which outlines how we address these challenges and advance the state-of-the-art in sexism detection.
Our study is based on the EXIST 2025 dataset, which ofers a rich collection of tweets and memes annotated for online sexism detection. We draw upon these subsets to train and evaluate our approach. Tables 1 and 2 resume the datasets for tweets and memes respectively.
In the subsequent experimental phase, we will conduct model fine-tuning using the labelled training set, followed by evaluation on the development dataset. Since the meme dataset does not explicitly provide for a development dataset, the training set was divided into two 80/20 partitions (seed: 1234), respectively for fine-tuning and for evaluating results on never-before-seen data. Ultimately, our approach will be benchmarked against challengers using the held-out test set.
The oficial evaluation metric for this challenge is the Information Contrast Measure (ICM) [
This methodology is structured into two distinct components. The first part focuses on our approach to tweet analysis, while the second part details our method for meme analysis.
The previous comprehensive literature review on classification techniques revealed that BERT and
LLM models are at the forefront of natural language processing tasks. Given their state-of-the-art
performance, we focused our eforts on these models. Our initial step involved conducting multiple
tests to determine the optimal formatting for tweets to be processed by BERT. This process ensured
that the input data was structured to maximise the model’s performance. For LLM, Quan and Thin [
eninID https://t.co/UJvloR0IP
To investigate the impact of annotator characteristics on sexism detection, we conducted a
comprehensive analysis of annotator information using Chi-Squared tests and Logistic Regression models
with feature importance. To improve the model’s understanding of the subjective nature of sexism,
we identified study level, country of origin, and ethnicity as relevant annotator attributes through
our analysis. By integrating these attributes, we aimed to enhance the model’s ability to capture
diverse perspectives and biases. To achieve this, we vectorised the selected annotators’ information and
embedded it into the CLS token of the BERT model, prior to passing it to the classification head.
Table 5 illustrates a simplified example of the vectorisation process of annotator information, featuring
three annotators for clarity. Note that in our actual implementation, the final vector is 65 elements
long, encompassing a more extensive range of ethnicities (more than 3), study levels (more than 2),
and countries (more than 2). This simplified representation is intended to facilitate understanding and
presentation.
“ethnicities_annotators”: ethnicities_annotators:
[“White or Caucasian”, “Hispano or Latino”, [
Our fine-tuning eforts with both BERT and LLM models yielded promising results (cf. Table 6), closely
approaching the top performances achieved in the previous year [
However, we sought to further enhance our approach. An analysis of misclassification revealed that certain tweets were incorrectly classified due to their ambiguity or references to recent topics not present in the training data. For instance, tweets referencing very recent events or slang not included in the model’s vocabulary posed significant challenges.
To overcome the limitations of traditional models, we leveraged the capabilities of AI agents that can
dynamically interact with their environment with tools, plan actions, and integrate external data in
real-time [
tpmorP ksaT tewT eport tnegA petS nopseR .rgnesap 2 ehT sihT tangerp“ mhalko dna hcraeS
We equipped our agent with the DuckDuckGo web search tool. We considered several options for utilising this AI agent: • Direct Classification by the Agent: The agent classifies the tweet directly using relevant information gathered from the web search. Its user prompt is available in Appendix B • Context Retrieval by the Agent: The agent retrieves contextual information around the tweet using the web search (the AI agent user prompt is available in Appendix C), which can then be fed into a BERT or LLM.
– BERT-based Architecture: We fed the retrieved context into a BERT model, employing a
Siamese Dual Encoder architecture (SDE) [
To optimise the LLM performance and output format for each experience configuration, various prompts have been empirically tested. Furthermore, a system prompt is appended to the LLM Agent by the library, facilitating correct parsing of its output for tool calls. For more information on the library details, see Section 5.1.
We employed two distinct LLMs in our approach: one for powering the autonomous AI agent and another for fine-tuning to classify tweets.
• Autonomous AI Agent: We chose the Llama-3.3-70B-Instruct model for its ability to handle complex tasks, which requires well-formatted responses to efectively leverage tools. • Fine-Tuned LLM for Classification: Due to computational limitations, we used a smaller LLM, Llama-3.2-3B-Instruct, for fine-tuning. Despite using 4-bit quantisation, we lacked the necessary computational resources to fine-tune a 70B model.
Our experiments, presented in Table 7, reveal that BERT models augmented with contextual
information outperform LLM with context, underscoring the eficacy of contextual enrichment on encoder-only
architectures. In contrast, the incorporation of context into fine-tuned LLM appears to degrade
performance, potentially due to the phenomenon of context hijacking [
Moving forward, our approach will leverage contextual information retrieved and formatted by an AI agent. As this external data is generated, evaluating its quality is a necessary consideration. An initial evaluation of the generated contexts is presented in Appendix D. While we do not delve further into this aspect in this paper, as it is not the primary focus, additional analysis may be merited for this case and future applications.
One of the significant challenges we encountered was annotator disagreement, the Unlabelled data.
When there was no clear majority—such as three "YES" and three "NO" or three "DIRECT" and three
"JUDGEMENTAL"—we could not use these data points because we were training the model for hard
label classification. This was not a trivial detail, as the amount of training data can impact model
performance [
A solution we identified was to train the model with probabilities rather than hard labels, aligning with
the principles of soft label learning (SLL) as explored in [
Now that we have selected the BERT architecture, in Figure 2 we conducted extensive runs with various
of these models, including XLM-Roberta [
One of the key advantages of selecting the BERT architecture is that, with minimal additional efort and
computational resources, we can accommodate all three tasks and both hard and soft labels within a
single multi-task BERT model [
Building upon the best existing approach, which employed a multi-task BERT [
A crucial aspect of our proposed architecture is that we leverage the consistency of "NO" probability across all three tasks. By recognising this consistency, we adapted our training approach to compute the loss of the Classifier B (subtask 1.3) only when the tweet is classified as sexist by the Classifier A. This hierarchical design enables us to filter out non-sexist examples and focus on the relevant samples for subtask 1.3, thereby improving performance and establishing coherence between the two classification heads despite their distinctness.
In contrast, our experiments with a single classification head for all categories did not perform well, likely due to the large number of categories. Similarly, attempting to predict only "YES" labels and computing "NO" labels by doing 1-"YES" also yielded subpar results.
Notably, this 2+1 architecture significantly impacted the performance of our results for subtasks 1.2 and 1.3. While subtask 1.1 results remained relatively consistent, our proposed architecture demonstrated substantial improvements for the latter two tasks. In particular, it led to a substantial improvement in soft classification, with an increase of two to three ICM Soft Norm points. The final results of our model are presented in the Section 5.2. gnidebmE-tewT LATNMEGDUJ
TCERID DTOPER DTOPER refisalC tewT xamtfoS ksaTptuO
TREB TCERID 4 CNELOIV-LAUXESNO-NYGOSIM
ON xetnoC YTLAUQENI-LACIGLOEDI gnidebmE-xetnoC 1 refisalC )ON(P
CNELOIV-LAUXES 2 domgiS ksaTptuO
ECNAMOD-GNIPYORETS
NOITAFITCEJBO
To format the results, we rounded the probabilities to the nearest 1/6–as there are six annotators–and ensured that the sum of probabilities was 1 for subtasks 1.1 and 1.2. For hard classification, we adopted the following strategies: for subtasks 1.1 and 1.2, we selected the feature with the maximum probability; for subtask 1.3, a multi-label classification task, we chose all features with probabilities exceeding 0.25. This threshold was determined through testing on the training and development datasets during soft-to-hard label conversion.
In summary, our methodology involved a thorough literature review, extensive testing, and innovative use of AI agents to enhance contextual understanding. It also incorporates annotator information to address subjectivity and employs a multi-task headed approach, sharing base layers across tasks while capturing unique characteristics through specific output heads. 4.2. Memes
Regarding the Meme Dataset, we first wanted to verify the accuracy of the text and image pairs provided together. For each meme, we extracted the superimposed text using Florence-2 Large and we then compared it with the provided one. Average Jaccard similarity in terms of unigrams and bigrams showed respective values of 0.9518 and 0.9495, marking a minor diference that could be explained as follows: • for unigrams, since diacritics matters, two semantically equal words could be treated as diferent (e.g. "tenia" vs "tenía") • for bigrams, being defined as sequences of two adjacent words, the sequence of words has an efect on the computed Jaccard similarity
Through this comparative analysis of the extracted and given texts, we observed that the superimposed texts provided with the data exhibit superior transcription quality compared to those extracted using Florence-2. Notably, these texts feature proper accentuation and sequentiality, resulting in a readability closer to human standards. An exemplary illustration of these findings is presented in Figure 4. Consequently, we opt to utilise the provided text instead of relying on a specific extraction technique.
To tackle meme classification, informed by our literature review which highlighted the necessity of exploring multiple approaches, we investigated three complementary strategies: • Caption-Based Classification: representation of meme images as textual captions and classification of the captions using a fine-tuned text model. • Frozen Multimodal Classification: usage of pretrained VLMs in zero-shot and few-shot settings without fine-tuning. • Fine-Tuned Multimodal Classification: fine-tuning of medium-to-large VLMs on labeled sexist and not sexist memes for task-specific performance.
In this text-based classification approach, represented in Figure 5, meme images were first transformed into textual descriptions using Qwen 2.5 VL 32B (1) and these captions (2), jointly with their respective ground truths (3), were then used as input for fine-tuning XLM-RoBERTa (4). This two-stage pipeline was designed to exploit the visual understanding of vision-language models and the adaptability of multilingual transformers.
To analyse the impact of visual description granularity, we generated two types of captions: • Short Captions: concise descriptions capturing minimal visual content. • Detailed Captions: rich, context-aware descriptions reflecting nuanced or subtle cues in the image.
Figure 6 shows how the textual representation can difer from the same meme. The prompts employed for the generation of captions are disclosed in Appendices L to N.
This approach used frozen vision-language models in zero-shot and few-shot scenarios without taskspecific fine-tuning, in order to simulate realistic low-resource classification settings. We evaluated the following VLMs: • Qwen-VL 2.5 in its 7B, 32B, and 72B variants (zero-shot and few-shot) • Aya Vision 8B (zero-shot) • Mistral Small 3.1 24B (zero-shot) In the zero-shot setting, models were given only the meme image and a minimal classification prompt (showed in the Appendices G to K), with no prior examples. The employed prompts were significantly based on the guidelines provided to annotators for meme labelling across the three subtasks. We also tested variants where the model received either only the image, image plus superimposed text, or only the superimposed text. These variations aimed to quantify the importance of superimposed textual content over the final prediction. To evaluate few-shot performance, we included six example memes in the prompt using two diferent sampling strategies: • Random Few-Shot Sampling: six random examples from the training set, imposing a balanced extraction between sexist and not sexist memes • Polarised Few-Shot Sampling: three clearly sexist and three clearly non-sexist memes (i.e., with ≥5 of 6 annotators in agreement).
Images were resized to a maximum of 262,144 pixels (e.g., the size of a 512×512 image) maintaining their original proportions to fit within GPU memory constraints.
Finally, we tested the efectiveness of fine-tuning a set of VLMs for both sexism identification and classification. Specifically: • For subtask 2.1, we fine-tuned Florence 2 and Qwen 2.5 VL (7B and 32B). The dataset used for ifne-tuning was gathered from the available ground truths for the task, for a total 3,420 meme image-label pairs. • For subtasks 2.2 and 2.3, only Qwen 2.5 VL 32B was fine-tuned. The data curation criteria was slightly diferent with respect to subtask 2.1, since we excluded the memes labelled as not sexist from the ground truths of the sexism identification task. Eventually, the number of considered records was 1,815 (over the 3,197 available ground truths) for the source intention classification and 2,868 (over 4,250 available ground truths) for sexism categorisation.
The experimental setup and the fine-tuning hyperparameters for both Florence-2 and Qwen 2.5 VL are presented in detail in Appendix O. In contrast to previously proposed methods for meme analysis, our proposed LLM-based solution ofers a lightweight yet efective approach to detecting and classifying sexism in memes while incorporating the entire visual content into the classification pipeline. By avoiding high inference costs and proprietary APIs, this approach ensures compatibility with low-to-mid-tier hardware and promotes reproducibility by reducing computational requirements.
All experiments were conducted using PyTorch 2.5.1, the Hugging Face Transformer 4.50 library, and the Smolagents 1.4 library for AI agent development. The computational environment consisted of two GPUs with the following specifications: • NVIDIA A40 (46 GiB), driver version 555.42.06, CUDA 12.5 • NVIDIA A100 (40 GiB), driver version 555.42.02, CUDA 12.5
Additionally, VLMs with more than 7 billion parameters were loaded using Bitsandbytes 4-bit
quantisation technique, which reduces the size of the model and computational costs by representing
weights and activations with just 16 discrete levels [
In this section, we present the performance metrics of our proposed methods across all the three tasks. For tweets, evaluations were conducted under both soft and hard contexts, whereas meme-based methods were assessed under the hard evaluation setting. Our model was trained on the provided training dataset and evaluated on the corresponding validation dataset for tweets. For memes, as mentioned before, we split the training dataset to create a validation dataset, thereby enabling us to assess the model’s generalisation capabilities.
Regarding the sexism identification task in memes, the main results presented in Table 10 (full results in the Appendix F) indicate that models incorporating multimodal inputs generally have better performance on the task. Efectively, since the creation of memes and their virality across communities is based on a strong correlation between textual and visual elements, the analysis of the textual content alone could result in partial or impractical comprehension of the content.
However, it is worth mentioning that zero-shot classicfiation of superimposed text using Qwen 2.5
VL 32B achieves results that are relatively close to the best ICM values obtained, while outperforming
other methods that leverage meme images in their pipeline. This suggests that, for this specific type of
memes, text plays a significant role in the final prediction. This may be explained by the fact that the
creators of the EXIST Meme dataset gathered images by curating a lexicon of 250 terms that were used
as search queries on Google Images. Additionally, textless images were removed manually, centring the
dataset on textual elements [
The fine-tuned Qwen 2.5 VL 32B model achieved the best results across all metrics, showing a +7.8% points improvement in the ICM-Hard Norm metric compared to the zero-shot classification performed using the of-the-shelf version of the same model.
To gain a clearer view of the results obtained by the best-performing method on subtask 2.1 Hard, we calculated the proportion of misclassified memes for which the annotators gave unanimous answers (e.g. all YES or all NO answers). Only 11.33% of misclassifications fall into this category, indicating that the model is highly confident in predictions for which there is a human agreement. Considering memes for which there is only one disagree answer, they account for 34.13% of misclassifications. More than half of the misclassifications (54.54%) come from memes for which two annotators disagree with the others., i.e. situations in which the evaluation of content is inherently more intricate from a human perspective.
Given the superior performance of fine-tuned Qwen 2.5 VL 32B on subtask 2.1, we adopted this method for subtasks 2.2 and 2.3. This decision allowed us to focus on the scope of the study and avoid redundant evaluations. Additionally, we reduced the number of experiments to minimise computational cost and environmental impact, striking a balance between empirical validation and responsible resource usage. Tables 11 and 12 show the results for source intention classification in memes. A thorough exploration into the sub-values of the F1 score indicates that the model demonstrates a high capacity in identifying memes that overtly promote sexist ideologies. Indeed, the relatively high F1 score for the DIRECT class indicates that this category of content is more easily identifiable by the model. Performance drops sharply for the JUDGEMENTAL class, as the low F1 score of 0.1413 suggests that the model has dificulty to identify contents that criticise sexism. This may be due to the complex nature of such memes, which often rely on sarcasm, as shown in Figure 7. Additionally, this degradation in performance may be correlated with the under-representation of this class, accounting for just 14.38% of all ground truths.
Similar considerations could be applied to the results obtained in the sexism categorisation task
displayed in Tables 13 and 14. Moderate F1 scores are observed among sexist categories for IDEOLOGICAL
INEQUALITY, STEREOTYPING DOMINANCE and OBJECTIFICATION (between 0.56 and 0.58), each of
which also appears in over a quarter of the ground truth data. However, the model struggles to identify
the categories MISOGYNY NON SEXUAL VIOLENCE and SEXUAL VIOLENCE, which represent 11.65%
and 14.18% of the ground truths, respectively. The observation that the two lowest F1 sub-values are
associated with these classes, jointly with the considerations made on the results of subtask 2.2, suggests
that a low statistical representation constitutes a strong learning limit for this model. In this field of
research, the relationship between the volume of data available and the precision of classification has
been already examined for other types of models [
The present section is dedicated to the presentation of results that have been obtained by our team in the EXIST 2025 challenge on the given test data.
We trained our tweet classification model with three diferent seeds (0, 1, and 42), resulting in three submissions: GrootWatch_1, GrootWatch_2, and GrootWatch_3. The performance of these models on the tweet test set is shown in Tables 15 to 17. Notably, our model consistently ranked first in the Soft-Soft category across all languages for subtasks 1.1, 1.2, and 1.3. In the more challenging Hard-Hard category, we always placed within the top 20 out of over 130 submissions.
Based on the results on memes for development dataset, we submitted our runs using the following methods: • GrootWatch_1: Zero-shot classification of the superimposed text with Qwen 2.5 VL 32B • GrootWatch_2: Zero-shot classification of the meme images with Qwen 2.5 VL 32B • GrootWatch_3: Classification with fine-tuned Qwen 2.5 VL 32B For subtasks 2.2 and 2.3, we used the fine-tuned Qwen 2.5 VL 32B model based on the YES predictions from the three distinct submissions on subtask 2.1. The results for meme classification in the hard evaluation setting are shown in Tables 18 to 20. Our methods demonstrated remarkable strength, with eight out of nine submissions achieving a top five ranking. The predictions obtained by fine-tuning Qwen 2.5 VL 32B consistently ranked second across all subtasks, achieving first place in subtask 2.1 on the Spanish instances.
Our sexism detection approach achieved state-of-the-art performance in Soft-Soft classification for
tweet analysis. The combination of contextual information search, annotator profile integration, soft
label learning, and multi-task architecture proved particularly efective in this category. However, the
Hard-Hard category remains a challenging task to overcome. Notably, our results revealed that simply
using the soft probabilities to infer the hard label is not a sucfiient strategy for tackling this challenge.
One potential avenue for future research lies in optimising the inference time for context retrieval with
AI agents. Currently, this process is relatively slow compared to traditional language models like LLM
or BERT. To address this limitation, a possible solution could be the development of a shared dictionary
or database of contexts that can be eficiently queried and retrieved. In cases where the desired context
is not already present in the database, the system could be designed to search for it online and then
store it in the database for future reference. This approach has the potential to significantly reduce
inference times, enabling more eficient and scalable AI-powered language understanding.
Furthermore, despite the promise of incorporating context into language models, our experiments
suggest that fine-tuning LLM with context actually degrades performance. A possible explanation for
this phenomenon is the concept of context hijacking [
With respect to the best results obtained on meme classification in the past edition of EXIST, which very mostly based on textual elements, the results obtained by our team in the current edition confirmed that a full integration of meme images into the classification pipeline leads to better performance. Despite the top-tier results achieved, the proposed approaches present some limitations: • Multi-task learning: Qwen 2.5 VL and Florence-2 were fine-tuned using the available ground truths for the three subtasks to minimise Cross-Entropy Loss. However, introducing a specific loss function that captures the interaction between subtasks could help the model to leverage the full potential of the given data and achieve better performance • Meme Dataset split: The dataset was split 80/20 for training and testing. Despite the significant computational time required for repeated VLM fine-tuning, future work may consider crossvalidation to obtain a more comprehensive assessment of model generalisation
Using optimal transport theory and the principle of maximum entropy, Erbani et al. [
The class NO has the lowest row and highest column values, indicating over-prediction that harms other classes. Notable confusions include MISOGYNY-NON-SEXUAL-VIOLENCE being misclassified as NO, and SEXUAL-VIOLENCE being confused with MISOGYNY-NON-SEXUALVIOLENCE, STEREOTYPING-DOMINANCE, or NO.
Future work could build on this analysis to reduce current misclassifications and enhance our method.
Experiments presented in this paper were carried out using the Grid’5000 testbed, supported by a scientific interest group hosted by Inria and including CNRS, RENATER and several Universities as well as other organisations (see https://www.grid5000.fr).
During the preparation of this work, the authors used GPT-4 and DeepL Write in order to: grammar and spelling corrections, rewriting of unnatural phrases, tone improving. After using these tools/services, the authors reviewed and edited the content as needed and take full responsibility for the publication’s content.
To achieve the results, we fine-tuned BERT with the following hyperparameters: • Learning rate: 1e-5 • Batch size: 64 • Weight decay: 0.05 • Number of epochs: 5 For LLM fine-tuning using LoRA, we employed the following configuration: • Batch size (training and evaluation): 32 • Gradient accumulation steps: 4 • Optimizer: PagedAdamW_8bit • Learning rate: 5e-5 • Precision format: bf16 • Warm-up ratio: 0.1 • LoRA’s matrix decomposition rank (r): 4 • Alpha LoRA: 16 • Targeted modules: self_attn.q_proj, self_attn.k_proj, self_attn.v_proj, self_attn.o_proj, mlp.gate_proj, mlp.up_proj, and mlp.down_proj The zero-shot and fine-tune prompts used in our experiments are:
Task: Classify tweets as YES (sexist) or NO (not sexist).
YES: Explicit sexism, descriptions of sexist situations, or criticism of sexism (even implied). NO: Neutral content. Ignore non-sexist vulgarity. Use societal context.
Answer: (Only YES or NO)
Tweet: {tweet}
Task: Determine whether a tweet is sexist. Categories: YES: The tweet is inherently sexist, describes a sexist situation, or criticises sexist behaviour. Examples: • “Women are too emotional to hold leadership positions.” • “At the meeting, all my ideas were ignored until a male colleague repeated them.” • “Catcalling is not a compliment; it’s harassment.” NO: The tweet does not contain sexist content, nor does it describe or criticise sexist situations or behaviours. Examples: • “Looking forward to the weekend!” • “Really looking forward to today’s ‘women in web3’ lunch meetup! If you’re in the la area and want to join, send me a dm!! See you ladies soon.” • “Wow! Trouble making witches unite!” Additional Guidelines: • Ambiguous Language: If the tweet’s sexism is implied rather than explicit, classify it as ’YES.’ If context is insuficient, classify it as ’NO.’ • Strong or Vulgar Language: Classify based on content relevance to sexism, not on the presence of strong language alone. • Contextual Understanding: Consider societal norms and the broader conversation when evaluating the tweet.
Your final answer will be YES or NO.
Tweet: {tweet}
Task: Retrieve concise external context to clarify ambiguous tweets or cultural references for sexism classification. Do NOT classify the tweet—only provide context that would help a downstream model to decide.
When to retrieve context: • The tweet references events, lyrics, memes, or cultural artefacts unfamiliar to a general audience. • The language is ambiguous (e.g., sarcasm, coded terms, or terms with dual meanings). • The tweet hints at a broader societal debate or news story.
Guidelines: 1. No classification: Never output YES/NO. Your role is purely contextual. 2. Conciseness: Summarise external context in ≤ 100 tokens. 3. Relevance: Only include context directly tied to potential sexism (e.g., explain a referenced event’s sexist controversy, not general info).
4. No context? Output ”No external context needed.” Output Format: [Summary of context, or ”No external context needed.”] Examples: 1. Tweet: ”Ugh, not another ‘Boss Babe’ anthem. . . ”
Output: ”The term ’Boss Babe’ is associated with MLM schemes targeting women, often criticised for exploiting feminist rhetoric. Some view it as empowering, others as patronising.” 2. Tweet: ”This is why we need more #NotAllMen energy.”
Output: ”#NotAllMen is a hashtag used to critique men who derail conversations about sexism by insisting ’not all men’ are problematic. Often cited in debates about systemic misogyny.” 3. Tweet: ”Finally got tickets to the concert!”
Output: ”No external context needed.
We conducted a preliminary assessment of the generated contexts to explore at their quality, relevance and accuracy. Our aim was to explore how well the generated contexts align with the original tweets. Methodology We randomly selected 30 context samples from each dataset (train, dev, test) and evaluated them based on three criteria: • Relevance: How well did the generated context align with the original tweet? (Score: 1-5) • Accuracy: Did the generated context provide correct information or insights? (Score: 1-5) • Quality: Was the generated context coherent, well-structured, and easy to understand? (Score: 1-5) • In case of ‘No external context needed.’: Was it appropriate not to generate external context for the given tweet? (Score: 1-5) Results The small-scale study reveals that the generated contexts consistently achieve perfect scores in terms of relevance (100%) and quality. The accuracy, however, is satisfactory but not outstanding, with an average score of 3.7/5. Notably, our model demonstrates 100% capability in identifying when no additional context is required. Neither did we observe any hallucinations in the generated texts. To delve deeper into context accuracy, we stratified the results according to the agreement rate of the six annotators on the binary sexist classification of the tweet (only applicable for training and development datasets, as test dataset results are not available). 100% 83% 66% 50% 3 4.4 4.3 4.5
As shown in Table 21, we observe that accuracy is less satisfactory when there is a high annotator agreement rate for subtask 1.1. However, with lower agreement rates, accuracy tends to improve. While this limited analysis provides an encouraging initial look at the generated contexts, we acknowledge that more samples and evaluators are necessary to draw more robust conclusions.
Task: Classify tweets as YES (sexist) or NO (not sexist).
YES: Explicit sexism, descriptions of sexist situations, or criticism of sexism (even implied). NO: Neutral content. Ignore non-sexist vulgarity. Use societal context. Tweet: {tweet}
Context: {context}
The full set of experiments conducted for subtask 2.1 Hard is showed in Table 22. With regard to multimodal models, it is interesting to note that an increase in model size does not necessarily lead to improved performance. For instance, zero-shot classification using models from the Qwen 2.5 VL family with the same prompt shows that Qwen 2.5 VL 32B outperforms the 7B release, while the 72B version performs worse than the medium-sized model. Interestingly, despite diferences in architecture and parameter count, the results obtained from Aya Vision 8B, Mistral Small 3.1 24B, and the fine-tuned Florence 2 model exhibit notable similarity. This convergence in performance suggests that these models have achieved a similar level of representational capability for sexism detection in memes. Fine-tuned Qwen 2.5 VL 7B has shown that an increase of +8.7% in the ICM-Hard Norm metric when compared to zero-shot classification with the same model. This percentage increase is comparable to the value computed for the 32B release.
"You are an Artificial Intelligence for sexism detection and classification in social media contents."
"Is this meme sexist towards women? Answer just YES if you judge the meme as sexist towards women. Answer just NO if you judge the meme as not sexist towards women."
"Given the superimposed text of a meme ’[superimposed_text]’, is this meme sexist towards women? Answer just YES if you judge the meme as sexist towards women. Answer just NO if you judge the meme as not sexist towards women."
"Answer just DIRECT if the meme promotes or encourages sexism. Answer just JUDGEMENTAL if the meme criticises or condemns sexist behaviour." Model XLM-RoBERTa base XLM-RoBERTa large XLM-RoBERTa base XLM-RoBERTa large
ICM-Hard Norm
F1(YES) "Classify the given meme into one or more of these categories (multi-label allowed): • IDEOLOGICAL-INEQUALITY if it rejects feminism or denies gender inequality. • STEREOTYPING-DOMINANCE if it promotes traditional gender roles or male superiority. • OBJECTIFICATION if it reduces women to appearance or sexualises them. • SEXUAL-VIOLENCE if it contains sexual harassment or assault references. • MISOGYNY-NON-SEXUAL-VIOLENCE if it expresses hatred or non-sexual violence toward women.
The answer is just and strictly a list of strings, as the following example:
"You are an Artificial Intelligence for meme captioning."
"Generate a caption in plain text of this meme without expressing a judgement on it. Answer in 80 words maximum."
"Generate a detailed caption in plain text of this meme without expressing a judgement on it."
On Florence-2, the experiments were conducted by freezing the DaViT vision encoder and using a batch size of 5. The training was conducted over 3 epochs using the AdamW optimiser with a linear learning rate scheduler and no warm up steps. The model was optimised to minimise the cross-entropy loss between predicted and target YES/NO labels, performing the validation after each epoch. For Qwen 2.5 VL 7B and 32B, due to a larger size of the models the fine-tuning strategy was diferent, in order to keep reasonable training times. We applied Low-Rank Adaptation (LoRA) to the query and value projection layers using a rank of 8, a scaling factor of 16, and a dropout rate of 0.05. Only the low-rank adaptor weights were updated during training, resulting in a significant reduction in the number of trainable parameters. For the 7B model, the trainable parameters were 2,523,136 (0.0304% of the total), while for the 32B model the number of trainable parameters was 8,388,608 (0.0251 % of the total). The models were fine-tuned on 3 epochs by scaling the image resolution up to 262,144 pixels, using a batch size of 5. As for Florence-2, the loss function to minimise was Cross Entropy Loss.