This paper presents Mumul03's submission to the EXIST 2025 shared task on sexism detection. We employ ModernBERT-large, used in this task for the first time, and incorporate demographic information from the annotator such as gender, ethnicity, age, and other attributes into the model input. By modeling individual annotator perspectives and aggregating predictions across submodels, our system efectively captures subjectivity in the annotation process. Our system achieved a ranking of 7th out of 64 submissions for Task 1 in the Soft-Soft category setting. This paper reports our findings on the classification of sexism within textual content on social media, ofering substantial insights for the EXIST 2025 challenge.
Sexism, defined as prejudice or discrimination based on sex or gender, often targets women and girls
through subtle and explicit expressions in everyday life [
In this context, NLP technologies have become vital tools in detecting and mitigating online sexism
[
As part of the EXIST 2025 evaluation, Task 1 is defined as a binary classification task that determines
whether a tweet—originally posted on Twitter (now rebranded as X)—contains sexist content, including
both explicit and implicit expressions of gender bias. Task 2 is a three-class classification task focused on
identifying the author’s intent, categorizing tweets into direct sexism, reported sexism, or judgmental
commentary [
To efectively address these tasks, it is essential to select a model capable of understanding subtle
linguistic cues. ModernBERT-large ofers an enhanced bidirectional attention mechanism and
contextaware encoding, enabling it to capture nuanced semantic signals such as sarcasm and implicit bias
[
However, relying on a single model to assess whether a statement is sexist introduces bias, as
such judgments are inherently subjective. Prior research shows that annotator backgrounds such
as gender and cultural context significantly influence their perceptions [
To address this, we adopt the Learning with Disagreement (LwD) paradigm [
The rest of this paper is organized as follows. Section 2 reviews related work; Section 3 details our system; Section 4 presents experiments and results; Section 5 concludes.
Recent advances in sexism detection have generally followed three strategic directions: multiview
ensembles, which aggregate predictions from multiple models or perspectives to improve robustness;
transformer fine-tuning, where large pretrained models are adapted to task-specific data using
techniques like data augmentation; and prompt-based large language models (LLMs), which use zero- or
few-shot in-context learning to perform sexism detection without task-specific fine-tuning. Each of
these approaches has shown strong empirical performance in benchmark evaluations such as EXIST[
Model ensembling enhances robustness, particularly in multilingual and cross-domain contexts. Methods
such as prediction averaging have been shown to improve F1 scores and reduce classification uncertainty
[
Transformer-based encoders are widely used for supervised sexism detection due to their contextual
understanding and multilingual adaptability. Commonly used models include XLM-RoBERTa,
DeBERTaV3, mBERT, and Spanish-specific variants like BETO and BERTIN. To improve generalization, teams
employed techniques such as soft-label learning, multi-label classification, and data augmentation. The
AlexPUPB team, for example, fine-tuned compact models like XLM-RoBERTa and MiniLMv2 using soft
labels derived from annotator distributions [
Prompt-based large language models (LLMs) have emerged as strong baselines in recent EXIST tasks
[
Recent studies have extended prompting by incorporating socio-demographic context. Magnossão
de Paula et al. prompted LLMs with demographic personas (e.g., “a male over 45”) and found that
model outputs tended to align more with female annotators by default, though alignment efects were
inconsistent [
In this study, we only participated in the English subset of Task 1 (sexism identification) and Task 2 (source intention classification) under the soft evaluation setting. Therefore, this section focuses solely on the portion of the dataset relevant to our experiments.
The English subset of the EXIST 2025 dataset consists of 4,727 tweets, which are annotated for various types of sexist expressions, including both explicit and reported sexism. The dataset is divided into a training set (3,260 tweets), a development set (489 tweets), and a test set (978 tweets).
For each sample, the following attributes are provided in a JSON format: • id_EXIST: a unique identifier for the tweet. • tweet: the text of the tweet. • number_annotators: the number of persons that have annotated the tweet. • annotators: a unique identifier for each of the annotators. • gender_annotators: the gender of the diferent annotators (“F” or “M”, for female and male respectively). • age_annotators: the age group of the diferent annotators (grouped in “18–22”, “23–45”, or “46+”). • labels_task1: a set of labels (one for each of the annotators) that indicate if the tweet contains sexist expressions or refers to sexist behaviors or not (“YES” or “NO”). • labels_task2: a set of labels (one for each of the annotators) recording the intention of the person who wrote the tweet (“DIRECT”, “REPORTED”, “JUDGEMENTAL”, “–”, and “UNKNOWN”).
The dataset is annotated by a diverse group of individuals in terms of gender, age, ethnicity, education level, and country of residence. This diversity contributes to the robustness and fairness of the annotations, helping the dataset to capture a wide range of perspectives and reduce potential annotation bias.
This evaluation targets systems that output probability distributions over categories instead of singleclass predictions. To efectively assess system performance in scenarios involving multiple and potentially conflicting annotations, ICM-Soft (a soft-label extension of the Information Contrast Measure) is adopted as the oficial evaluation metric. Additionally, results are also reported using its normalized form (ICM-Soft Norm) and Cross Entropy, providing complementary views on prediction quality. • ICM-Soft: a soft-label extension of the ICM metric, designed to compare the predicted probability distribution with the distribution of human annotations. It is particularly suitable for tasks where label disagreement or subjectivity is common, as it evaluates how closely the model’s predictions align with the collective opinions of annotators. • ICM-Soft Norm: a normalized version of ICM-Soft, which rescales the raw scores to facilitate comparison across diferent tasks or systems. • Cross Entropy: a widely used metric in classification tasks that measures the diference between the predicted and true probability distributions. Lower values indicate better alignment with the true distribution.
Our architecture, illustrated in Figure 1, consists of multiple models based on transformers independently trained. Each model receives a version of the same tweet augmented with the demographic information of a specific annotator. These models are trained using a unified configuration and are designed to capture diverse annotator perspectives. Their predictions are then aggregated to produce the final output. The following subsections provide a detailed breakdown of our system’s construction, including data preprocessing, metadata integration, model fine-tuning, and output aggregation.
Given that the dataset comprises tweets from X, the textual content is inherently informal and often includes elements such as hashtags, user mentions, emojis, and URLs. These noisy components can hinder model performance if not properly normalized. To improve the quality and consistency of the input data, we applied a series of preprocessing steps aimed at reducing noise and linguistic variability—transforming raw tweets into normalized, standardized versions more suitable for model input. Below, we outline the key preprocessing operations applied during this stage: 1. Hashtags were stripped of the “#” symbol to retain only the core word. 2. User mentions were replaced with the placeholder token user. 3. URLs were substituted with the token url. 4. Emojis were converted into their textual representations using the emoji Python package, enabling standardized tokenization and reducing encoding inconsistencies.
To further increase data diversity and robustness, we employed the AEDA (An Easier Data Augmentation) technique [22]. AEDA introduces controlled perturbations by randomly inserting punctuation marks (e.g., ., ;, ?, :, !, ,) into sentences. In this way, the normalized tweets are transformed into augmented versions that enrich the training data without altering semantic content, thereby promoting model generalization.
Each data sample is annotated by up to six annotators, with each annotator providing demographic metadata, including gender, age, ethnicity, education level, and country of residence. We split the dataset by annotator, resulting in six parallel subsets. Each subset contains the same tweet texts but is paired with metadata corresponding to a specific annotator, forming one-to-one tweet–annotator pairs.
To construct the input, we concatenate the original tweet with the annotator’s metadata using the [SEP] token as a separator between fields. This format allows both the tweet content and the associated demographic information to be included in a single input sequence. Figure 2 illustrates an example of this formatting.
This method integrates annotator information directly into the model input without requiring architectural modifications. Each tweet is thus associated with multiple instances, each reflecting a distinct annotator perspective.
To model the subjective perspectives of individual annotators, we fine-tune six model instances independently. While all sub-models share the same architecture and training configuration, each is trained on a distinct subset created by pairing tweets with the metadata of a specific annotator. Input sequences are formed by concatenating the tweet text with the annotator’s demographic information, ensuring structural consistency across samples while enabling the model to capture perspective-specicfi patterns.
During training, we use the standard cross-entropy loss function as the optimization objective, defined as: ℒCE = − ∑︁ log() =1 where denotes the one-hot encoded ground-truth label, and is the predicted probability for class . This loss function measures the discrepancy between prediction and truth, helping improve classification accuracy.
The final output is obtained by aggregating predictions from six independently fine-tuned models, each generating a discrete class label for a given input. We then aggregate these class predictions to construct a soft label—a probability distribution reflecting the relative frequency of each predicted class. To formalize this process, the probability ˆ of class is computed as:
ˆ = 1 ∑︁ I(() = )
=1 where denotes the number of models, () is the predicted label from the -th model, and I(· ) is the indicator function. This method captures both model consensus and uncertainty, aligning with the soft-soft evaluation protocol and efectively reflecting the subjective variation among annotators.
All experiments were conducted on a single NVIDIA A800 GPU (80GB memory). We utilized the HuggingFace transformers library for model training, with AdamW as the optimizer. The learning rate was set to 2e-5, and the weight decay was 0.1. Training was carried out for 8 epochs with a batch size of 16, and a linear warmup strategy was applied with 10% warmup ratio. The maximum input sequence length was set to 128. Mixed-precision training (fp16) was enabled to improve memory eficiency. Model evaluation was performed at the end of each epoch, and the checkpoint with the best validation performance was saved.
This section presents the results of our models during the development phase, under the soft-label evaluation setting. We assess three transformer-based architectures: ModernBERT-large, DeBERTaV3large, and XLM-RoBERTa-large. Each model was tested across four configurations: (1) the baseline model, (2) baseline + data augmentation (DA), (3) baseline + annotator metadata integration (AMI), and (4) baseline + data augmentation (DA) + annotator metadata integration (AMI). To ensure a fair comparison, Task 2 models used ground-truth labels from Task 1 for the hierarchical structure, instead of predictions, allowing us to isolate the efect of architecture and training strategy.
The results, as shown in Table 1, demonstrate that combining data preprocessing, data augmentation (DA), and annotator metadata integration (AMI) leads to consistent performance improvements across all models. In Task 1, XLM-RoBERTa-large improved from an ICM-Soft score of 0.5042 (baseline) to 0.7123 (+DA+AMI), and DeBERTaV3-large from 0.5187 to 0.7486. Similar gains were observed in Task 2, confirming the efectiveness of this strategy.
When examining the individual contributions of DA and AMI, we find that AMI yields stronger improvements when applied independently. For instance, with ModernBERT-large, the ICM-Soft score increased from 0.5619 (baseline) to 0.6937 using AMI, whereas DA achieved 0.6781—an approximate 13% relative improvement in favor of AMI. This suggests that annotator metadata contributes more directly to modeling subjectivity and improving the accuracy of learned label distributions.
Among all architectures, ModernBERT-large consistently achieved the best performance, with ICMSoft scores of 0.7839 in Task 1 and -3.2791 in Task 2. Its strong adaptability to diferent annotator perspectives further demonstrates its superior ability to model subjective label distributions and to address the complex classification challenges posed by sexism detection. These results validate our choice of using it as the backbone for the EXIST 2025 task.
To further understand the model’s behavior, we conducted an analysis of misclassified samples. We found that tweets with strong emotional tone or vulgar language were often misjudged as sexist, even when annotated as non-sexist. This suggests a bias in handling emotionally charged but gender-irrelevant content. Moreover, the model exhibited instability on samples with high annotator disagreement, reflecting the need for further improvement in handling highly subjective or semantically ambiguous texts.
Our final submission was based on the ModernBERT-large model, incorporating data augmentation and annotator metadata integration. In the final evaluation of EXIST 2025, our system ranked 7th out of 64 submissions for Task 1 in the Soft-Soft setting, and 20th out of 53 submissions for Task 2.
As illustrated in Tables 2 and 3, our proposed approach demonstrates competitive performance, particularly in Task 1, where it ranked 7th with an ICM-Soft score of 0.7135—substantially exceeding the baseline systems, which yielded scores of -2.1991 and -3.8158. While the system achieved a relatively lower rank of 20th in Task 2, it still outperformed the majority of baseline models in both ICM-Soft and cross-entropy metrics, reflecting its robustness and adaptability across tasks. These results underscore the efectiveness of the ModernBERT-large architecture in modeling annotator subjectivity. Overall, our method successfully captures the distributional nature of annotator disagreement under the SoftSoft evaluation paradigm, delivering high-fidelity probabilistic outputs without introducing additional architectural complexity, thereby highlighting its practicality and scalability for real-world deployment.
This paper presents our approach for Tasks 1 and 2 of the EXIST 2025 shared task. We employed the ModernBERT-large model, enhanced through data augmentation and the integration of annotator demographic metadata to better model subjectivity in sexism detection. Our system demonstrated strong performance, ranking 7th in Task 1 under the Soft-Soft evaluation setting. These results highlight the efectiveness and practicality of our method in capturing annotator disagreement and addressing the nuanced nature of social bias in language.
This work is supported by the Quality Engineering Projects for Teaching Quality and Teaching Reform in Undergraduate Colleges and Universities of Guangdong Province (No.20251067).
During the preparation of this work, the author(s) used GPT-o3 in order to: Grammar and spelling check. After using these tool(s)/service(s), the author(s) reviewed and edited the content as needed and take(s) full responsibility for the publication’s content. [21] A. Jiang, N. Vitsakis, T. Dinkar, G. Abercrombie, I. Konstas, Re-examining sexism and misogyny classification with annotator attitudes, in: Y. Al-Onaizan, M. Bansal, Y.-N. Chen (Eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, Association for Computational Linguistics, Miami, Florida, USA, 2024, pp. 15103–15125. URL: https://aclanthology.org/2024. ifndings-emnlp.887/. doi: 10.18653/v1/2024.findings-emnlp.887. [22] A. Karimi, L. Rossi, A. Prati, Aeda: An easier data augmentation technique for text classification, 2021. URL: https://arxiv.org/abs/2108.13230. arXiv:2108.13230.