Due to the growing impact of social media, the necessity for automated mechanisms that can identify sexism and other forms of disrespectful and hateful conduct is rising, aiming to create a more inclusive and respectful digital space. However, it poses significant challenges due to the variety of hate categories and the complexity of interpreting the author's intent, particularly under the multilingual learning framework. This paper describes Team Aditya's participation in the EXIST (sEXism Identification in Social neTworks) Lab at CLEF 2024 . The proposed system makes use of large language models (i.e., Bertweet, mBERT and XLM-RoBERTa) for identifying sexism in English and Spanish language. This work describes our participation in EXIST task 1. Considering a hard evaluation, we obtained F1 score of 0.7691 using best epoch trained with XLM-Roberta. We are ranked 14th in the given task.
Sexism is prejudice or discrimination based on one’s sex or gender, often targeting women due to their gender. This harmful mindset causes inequality, limits opportunities, and reinforces oppressive power dynamics, limiting progress toward a fairer society.
The rise of social media platforms such as Twitter and Facebook has led to a significant change in communication methods. Identifying and reducing hate speech on these platforms can be a daunting task, due to large volumes of data generated. This requires using automated techniques and advanced technologies to eficiently process and classify the content.
The EXIST 2024 [
The dataset provided by the Exist2024 initiative consists of about 7K tweets, equally split between Spanish and English language. To mitigate label bias, organisers have considered two diferent social and demographic parameters: gender (MALE/FEMALE) and age (18-22 y.o./23-45 y.o./+46 y.o). The dataset was split into train, dev, and test sets, roughly distributed as 70%, 10%, and 20%, respectively, for both languages.
The labels for the tweets in Subtask 1 were categorized as "YES" or "NO" to indicate whether they conveyed a sexist meaning.
To derive the definitive hard label, we utilize the annotations by multiple annotators and apply majority voting. Only if 3 or more of the 6 annotators unanimously agree to a label YES, then the label is set to YES. Prior to training the model, preprocessing steps were carried out to remove any emoji’s, URLs and mentions from the samples. This was done to remove any undue bias.
We conducted experiments with various models and found XLM-R to be the most efective, particularly due to its strong performance on multilingual data, as shown in Table 2.
We finetune XLM-Roberta Large, a multilingual version of Roberta, trained on 2.5TB of filtered CommonCrawl data. This allows us to handle both English and Spanish samples by utilizing a single model.
The model was trained using contrastive learning, enabling it to diferentiate between samples efectively. It learns an embedding space where similar pairs are positioned in close proximity, while dissimilar pairs are distinctly separated.
To improve the representation of each example in a batch we created label-aware embeddings by
prefixing the text with its corresponding label [
We utilized three datasets during the finetuning process: The training dataset was used to learn an embedding space using contrastive learning. The validation dataset was used to retain the most efective checkpoint and the test dataset was used to evaluate performance on unseen data.
We submitted the following systems for evaluation, with each system being trained with the following hyperparameters: • Batch size: 16 • Learning rate: 1e-6 • Dropout: 0.35
These hyperparameters were chosen using the Optuna library. We selected the hyperparameters with the best performance on the validation set, after running 50 trials with varying configurations. These optimal hyperparameters help improve performance and reliability.
1. ADITYA1: The model was trained for 30 epochs, leveraging the combined train and dev datasets.
The optimal epoch was subsequently saved based on the system’s performance on the test set.
The saved model was used to make predictions on the unseen test set. 2. ADITYA2: The model was trained for 30 epochs on the train set. The optimal epoch was subsequently saved based on the system’s performance on the test set. The saved model was used to make predictions on the unseen test set. 3. ADITYA3: The model was trained for 12 epochs on the combined train and dev datasets. This model was used to make predictions on the unseen test set.
For the exist-Task1 we utilized three systems as described in Section 3. The evaluation metric used for these systems was the ICM Metric.
The ICM metric [
These labels were used to evaluate the models across three schemes, described as follows: • Hard-hard evaluation: the ICM similarity between the hard system output and hard ground truth • Soft-soft evaluation: the ICM similarity between the soft system output and the soft ground truth All three of our submitted systems generated Hard Labels, which were subsequently utilized for the Hard-hard evaluation scheme. A summary of our experiments is presented in Table 3.
The rise of social media has led to an increase in sexist content in society, necessitating the de- velopment
of automated systems to detect and counteract sexism. However, the discrepancy in the composition
of the tweets and the multilingual nature of the dataset cause problems . To address these problems,
we used pre-processing to improve the efectiveness of our system. To solve the sexism identification
challenge, contrastive learning with RoBERTa language model [
This paper presents the participation of team Aditya in the Task1 of the EXIST2024 [
Throughout our experimentation, we evaluated various models, including BerTweet, mBERT and T5. However, XLM-Roberta Large consistently demonstrated superior results compared to its counterparts and the best performance was obtained with its help.
There is tremendous scope for advancements and progress in this field. In future works, we would like to refine and enhance our approach with diferent contrastive learning strategies, to improve the model’s ability to distinguish between diferent classes. This approach can be also used in multiclass classification and multilabel classification problems. Large Language Models trained specifically on Spanish text could also be utilized to further improve performance. B. Vidgen, Z. Talat (Eds.), Proceedings of the Sixth Workshop on Online Abuse and Harms (WOAH), Association for Computational Linguistics, Seattle, Washington (Hybrid), 2022, pp. 230–230. URL: https://aclanthology.org/2022.woah-1.21. doi:10.18653/v1/2022.woah-1.21. [10] A. F. M. de Paula, R. F. da Silva, I. B. Schlicht, Sexism prediction in spanish and english tweets using monolingual and multilingual bert and ensemble models, 2021. arXiv:2111.04551.