This paper describes the approaches developed by the I2C Group to participate on sub-task 1 in the CLEF 2023 task EXIST: sEXism Identification in Social neTworks. Our main contribution is to show the benefits of translating a bilingual dataset to a single language, as well as the efectiveness of using a group of classifiers based on transformers architecture. By combining diferent models, the individual advantages were exploited, resulting in better performance than using a single model. Moreover, the importance of choosing suitable hyperparameters during the model training process was highlighted by the results. Through careful experimentation and evaluation of diferent hyperparameter combinations, the settings that achieved the best performance for the given task were found. In our experiments we ifne-tuned several pre-trained language models and decided to ensemble the three models that reached the best F1-scores. With this approach, we achieved an ICM-Hard score of 0.5075, ranking 25th in the competition.
role of women in our society [3]. Given the sate-of-the-art results Transformers have achieved in various natural language processing tasks[4], all the models we developed are based on this technology. Regarding our final results, we translated our dataset to only one language, trained three models and built an ensemble to improve the binary classifier performance [5].
Transformers are neural network architectures that rely on self-attention mechanisms to encode the semantic and syntactic information of natural language. They have achieved stateof-the-art results in various natural language processing tasks, such as machine translation, text classification, and sentiment analysis. We use diferent transformer models, such as BERT, RoBERTa, and XLM-RoBERTa, to encode the tweets and classify them into sexist or non-sexist categories. We also explore the use of multilingual models that can handle both languages simultaneously.
The following section reviews some relevant literature. Section 3 presents Task 1 and the Corpus provided by the organizers. Section 4 and 5 report the experimental methodology and evaluation results. Section 6 concludes the study and outlines some directions for future work.
Several studies have addressed the problem of sexism detection on social media using natural language processing and machine learning techniques. Most of them rely on deep neural networks, such as convolutional neural networks (CNNs) or recurrent neural networks (RNNs), to learn features from text and classify it as sexist or not. For example, in [6] A. Kalra and A. Zubiaga (2021) used CNNs and RNNs to detect sexism in tweets and gabs, a social network known for hosting extremist content. They also applied data augmentation and transfer learning with BERT and DistilBERT models to improve their performance.
However, deep neural networks require large amounts of labeled data to achieve good results, which is often scarce or imbalanced for sexism detection. Moreover, they may not capture the nuances and subtleties of sexist language, which can vary across cultures, contexts, and domains. Therefore, some recent studies have explored the use of transformer models, which are pre-trained on large corpora of text and can leverage contextual information and semantic representations. For example, in [7] M. Schütz et al (2021) used multilingual transformer models based on BERT and XLM-R to perform binary and multiclass sexism detection on tweets in Spanish and English. They also used unsupervised pre-training and fine-tuning with additional datasets to adapt the transformers to the task . Similarly, in [8] A. Gómez et al. (2021) used ensembles of transformer models trained on diferent background corpora and fine-tuned on the EXIST2021 dataset, which contains tweets and gabs labeled for sexism.
Transformer-based models have shown great potential in detecting hate messages, specially against women. Researchers have trained these models on large, annotated datasets and finetuned them for hate speech detection, achieving significant improvements in identifying and categorizing discriminatory content. Transformer models have been instrumental in capturing the complex linguistic features of hate speech, enabling more efective moderation of online platforms, the protection of vulnerable communities, and the creation of a safer and more inclusive digital environment [9].
This paper is focused on subtask 1: Binary classification. The corpus was provided by organizers, and contains three datasets, one for training, other for validation and the last one for testing. Training and validation datasets have 11 columns each, which are: id EXIST, lang, tweet, number annotators, annotators, gender annotators, age annotators, labels task1, labels task2, labels task3 and split. As test dataset is not labelled yet, it only has columns id EXIST, lang, tweet, and split. The columns we used for training our models in subtask 1 were: • id EXIST: Id of the tweet in the competition. • lang: Language of the tweet ("en" for English or "es" for Spanish). • tweet: Raw text of the tweet. • labels task 1: This field contains one label per annotator, indicating if the tweet is sexist or not. A small processing had to be done in order to achieve a binary label. The majority of labels in the list was elected as the definitive label (we labelled 0 for NO and 1 for YES), and in case of tie, the row was eliminated from the dataset.
Regarding datasets, training dataset consists of 6920 tweets, validation dataset 1038, and test 2076. Tables 1 and 2 show the distribution of the labels and languages for each dataset.
The methods used in this study involved several important steps. First, we converted the entire dataset into English and Spanish using the Google translator python library Googletrans A, to
compare how well multilingual and single-language classification models performed. Next, we searched for the best hyperparameters to train the models for this specific task. Lastly, we built a classification model by combining the three top models we found and using hard voting methods to improve the results.
The pre-trained models selected, obtained from the Hugging Face Transformers library B, were: • xml-roberta-base [10]: Multilingual version of RoBERTa. • bert-base-multilingual [11]: Multilingual version of BERT. • PlanTL-GOB-ES/roberta-base-bne [12]: RoBERTa base model pre-trained using the largest Spanish corpus known to date. • bert-base-uncased [11]: Base version of BERT, trained with 110 million parameters in
English language. • bert-large-uncased [11]: Advanced version of BERT, trained with 340 million parameters. • NLP-LTU/distilbert-sexism-detector [13]: Distilled version of BERT, with 40% less parameters but 95% of performance compared to BERT. This models has been fine-tuned previously with a sexism classification corpus.
To compare the results obtained by the diferent models and developed strategies, all models were fine-tuned using the training dataset, and their performance was measured with the validation dataset. Multilingual models used the datasets with tweets both in English and Spanish, as provided by the organizers. The roberta-base-bne model uses the datasets fully translated into Spanish, while the rest of the models use them fully translated into English.
The same small text preprocessing was done for all datasets in all languages. It consisted on removing links, usernames, numbers, words with length of one character and emojis. Hashtags (#) were preserved as they can be a key indicator of sexism in some tweets.
The hyperparameter optimization [14] is a crucial step for models fine-tuning. For this reason, multiple iterations of training and evaluation were performed using diferent combinations of the most significant Transformers hyperparameters. The platform used for this purpose was WandB (Weights & Biases) C, which provides a clear graphical interface for tracking and visualizing machine learning experiments. Table 3 shows the hyperparameters space used in this experimentation phase.
The optimal hyperparameters for each model are presented in Table 4. Using these values, we obtained the results shown in Table 5. These results demonstrate the benefits of conducting a proper hyperparameter search for fine-tuning, and using specific-language models instead of multilingual for this task.
Hyperparam xlm-roberta bert-multilang roberta-bne bert-base bert-large distilbert-sexism
The final output was determined by a hard voting technique [ 15], which selected the most frequent prediction among the models. This ensured a more reliable and robust prediction based on consensus. The ensemble [16] and model voting techniques improved the overall predictive performance by combining the strengths and diversity of multiple models, resulting in more precise and accurate predictions. The models included in the ensembles were the three best ones in terms of f1-scores, namely bert-base, bert-large, and distilbert-sexism.
Since the three models were pre-trained with English tweets, the dataset provided by the organizers was fully translated into English for the evaluation phase.
In this section, we present the final results submitted to the competition. The predictions were evaluated using the oficial competition metrics, specifically the ICM-Hard and F1-Score. Results were given for the full test dataset, only the Spanish tweets and only the English Tweets.
The final prediction was constructed using a voting scheme among the three models. The achieved ICM-Hard and F1-Score for this task including all languages was respectively 0.5075 and 0.7611, resulting in a 25th position out of 69 participants. Table 6 shows our ranking and results achieved in each language.
The results obtained demonstrate the efectiveness of our approach, and how translating all the datasets to English for training made us rank higher classifying tweets in this language.
In this paper, our approach for sEXism Identification in Social neTworks and the results obtained in subtask 1 for CLEF 2023 are presented. Our proposal consisted on fine-tuned transformerbased models using diferent approaches for each classifier to optimize the results. Six diferent models were fine-tuned using hyperparameter optimization, generating more that 300 diferent combinations. Finally an ensemble of the three best models was done using hard voting for binary classification achieving a ICM-Hard of 0.5075 and a f1-score of 0-7611, being ranked 25 out of 69 participants.
In future works we will apply data augmentation using backtranslation and other techniques as well as new ensembles approaches. Moreover, we will conduct a thorough hyperparameter search to train the models in order to enhance the detection of sexist messages on social media.
This paper is part of the I+D+i Project titled “Conspiracy Theories and hate speech online: Comparison of patterns in narratives and social networks about COVID-19, immigrants, refugees and LGBTI people [NONCONSPIRA-HATE!]”, PID2021-123983OB-I00, funded by MCIN/AEI/10.13039/501100011033/ and by “ERDF A way of making Europe”.