We present our approach to EXIST (sEXism Identification in Social neTworks) Task 1, at CLEF 2023, comprising automatic sexism detection in both English and Spanish on Twitter data. We compare two methods, the first being a bi-ensemble method that combines two pre-trained BERT architectures, BETO and RoBERTa, for each specific language, Spanish and English, respectively. The second method utilizes the larger multilingual transformer, RoBERTa-XLM-base, and considers the entire dataset despite language diferences. We show that the language-specific ensemble performs better than the generalized model and is a better choice when looking at sexism detection in mixed Spanish and English data.
violence against women” with “the overall prevalence rate by region at 76% in North America
and 91% in Latin America and Caribbean” [
From outwardly explicit misogyny to more subtle implicit misogyny, automatic classification of sexism will create a more eficient process of creating a safer environment online. With rising hate speech towards women online and little research into the detection of sexism, contributions to this issue are urgent.
We present the approach by team ZaRa-IU-NLP. This approach was developed during a course on machine learning in NLP. Our work focuses on comparing a multilingual neural model with an ensemble of language specific models.
The remainder of this report will proceed as follows: Section 2 outlines previous research for prior EXIST shared tasks, Sections 3 and 4 provide details on the task description, the dataset, and the preprocessing methods, respectively. Section 5 describes the two rival model architectures. In section 6 we analyze the results of the experiments before concluding in section 7.
Team avacaondata [
Team CIMATCOLMEX ranked first in the evaluation of Spanish tweets, with an accuracy of
0.7801 [
EXIST provides an opportunity to address the issue of online sexism with a wider reach, by providing participants with data containing bilingual, sexist speech, in both Spanish and English. Task 1 focuses on a binary classification of whether the text is sexist or not for both explicit or implicit examples. The data is constructed out of tweets with any form of oppression or prejudice against women because of their sex, explicit or implicit. It is worth noting that while the dataset labels every tweet as either English or Spanish, it includes tweets mixing both of the languages. Data
English Spanish
RoBERTa Tokenizer
BETO Tokenizer
RoBERTa
BETO
The EXIST 2023 dataset consists of 6 920 tweets for training, 1 038 tweets for validation, and 2 076 tweets for testing, where all sets are randomly selected from the 9 000 and 4 000 sets sampled and created by the CLEF 2023 organizers, for balance. The annotation process was carried out by a balanced group of 3 women and 3 men, in order to avoid gender bias. When choosing the label for a given tweet from these 6 labels, we used a simple majority. In the case of ties, we randomly selected either of YES or NO.
Our focus is on a comparison of two model architectures, a generalized model, processing both English and Spanish tweets, and a specialized model using language specific classifiers for each language. As such, preprocessing was kept to a minimum. Tweets were used with URLs, emojis, duplications of characters, etc in place. Further, the dataset included a set of tweets containing varying levels of both languages. No alterations were made to the content of these individual bilingual tweets. Rather, each architecture was given an opportunity to make judgement on tweets based on the language information provided by the shared task. I.e., tweets labeled English were passed through RoBERTa and tweets labeled Spanish were passed through BETO. The fluidity of language, specifically in the informal context of twitter, creates ample opportunity for bilingual users to hide implicit sexism through language switching and as such, should be included in each model’s ability to detect implicit bias.
In our work, we compare two diferent approaches, a multilingual model, and a language specific model. The multilingual model uses XLM-RoBERTa, while the language specific model, RoBERTa for English, and BETO for Spanish. Figure 1 shows an overview of the data flow in the two models.
One of the first approach to sexism detection in a Spanish-English context was using
XLMRoBERTa [
We also use this model for our multilingual approach and finetune it on the EXIST dataset on the binary classification task. The data is tokenized using XLM-RoBERTa’s tokenizer before passing into the model that was finetuned in 3 epochs. The finetuned model is used for inference.
This model is a pipeline composed of two models pre-trained on English and Spanish texts respectively. For English, we chose RoBERTa, a model trained on English texts using an MLM objective. RoBERTa is an improved model over the core BERT architecture with more parameters and trained on a larger corpus to provide a more robust performance than the base BERT model.
We chose BETO [
We fine-tuned both these models using the data for their respective languages. Since the data was split fairly evenly between English and Spanish texts, RoBERTa and BETO had about half the training data to finetune in comparison to XLM-RoBERTa.
For this ensemble, we performed tokenization based on the specific language using the pre-trained tokenizer. The data is then passed into the corresponding model.
We participated in the HARD-HARD evaluation, i.e., we provided a single label per tweet, which
was evaluated against the gold label. For the oficial evaluation on the test set, we report ICM
and F1 for the positive class (sexist). For our internal results on the development set, we report
the macro-averaged F1 score, the F1 score per class, and ICM. ICM is a score developed to
measure the similarity between two datapoints more accurately [
Table 1 shows the oficial scores on the test set. These results show the ICM metric scores and the F1 scores of the positive class for the two models in the HARD-HARD evaluation ZaRaIU-NLP_1 is based on the multilingual XLM-RoBERTa, and ZaRa-IU-NLP_2 on a combination of RoBERTa and BETO. Our scores show that the combination of language specific models outperforms the generalized model in each metric, even though the individual language models were trained on only half the data. The models produce consistent results across languages for sexist tweets, with the language specific models producing an F1 score of 0.7332 for Spanish tweets and 0.7263 for English tweets, while the single, large model reached 0.6956 and 0.6954, respectively.
Table 2 compares the performance of our two models on the validation set, including language specific results for both architectures. The results are in line with the oficial results, ZaRa-IUNLP_2, the combination of language specific models, outperforms the multilingual ZaRa-IUNLP_1 with an ICM of 0.4444 as compared to 0.3168. This shows that it is possible to obtain solid results given a small set of data and that the quality of the data (wrt. the language) is more important than the size of the training data. A look at the F1-scores per class shows a balanced performance, the score for the sexist class is only slightly lower than the one for the majority class of non-sexist tweets.
We then had a closer look at precision and recall per language and per class. These results are shown in Table 3. These results show that the multilingual model ZaRa-IU-NLP_1 shows the same preference for the non-sexist class given the higher recall for this class. However, this trend is much more pronounced for Spanish where the recall for non-sexist reaches 81.08% while the recall for the sexist class is at 63.24%. In the combined model ZaRa-IU-NLP_2, the Spanish classifier has a preference for the non-sexist class (recall: 85.81) while the English classifier has a preference for the sexist class (recall: 86.08). These results suggest that we might be able to gain better performance for Spanish if we upsample the sexist class. We leave this for future research.
Overall, the study addresses the dificulties of online sexism detection in social networks in a bilingual setting. We carried out a comparison between two distinct architectures: the first, a large, multilingual model that processes the entire data set, and the second, a combination of language specific, smaller models that implement a split in language categorization for processing. Given our minimal pre-processing, the higher accuracy of the language specific model can be attributed to the more specialized language models. Our results, when tested on the developmental set, show that the combination of two specialized models outperforms the single, generalized one by a diference of 3 percent points in macro-averaged F1 score, and a diference in ICM of 0.13. Overall, this study contributes to the process of automatic sexism detection in social networks in highlighting the greater eficiency and accuracy of two specialized models, rather than a multi-lingual model.
As described above, this system was developed as a project in a course on machine learning. For this reason, the system was intentionally kept simple, so that it could be carried out in a short amount of time. This is the reason, for example, why we focused on comparing the two systems without preprocessing the data, and without investigating other language models, etc.