This document presents in detail the work done for the sexism detection task at EXIST2021 workshop. Our methodology is built on ensembles of Transformer-based models which are trained on di erent background and corpora and ne-tuned on the provided dataset from the EXIST2021 workshop. We report accuracy of 0.767 for the binary classi cation task (task1), and f1 score 0.766, and for the multi-class task (task2) accuracy 0.623 and f1-score 0.535.
EXIST workshop (sEXism Identi cation in Social neTworks) is the rst shared
task at IberLEF 2021 [
As part of the workshop, an annotated dataset [
The training dataset consists of 6977 texts, 3436 in English and 3541 in Spanish. In the training set, all of the texts have Twitter as their source. The training dataset has been labelled with two separate label sets, which are essentially the workshop tasks: rst, a higher level binary annotation per text, indicating whether the particular text is sexist or not, and a second layer of annotation per text, where if a particular text is identi ed as sexist, it is also assigned one of the following labels : fIDEOLOGICAL AND INEQUALITY, STEREOTYPING AND DOMINANCE, OBJECTIFICATION, SEXUAL VIOLENCE, MISOGYNY AND NON-SEXUAL VIOLENCEg.
For the rst task, the dataset contains 3377 texts labelled as sexist and 3600 labelled as non-sexist, so it is rather balanced. For the second task, the distribution of the tweets labelled as sexist in their subcategories is as shown in Table 1:
The test dataset consists of 4368 texts, 2208 in English and 2160 in Spanish. In this dataset 3386 of the texts are sourced from Twitter and 982 from the social network Gab. 2
We combine two major approaches in our design: ne-tuning separate BERT [
We ne-tune six separate models starting from di erent weight con gurations, three for English and three for Spanish.
During testing, we report the majority vote of the ensembles, e.g. if more
than one of the classi ers agree on a prediction, we report that prediction as the
nal decision. Our work is following the reasoning and the results of [
In the training process, we ne-tune the BERT models [
For English texts we use the pre-trained model from [
We use PyTorch [
Before feeding the texts to the classi ers, we pre-process them by
replacing mentions with the mention token and URLs by the URL token. While
theoretically user mentions and handles can implicitly capture social graph
structure and potentially increase the classi er performance [
We rst lter the input training set based on the language indicator and then
seed three neural networks with di erent initial random weights. We train the
networks for 10 epochs and select the one with the best accuracy as the nal
model per training. In testing, we feed the same text to all three neural networks
and report the majority vote of the classi ers as the nal classi cation. The nal
output will be classi ed as sexist if at least two of the three classi ers report it
as sexist and similarly, a text will be classi ed as non-sexist if at least two out of
three classi ers report it as non-sexist. Based on our experiments this tends to
give a 2% increase in the overall reported accuracy. We use Adam optimiser
for the training [
For the second task, we train a classi er to distinguish between sexist tweets. That is, we only keep the texts in the training dataset that have been agged as sexist and we train a multi-class model for these texts.
Again, we train di erent models for Spanish and English language texts. Since we have the language labels of the texts there is no need to attempt to determine the language of a text; however, in the general case it would be straightforward to add one more step in the pipeline for language detection using an o the shelf language detection library, such as langdetect (https://pypi.org/project/langdetect/).
In testing time, we rst apply the classi er of task1 to detect whether a tweet is sexist or not. If the text is labelled as non-sexist we report this label for task2 as well. If the task1 model reports sexist as the label, we feed this tweet to the second classi er to obtain a prediction for the sexism categorisation label as described above. 3
For the rst task, we achieve accuracy of 0.767, with f-score (macro) of 0.766 (10th placement in the ranking). For the second task, accuracy is at 0.623 and f1-score at 0.535 (19th placement in the ranking).
Breaking down the nal results by language, for task1 the accuracy in English texts is at 0.7445 and for Spanish texts at 0.789. For task2, the accuracy in English texts is 0.583 with f1-score 0.493, whereas in Spanish texts accuracy is at 0.664 and f1-score at 0.575.
Overall, the system tends to perform better in Spanish, and this is most probably due to the fact that the underlying BERT model for Spanish (BETO) is trained exclusively in Spanish texts.
Whereas we are using ensembles in the rst task, we decided not to adopt
this strategy for the multiclass classi cation for task2, as it would have increased
signi cantly training and testing time as well as computational cost. Instead,
we are using the output of the rst task and we only train one classi er per
language for the second task. It is worth noting here that the observation from
[
It is interesting to see that our models fail to correctly identify texts that have been labelled as non-sexist due to the presence of words that commonly appear in sexist environments. It also fails to correctly categorise sexist tweets that appear in the same context with non-sexist words (for example, the word friend ), short texts or even sometimes subtle or contextual use of sexists language.
For task2, the predictions are inheriting the error from the task1 classi er, e.g. falsely reporting sexist tweets, which accumulates with the error of the second model. The confusion matrices for the 2 tasks are shown in gure 1.
One question we aim to investigate is what is the optimal number of models in the ensemble from a classi cation accuracy perspective. This is a more general question and direction to investigate for Transformer based models for text classi cation.
Additionally, most recently it has been shown that Convolutional Neural
Networks can over-perform Transform-based architectures in Natural Language
Processing tasks [