Sexism in social networks has been increasingly present since their adoption among the connected population, partially due to the disinhibition of individuals when expressing their opinions in social networks and the lack of regulation nor control. Sexism permeates many layers of our society such as media, education or the work environment, and involves many aspects of our society such as gender equality, social respect, gender discrimantion, human rights, etc. In order to ght against this phenomenon, the rst step is detecting it properly and in all its varieties. For that purpose, we present our candidate systems for participation in the EXIST challenge (task 1) at IberLEF 2021 for sexism identi cation. We explore an approach based on the use of pre-trained transformers, such as BERT and roBERTa, in similar tasks and compare them to traditional Machine Learning approaches, such as SVM, Logistic Regression, SGD-based classi er and XGBoost. We achieve our best results of F1=72.4% for the multi-language binary classi cation task for our system combining transformer-based and traditional models with a majority vote mechanism, positioned #32 in the challenge's ranking, with a F1 di erence of 6% against the best result.
Sexism [
One of the extreme form of sexism is misogyny, the hatred of women. Societies
with high levels of misogyny present high rates of domestic violence, rape and
commodi cation of women and their bodies, and they are seen as property or
second-class citizens [
The automatic identi cation of sexism in a broad sense may help to create, design and promote the evolution of new equality policies, as well as encourage more adequate behaviors in our society.
The aim of our work is to develop a multi-language binary classi cation system for sexism in the context of our participation in the EXIST 2021 challenge (task 1). In the scope of the challenge, participants are asked to classify tweets and gabs (in English and Spanish languages) as sexist or non-sexist. For such purpose, we have designed several binary classi cation systems, combining stateof-the-art transformer-based models with traditional ones for both languages, Spanish and English, and benchmarked them to select the best candidates for our participation in the challenge. We found our best performing system (referred as Task1 MessGroupELL 3, ranked as #32 in the challenge results) to be our proposal of majority vote of two traditional models and a transformer-based model. These results, compared to the obtained with only traditional approaches or only transformer-based approaches, showed that di erent features are being complementary captured by di erent systems, and that using pre-trained transformers models ne-tuned only with the EXIST dataset do not achieve considerable improvements compared to traditional Machine Learning approaches. 2
Our proposal is focused on addressing the task separately by language. This is based on the fact that the state-of-the-art of techniques and available resources and datasets for this matter are quite di erent for English and Spanish. Our assumption is that we could achieve good performance results if we attempt to use these elements from the state-of-the-art separately. Additionally, we also consider that sexism could have strong language-dependent (and culture-dependent) characteristics that might be undermined considering a single multi-language approach and better explored separately.
Our purpose is to explore a data-driven approach more than a feature-driven one. In this direction, we attempt to use existing related datasets to support and enrich our systems, alongside with pre-trained transformers that can be trained further in our task, but than can already give us an initial advantage. The comparison of performances between traditional methods and transformers is especially interesting within our goals. We assume that our approaches can be further extended and improved combining our proposal with a feature-driven one, due to the speci c linguistic characteristics of sexist language.
For both languages, we have considered a set of common traditional Machine Learning models based on tf-idf characteristics for benchmarking purposes. 2.1
For the case of the English subset, we explore pre-trained transformers in other
tasks and datasets. We have considered the following pre-trained transformer
models to train in our task:
1. BERT [
previous roBERTa model and ne-tuned for hate speech with the TweetEval
benchmark [
based on the previous roBERTa model and ne-tuned for o ensive language
with the TweetEval benchmark [
The models (3) and (4) correspond to two di erent tasks (Hate-Speech Detection and O ensive-Language Identi cation) that we consider interesting and potentially relevant for the detection of sexism or, at least, some aspects of it. As mentioned, sexism is not found as a single uni ed behavior, but as a wide range of behaviors that might include from a subtle joke to a very aggressive comment.
We regard sexism as partially containing both aspects, hate-speech and o ensivelanguage characteristics, and that the previous training of these models could add a cross-task knowledge to our training. Alternatively, we might expect that both pre-trained models are downstream models of the original model (2) and could limit the performance of a new training, compared to the use of model (2) instead. 2.2
For Spanish language, we combine our available training data from the EXIST
Task in Spanish, with the MeTwo dataset presented in [
3.1
The texts are processed for both languages in the same manner, using the spaCy
[
We have split the EXIST [
The training and validation sub-datasets have been extracted from the original non-extended datasets, using the holdout method with an 80%-20% split for training and validation, randomly selected and preserving the original class proportions. In the case of Spanish extended dataset, the MeTwo instances have been added only to the training dataset for comparability purposes. 3.3
We consider a Support Vector Machine (SVM) implementation as baseline and
complementary traditional Machine Learning models for benchmarking
purposes, all using TF-IDF features. These models include Logistic Regression
(LogReg), XGBoost [
We have used our training datasets to ne-tune the pre-trained models mentioned in Section 2.1 for English and in Section 2.2 for Spanish. We have performed the training on GPU, with a con guration of 10 train epochs at learning rate 2 10 5, cross-entropy loss function and early stopping at 5 epochs, to prevent over tting. 3.5
English Language. Considering the results presented in Table 1 for English language on the validation dataset, we can observe that transformer-based approaches perform at least 2% better than traditional approaches in macroaverage and up to 5% better comparing the best two performances of transformerbased (Roberta) and traditional models (SGD or XGBoost). For the class sexist, the di erence is ever more pronounced, with almost 10% improvement in some cases. R
P
R
P
R
P
It is noticeable that the pre-trained models on other tasks such as hatespeech and o ensive-language don't provide us a clear advantage against the original Roberta. Although the results are quite similar and comparable, it seems plausible that we decrease generalizability using both cross-tasks approaches in the current con guration. This might be due to the heterogeneous composition of the EXIST dataset regarding the di erent forms of sexism (misogyny, o ensivelanguage, subtle sexism, etc.) that are not necessarily mutually inclusive.
Regarding the performance of traditional models, we can observe their di culties to identify correctly the sexist class, compared to the non-sexist against the performance of transformer-based models.
Based on these results, we select Roberta as our one of our candidates for English language for its higher values in almost every aspect, compared to the other candidates.
Nevertheless, the good performance of traditional models to detect the nonsexist class and the risk of over tting while performing ne-tuning in our transformerbased models, make us consider a combination of both approaches (Roberta and traditional models) using a majority vote mechanism as a good candidate to submit, due to its potential better generalizing capabilities. We can observe that, in macro-average, this approach does not achieve better results than other options, but it o ers a considerable well-balanced trade-o for both classes in precision and recall.
Spanish Language Reviewing the performance obtained for Spanish Language in Table 2 on the validation dataset, we can observe that the BERT-based approaches provide more positive overall results compared to traditional approaches. However, the baseline SVM provides the best F1 and recall for the non-sexist class in exchange for very poor results on F1 and recall for the sexist class. It seems obvious that our best candidates for the task are the BERT-based options. Speci cally, we choose the (+MeTwo) BERT approach, since the overall results are better and the per-class trade-o s precision/recall are more balanced than in the original BERT approach. In addition, the inclusion of the MeTwo instances in the training can improve the generalizability of our system.
R
P
R
P
R
P
For the prediction of the EXIST's test dataset, we have split the instances by language, preprocessed them with their respective language-dependent preprocessing steps and evaluated them through the selected systems.
As speci ed by the challenge's organizers, we can submit three runs of our candidate systems on the test dataset. Our strategy to select the candidate con gurations consists on using the best Spanish candidate and di erent options for English to observe how the contributions of one or another candidate impact the performance. The selected con gurations are described in Table 3.
We evaluate our run #1 as our transfomer-based approach representative, with the best candidates from both languages. Secondly, we evaluate #2 as a baseline approach, combining one of the best results from the traditional models, XGBoost, prioritizing a balanced precision/recall trade-o for the sexist class. Finally, we combine our best transformer approach with the two best traditional approaches (SGD and XGBoost) through a majority vote, expected to provide a good balance for the non-sexist and capture information not captured by the transformer approach. The results of the evaluation are shown in Table 4 and present, as expected, the best candidate to be #3 (the majority vote between the best transformer approach and traditional approaches), indicating di erent and complementary feature learning capabilities. The least performing candidate is, as expected, the traditional approach #2, o ering similar but slightly less optimal results than those obtained in the validation dataset. And nally, our intermediate results comes from #1 the combination of our best performing approaches for both languages. As anticipated, the use of our transformers alone in our current con guration has been proven not to be completely e ective on its own. We have presented as candidates for the challenge three systems, described in Table 3, representing a baseline system (run #1), a pure transformer-based system (run #2) and a voted traditional/transformer-based system (run #3) which have resulted in F 1 = :7237 for the best performing case (position #32 in the challenge's ranking): our run #3, i.e, the majority vote of two traditional models (XGBoost and SGD) and a transformer-based model (Roberta pre-trained on English tweets). Comparing to the best result in the challenge F 1 = :7802, we observe a di erence of 6% (only 3% for English language), which we consider a very positive outcome. On the other hand, our performance in Spanish language has been shown relatively low compared to the best result in the challenge (8% lower).
With a very similar performance but slightly lower, we nd our run #1 (transformer-based) in the next position of the ranking with F 1 = :7225. This 1% di erence in F1-performance relies on a more balanced precision/recall tradeo for the English language examples, showing that the majority vote helps to capture more relevant cases without losing much precision.
Finally, our lowest performing system, run #2 (#39 of the ranking) has shown acceptable results considering its position in the ranking, and regarding that we have used it as a baseline for English language and the same con guration for Spanish language to evaluate di erences between our other two candidates. These results are fundamentally based on the Spanish language performance, and show that it has performed considerably well compared to other candidates in the ranking.
In every case, we have presented a transformer-based model for the Spanish language, since our experiments have shown a noticeable di erence between this approach and traditional models (Table 2, especially for the sexist class, in which traditional approaches present signi cantly poorer results. For that matter, it is worth noticing that the use of the existing dataset MeTwo for sexism identi cation for Spanish language has been proven to be an improvement in the performance of every approach for the Spanish language (Table 2).
However, we observe a gap between English and Spanish languages
performances for our best result compared to the challenge's top result, that might be
due to the fact that we have ne-tuned a BERT model pre-trained in Spanish
texts [
In this work, we have explored the capabilities of the use of mono-lingual
pretrained transformers as a basis for training a binary classi cation task on sexism,
in the scope of our participation in the EXIST (task 1) challenge of IberLEF [18].
We have explored di erent state-of-the-art options for both English and Spanish
languages, separately, allowing us to test di erent modular con gurations for the
task. Di erent pre-trained transformers have been tested, contrasting the
performance of those trained in di erent, but related, tasks to our current task. This
has been shown not to be as well-performing as base pre-trained transformers
such as Twitter-roBERTa-base [
The results of the evaluation have shown that even well-performing transformers models can be complemented by traditional approaches, in this case, through a simple majority vote. In this case, the traditional approaches can achieve in overall a better recall regarding the secondary class (non-sexist ), whereas transformer-based approaches achieve better precision for the same class.
The presented proposal and methodology have been proven to be adequate to produce generalizable systems for this task, considering the similarity between performance seen in the training/validation work ow and performance on evaluation, with di erences for F1 of 4% for English and 1% for Spanish.
This work was supported by the Spanish Ministry of Science and Innovation under Project Misinformation and Miscommunication in Social Media (PGC2018096212-B-C32). 18. Manuel Montes, Paolo Rosso, Julio Gonzalo, Ezra Aragon, Rodrigo Agerri, Miguel Angel Alvarez-Carmona, Elena Alvarez Mellado, Jorge Carrillo-de-Albornoz, Luis Chiruzzo, Larissa Freitas, Helena Gomez Adorno, Yoan Gutierrez, Salud Mar a Jimenez Zafra, Salvador Lima, Flor Miriam Plaza-de-Arco and Mariona Taule (eds.): Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2021), CEUR Workshop Proceedings, 2021.