Sexism has become an increasingly signi cant problem on social networks in recent years. The rst shared task on sEXism Identi cation in Social neTworks (EXIST) at IberLEF 2021 is an international competition in the eld of Natural Language Processing (NLP) with the aim to automatically identify sexism in social media content by applying machine learning methods. Thereby sexism detection is formulated as a coarse (binary) classi cation problem and a ne-grained classi cation task that distinguishes multiple types of sexist content (e.g., dominance, stereotyping, and objecti cation). This paper presents the contribution of the AIT FHSTP team at the EXIST2021 benchmark for both tasks. To solve the task,s we applied two multilingual transformer models, one based on multilingual BERT and one based on XLM-R. Our approach uses two di erent strategies to adapt the transformers to the detection of sexist content: rst, unsupervised pre-training with additional data and second, supervised ne-tuning with additional and augmented data. For both tasks our best model is XLM-R with unsupervised pre-training on the EXIST data and additional datasets and ne-tuning on the provided dataset. The best run for the binary classi cation (task 1) achieves a macro F1-score of 0.7752 and scores 5th rank in the benchmark; for the multiclass classi cation (task 2) our best submission scores 6th rank with a macro F1-score of 0.5589.
Discriminatory views against women are a common occurrence in the online environment. It's detection is challenging since sexism and misogyny may appear in di erent forms. The rst shared task on sEXism Identi cation in Social neTworks (EXIST) at IberLEF 2021 [12, 9] represents a systematic benchmark that attempts to tackle this challenge via machine learning and natural language understanding (NLU). The benchmark covers a wide spectrum of sexist content and aims to di erentiate di erent types of sexist content. This paper presents our contribution to the benchmark, describes our overall approach, the methods and models applied and summarises the obtained results. We summarise our results for both tasks: the binary sexism identi cation task (task 1) and the sexism categorization task (task 2). The EXIST benchmark incorporates English and Spanish content from Twitter and Gab which we account for by multilingual modeling. The peculiarity and contribution of our approach is the use of comprehensive data augmentation and the integration of external (unlabeled) data to make the classi cation models more robust.
Our paper is structured as the following: Section 2 describes our methodological approach, describing the employed datasets and models. Our experimental setup will be explained in Section 3 of this paper, followed by a documentation of the results (Section 4) and discussion and nal conclusion (Section 5). 2
A core challenge of the EXIST benchmark is the rather small size of the
provided dataset (approx. 7000 training instances). This rather small size makes
the robust training of complex NLP methods like transformers di cult. For
this reason, we approach the challenge with di erent transfer learning
strategies. As a basis for modeling the textual content, we apply two pre-trained
multilingual transformer models: mBERT [15] and XLM-R [
{ Pre-Training Strategy: Massively parametrised models such as transformers tend to over t on small datasets [13]. To overcome this issue pre-trained models are applied. In our experiments we evaluate di erent variants of pretrained transformers and further pre-train them in an unsupervised fashion on semantically related datasets. { Fine-Tuning Strategy: When using pre-trained models, it is necessary to adapt the models to the underlying task. For this purpose, either all layers or only the upper layers of the model are ne-tuned to the task-speci c data. Our aim here is to make the higher level feature representations in the model sensitive to the speci c task. { Fusion Strategy: As a third strategy we fuse predictions of the best models obtained by the previous two strategies to achieve a prediction.
A more detailed description of the implementation of these two strategies is provided in Section 3 and the results obtained with these approaches are presented in Section 4. 2.1
The challenge contribution is based on the EXIST2021 dataset which was provided by the EXIST2021 challenge [12]. The dataset contains 6977 training instances in English and Spanish. In total there are 3426 English and 3541 Spanish social media postings from Twitter and Gap. The test set contains 4368 instances, split into 2208 English and 2160 Spanish postings from mentioned sources. They are annotated in a binary fashion (task 1) as either sexist or non-sexist ; and in a more ne-grained categorization (task 2) as: ideologicalinequality, objecti cation, stereotyping-dominance, misogyny-non-sexual-violence, sexual-violence, non-sexist.
We evaluated the in uence of di erent pre-processing steps on the EXIST dataset (for both languages) covering ltering and normalization of varying intensities: { Removing only hashtags: e.g., to avoid over- tting on speci c hashtags. { Removing only punctuation { Removing mentions, hashtags, and links { Removing mentions, hashtags, links, digits, punctuation, and non-ASCII symbols Based on related work on sexism detection [11] and hate speech detection with transformer models [10], we decided to test di erent pre-processing pipelines for both languages. Also, corresponding approaches have shown promising results in detecting disinformation with transformer models and using various pre-processing pipelines [14]. Of all pre-processing steps, the last pipeline had the best ne-tuning results for the multilingual approach. Deleting punctuation and non-ASCII symbols seems to have a higher in uence on ne-tuning transformer models, when we add Spanish data. Further pre-processing steps such as stopword-removal, stemming or lemmatising were omitted since they are not required by the applied contextualised transformer models or would decrease their performance. For tokenisation the models' built-in tokenisers were used. 2.2
Data augmentation is one of the two strategies being pursued with our challenge
contribution. In addition to the EXIST dataset provided by the organisers we
pre-train di erent models on additional datasets which are semantically related
to the EXIST dataset. The intention is to learn additional patterns from
semantically similar or aligned tasks and to transfer them onto the EXIST tasks. We
conducted experiments using two additional datasets - speci cally the MeTwo [11]
and HatEval2019 [
{ MeTwo: is a Spanish dataset which consists of 3600 tweets to detect sexist innuendo, behaviors and expressions. The labels of the tweets are: SEXIST, NON SEXIST and DOUBTFUL. The original dataset consists of tweet-IDs labeled as "status id" and the associated label for the category. Content and metadata of the corresponding tweets was provided by the creator of the dataset upon request. { HatEval2019: is a dataset which can be used for detecting hate speech against women and immigrants. It is composed of 13000 English tweets and 6000 Spanish ones. From a total of 19600 tweets, 9091 have a negative relation towards immigrants and 10509 against women. Furthermore, the tweets are divided into 3 categories:
Hate Speech (HS): Binary value that indicates if hate speech against women or immigrants occurs in the tweet or not.
Target Range (TR): If hate speech occurs in the tweet, the target range speci es whether it targets a generic group of people or a speci c individual.
Aggressiveness (AG): If hate speech occurs in the tweet, additional information is provided whether this is aggressive or not.
We augmented the EXIST and the additional datasets by translating each post into the respective other language (i.e., from English to Spanish and vice versa). Due to this procedure, an English and a Spanish version of each dataset was created. The online tool Google Translator was used for this purpose. 2.3
To model the textual data, we employed two di erent transformers [16]: multilingual BERT (mBERT) and XLM-RoBERTa (XLM-R).
{ mBERT is based on the original BERT (Bidirectional Encoder
Representations from transformers) model [
Unlike the latter, XLM-R is not trained on Wikipedia data but on
monolingual CommonCrawl data. The model shows improved cross-lingual language
understanding in the results shown in the original paper [
For this system we used the already pre-trained XLM-R [
We used an already pre-trained multilingual, uncased BERT model (model size: L=12, H=768, A=12; number of total parameters = 110M) [15] and ne-tuned it on the provided EXIST dataset and its translations. Beforehand, the data was pre-processed by removing mentions, hashtags, links, digits, punctuation, and non-ASCII symbols. In a rst step, we ne-tuned the pre-trained mBERT using only the provided EXIST dataset. Subsequently, we conducted further experiments using the translated EXIST dataset and the additional datasets (HatEval2019 and MeTwo). We ne-tuned the pre-trained mBERT for all combinations of the datasets with and without translations. The best results in the development phase were achieved using only the EXIST dataset and translations. For both tasks, the proposed mBERT was trained separately using an Adam optimiser with a learning rate of 1e 5 and an epsilon of 1e 8. We preset the maximum sequence length for our mBERT to 384 and the batch size to 8. Furthermore, we empirically determined the optimal number of epochs for both tasks, i.e., 6 epochs. 4
The validation and test results for both tasks are presented in Table 1. The last column in Table 1 lists the ranking of our submissions in the EXIST 2021 benchmark. The top ranked submission in the overall benchmark achieved an accuracy of 78.04% and a macro-averaged F1-score of 78.02% for task 1 (team: \AI-UPV") and an accuracy of 65.77% and a macro-averaged F1-score of 57.87% for task 2 (team: \AI-UPV"). 4.1
Fine-Tuning Strategy: For run 1, we used the mBERT ne-tuned on the (preprocessed) EXIST dataset with translations. In our approach in run 1, mBERT seems to over t on the training data, as the validation accuracy of 79.97% is signi cantly higher than the test accuracy of 71.82%.
Pre-Training Strategy: In run 2, we aggregated the predictions from the XLM-R approach trained for task 2, where we pre-trained the model on the EXIST, HatEval2019 and MeTwo datasets (without translations). Our approach with pre-training XLM-R in run 2 achieves the best results. These results are closely followed by the late fusion approach. The performance in run 2 (and run 3) is similar for our validation and the test set, which indicates that this approach generalises well. Our run 2 ranked 5th overall in the benchmark and performed only 0.52% less accurate (in terms of classi cation accuracy) than the overall best submission in the EXIST benchmark.
We conducted experiments with the original XLM-R that we ne-tuned on the original (non pre-processed) EXIST dataset and the additional datasets (see Table 2). Interestingly, the model performed signi cantly better for English content than for Spanish content. Fine-tuning on the additional datasets did not improve the results, but rather made them worse. Comparing the results from the last row in Table 2 with our run 2 for task 1 from Table 1, we can see that the pre-training yielded an advantage over the ne-tuning.
Fusion Strategy: For run 3, we performed a late fusion. The predictions were determined by calculating the maximum of the the sum of the predicted classwise probabilities of run 1, run 2, and an additional mBERT model ne-tuned on the (pre-processed) EXIST and MeTwo dataset (without translations). Our run 3 performed slightly less accurate (in terms of classi cation accuracy) than run 2 and ranked 10th overall in the benchmark. 4.2
Fine-Tuning Strategy: For task 2 and run 1, we again used the mBERT ne-tuned on the (pre-processed) EXIST dataset with translations. The results indicate that mBERT seems to over t on the training data, as the validation accuracy of 68.24% is signi cantly higher than the test accuracy of 59.76%. Pre-Training Strategy: In run 2, we applied the XLM-R approach, where we pre-trained the model on the EXIST, HatEval2019 and MeTwo datasets (without translations). For task 2, a similar pattern can be seen in the results as for task 1. Our approach in run 2 achieved the best results of our or submissions and ranked 6th in the EXIST Challenge, performing only 1.98% less accurate (in terms of macro-averaged F1-score) than the overall best submission in the benchmark.
Fusion Strategy: For run 3, we also performed a late fusion in a similar manner as for task 1, but only with the predicted probabilities of run 1 and run 2. The classi cation accuracy of the late fusion approach in run 3 is identical to the result in run 2. For the macro F1 score of 55.59%, results show a slight di erence compared to run 2.
In this paper, we described our submission to the EXIST2021 benchmark, which
consists of two tasks on the classi cation of sexist content. In our experiments
we found that the unsupervised pre-training strategy of the XLM-R model [
This contribution has been funded by the FFG Project \Defalsif-AI" (Austrian security research programme KIRAS of the Federal Ministry of Agriculture, Regions and Tourism(BMLRT), grant no. 879670) and the FFG Project \Big Data Analytics" (grant no. 866880). and Short Papers). pp. 4171{4186. Association for Computational Linguistics, Minneapolis, Minnesota (Jun 2019). https://doi.org/10.18653/v1/N19-1423, https://www.aclweb.org/anthology/N19-1423 7. Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., Soricut, R.: Albert: A lite bert for self-supervised learning of language representations (2019) 8. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V.: Roberta: A robustly optimized bert pretraining approach (2019) 9. Montes, M., Rosso, P., Gonzalo, J., Aragon, E., Agerri, R., Angel Alvarez Carmona, M., Alvarez Mellado, E., de Albornoz, J.C., Chiruzzo, L., Freitas, L., Adorno, H.G., Gutierrez, Y., Zafra, S.M.J., Lima, S., de Arco, F.M.P., (eds.), M.T.: Proceedings of the iberian languages evaluation forum (iberlef 2021). In: CEUR Workshop Proceedings (2021) 10. Mozafari, M., Farahbakhsh, R., Crespi, N.: Hate speech detection and racial bias mitigation in social media based on bert model. PLOS ONE 15(8), 1{26 (08 2020). https://doi.org/10.1371/journal.pone.0237861, https://doi.org/10.1371/journal.pone.0237861 11. Rodr guez-Sanchez, F., Carrillo-de-Albornoz, J., Plaza, L.: Automatic classi cation of sexism in social networks: An empirical study on twitter data. IEEE Access 8, 219563{219576 (2020). https://doi.org/10.1109/ACCESS.2020.3042604 12. Rodr guez-Sanchez, F., de Albornoz, J.C., Plaza, L., Gonzalo, J., Rosso, P., Comet, M., Donoso, T.: Overview of exist 2021: sexism identi cation in social networks.
Procesamiento del Lenguaje Natural 67(0) (2021) 13. Schindler, A., Lidy, T., Rauber, A.: "comparing shallow versus deep neural network architectures for automatic music genre classi cation", booktitle="in proceedings of 9th forum media technology (fmt2016), st. polten, austria; 23.11. 2016-24.11. 2016; in:" proceedings of the 9th forum media technology (fmt2016)", st. polten university of applied sciences, institute of creative media technologies,(2016), isbn: 9781326881184; 5 s. 14. Schutz, M., Schindler, A., Siegel, M., Nazemi, K.: Automatic fake news detection with pre-trained transformer models. In: Bimbo, D., et al (eds.) Pattern Recognition. ICPR International Workshops and Challenges. ICPR 2021. Lecture Notes in Computer Sciences. vol. 12667. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-68787-8 45 15. Turc, I., Chang, M.W., Lee, K., Toutanova, K.: Well-read students learn better: On the importance of pre-training compact models. arXiv preprint arXiv:1908.08962 (2019) 16. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L.u., Polosukhin, I.: Attention is all you need. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems 30, pp. 5998{6008. Curran Associates, Inc. (2017), http://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf 17. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Scao, T.L., Gugger, S., Drame, M., Lhoest, Q., Rush, A.M.: Transformers: State-of-the-art natural language processing. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. pp. 38{45. Association for Computational Linguistics, Online (Oct 2020), https://www.aclweb.org/anthology/2020.emnlp-demos.6 18. Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., Le, Q.V.: Xlnet: Generalized autoregressive pretraining for language understanding (2019)