Sexism is harmful behaviour that can make women feel worthless promoting self-censorship and gender inequality. In the digital era, misogynists have found in social networks a place in which they can spread their oppressive discourse towards women. Although this particular form of oppressive speech is banned and punished on most social networks, its identi cation is quite challenging due to the large number of messages posted everyday. Moreover, sexist comments can be unnoticed as condescends or friendly statements which hinders its identi cation even for humans. With the aim of improving automatic sexist identi cation on social networks, we participate in EXIST-2021. This shared task involves the identi cation and categorisation of sexism language on Spanish and English documents compiled from micro-blogging platforms. Speci cally, two tasks were proposed, one concerning a binary classi cation of sexism utterances and another regarding multi-class identi cation of sexist traits. Our proposal for solving both tasks is grounded on the combination of linguistic features and state-of-the-art transformers by means of ensembles and multi-input neural networks. To address the multi-language problem, we tackle the problem independently by language to put the results together at the end. Our best result was achieved in task 1 with an accuracy of 75.14% and 61.70% for task 2.
This manuscript describes the participation of the UMUTeam in the shared task EXIST 2021 [12] proposed at IberLeF 2021 [7], focused on the identi cation and categorisation of sexist language. Despite all the bene ts of the raising of the Web 2.0. when it comes to reducing barriers to communication, misogynists have found in social networks a place in which they can spread their oppressive discourse towards women, making these networks an intimidating place. Sexism and misogyny are particular forms of discriminatory speech and conduct in which women are the victims of the harassment. It is worth mentioning the identi cation of sexist speech is more challenging than other forms of hate-speech, as sexist and misogynistic messages can go unnoticed within funny or complacent comments. Therefore, some sexist messages are subtle to distinguish even for humans if we are not aware of them, just how messages with a condescending treatment could be. Sexist behaviour includes stereotyping, ideological issues, or sexual violence [11].
The objective of EXIST 2021 shared task is the identi cation and
categorisation of sexism behaviours from a broad sense. Speci cally, the organisers of the
shared task proposed two challenges. On the one hand, a binary sexism classi
cation task and, on the other, a multiclass sexism categorisation task that consisted
in a ne-grained classi cation of those messages classi ed as sexist in the
following traits: (
Our participation is grounded on the usage of linguistic features combined with state-of-the-art transformers. As part of the doctoral thesis of one of the members of the UMUTeam, we are evaluating a tool for extracting linguistic features. This tool is inspired in LIWC [13] but designed to the Spanish from scratch. Our hypothesis is that the usage of linguistic features could be combined with statistical features, such as n-grams or any form of embeddings to improve the accuracy of the results and, at the same time, providing interpretable features. Therefore, one of the runs consisted into the usage of linguistic features in isolation as baseline and then we combined them with BERT in another run and nally we stack an ensemble for the last run.
The remainder of this manuscript is organised as follows. First, in Section 2 some experiments and corpus regarding misogyny and sexist behaviour are discussed. Next, in Section 3 we give some insights regarding the corpus that was made available to the participants. Our pipeline is described in Section 4. In Section 5 we show the results achieved by our team and compare them with the best results achieved by the rest of the participants and the baselines proposed. Finally, the conclusions and future research directions are shown in Section 6.
The identi cation of aggressive, hateful, and oppressive speech have been
recurrent tasks in NLP workshops. For our point of view, this trend is caused by two
main factors: (
Our research group has also focused on misogyny identi cation with the release of the Spanish MisoCorpus-2020 [6], focused on Spanish. The MisoCorpus2020 contains three subsets: VARW, SELA, and DDSS, focused, respectively, on the identi cation of misogynistic messages towards female politicians and journalists, cultural and background di erences between misogyny among European Spanish countries and Latin America countries, and tweets that contains speci c misogynistic traits. In this work we evaluate the combination of linguistic features and sentence word embeddings from fastText. The results shown that linguistic features regarding o ensive language, grammatical gender, spelling mistakes, punctuation symbols, and jargon from social networks are e ective for misogyny identi cation. 3
The dataset is composed by documents written in Spanish and English compiled between December 2020 and February 2021 that contains expressions used to underestimate the role of women in our society. The main data-source is Twitter but Gab was used to extend the dataset. Training and testing have a temporal separation as tweets were selected based on time in order to determine which ones belong to each split. According to the description of the task, a subset of the data were analysed in depth by two experts in gender issues.
The resulting dataset contains 6977 tweets for training and 3386 tweets for testing, where both datasets were randomly selected from the 9000 and 4000 labelled sets, to ensure class balancing for Task 1. This dataset was enlarged with 492 gabs in English and 490 in Spanish from the uncensored social network Gab.com following a similar procedure as described before. This set will be included in the EXIST test set in order to measure the di erence between social networks.
Each document was labelled by ve annotators. The nal label was selected using a majority vote. However, tweets with 3-2 votes were manually reviewed by two experts of di erent gender. The reader can nd more details regarding the dataset compilation in the overview of the task [12].
To evaluate the reliability of our proposal we extracted a validation split consisting in the 20% of the training dataset. Table 1 and Table 2 depict the label distribution we used for task 1 and 2 respectively. We can observe that the distribution of the sexism identi cation task is slightly imbalanced with more documents labelled as non-sexist. For subtask 2, we can observe that three of the sexist traits namely ideological inequality, stereotyping and dominance, and misogyny non-sexual violence have similar number of documents but the labels objecti cation and sexual violence have less examples. Our proposal is grounded on the combination of linguistic features with state-ofthe art transformers. During our experimentation, we also evaluated word and sentence embeddings.
For the linguistic features we use UMUTextStats [5, 6]. This tool is inspired in
LIWC [13] but designed for the Spanish language. Although LIWC has available
a Spanish version that has been evaluated in di erent domains, such as satire
identi cation [10], it does not take into account speci c Spanish linguistic
phenomena that UMUTextStats does. Speci cally, UMUTextStats handles a total
of 365 linguistic variables classi ed in the following groups: (
For the transformers we use BERT. Speci cally, the large cased version for those documents written in English and BETO [2] for those tweets written in Spanish. In addition, we train and evaluate models applying word and sentences embeddings from fastText, word2vec, and gloVe for the Spanish documents and from fastText from the English documents. These features were trained with recurrent, convolutional, and vanilla neural networks. Multiple feature sets in the same neural network were also evaluated using the functional API of Keras. It is worth noting that the evaluation of other neural networks architectures such as convolutional neural networks was performed because provided good results in the past for conducting sentiment analysis tasks [8].
Our strategy in this shared task consisted in dealing with documents written in Spanish and English separately, by splitting them in two datasets and evaluated with di erent models to merge the results just before the nal submission.
For the hyperparameter optimization we proceeded as follows. We evaluate a
total of 100 neural networks per feature set in isolation and in combination, both
for Spanish and English. As there was a slightly imbalance among the classes, the
best models were selected based on weighted F1 score. The features evaluated
were linguistic features (LF), sentence embeddings from fastText (SE), sentence
embeddings from BERT (BE), and word embeddings (WE) from fastText for the
English dataset and fastText, gloVe, and word2vec for the Spanish dataset. The
majority of neural networks evaluated consisted in multilayer perceptrons (MLP)
with di erent number of hidden layers (between 1 and 8), and di erent number
of neurons (
We participate with three runs. The rst one consisted in the usage of the linguistic features in isolation. As commented earlier, this run was used to set a baseline to evaluate the thesis objectives of the doctoral student of the team. The second one consisted in the combination of linguistic features and a BERT-based model, and the third run consisted in an ensemble of neural networks composed from LF, SE, BE, WE, and BERT.
First, the results based on the rst task concerning sexism identi cation are depicted in Table 3.
In regards of the o cial results for task 1 (misogyny identi cation), we achieved our best result with the ensemble model (run 3) with an accuracy and F1-score of 75.14%, reaching position 22 in the o cial rank. The overall best result was achieved by AI-UPV, with an accuracy of 78.04% and a F1-score of 78.02%. Our second run, based on BERT and LF, achieved a 74.4% of accuracy and F1-score, reaching position 27. Finally, our rst run, based on the usage of linguistic features in isolation, achieved position 63, with an accuracy of 59.66% and a F1-score of 59.64%. Note that this result does not outperform the baseline result consisted in a bag of words based on TF-IDF score. The poor reliability of linguistic features in isolation is not sparingly due to the fact that UMUTextStats is focused on the Spanish language. In fact, the results only with the Spanish partition achieved an accuracy of 61.94% whereas with the English partition only a 57.43% of accuracy was obtained. The feature selection stage for the English dataset mainly selected those linguistic features based on Corpus statistics such as the length of the text or the Type-Token Ratio (TTR).
Figure 1 depicts the Mutual Information of the top ranked LF for the task 1 for the Spanish split compared by label. We can observe that there are no important di erences between linguistic features among classes except for lexical related to sex and female groups that appears mostly in sexist posts.
non-sexist sexist stylometry corpus inflesz psycholinguistic processes
negative morphosyntax gender words
masculine morphosyntax morphology determiners interrogative social media reply male stylometry corpus words with 2ltr stylometry corpus words with
19ltr errors orthographics
lexical sex lexical social social female 0% 25% 50% 75% 100%
Second, the result of the second task (sexism categorisation) are depicted in Table 4.
For task 2 (see Table 4), our best result was achieved with the combination of BERT and LF with an F1-score of 53.62%, reaching position 18 in the o cial rank. Similarly to task 1, our best run is not far for the best result achieved by UPV with an F1-Score of 57.87%. However, contrary to the rst task, our third run based on the ensemble model, achieved lower result than our second run. The rst run, consisted on the linguistic features, achieved lower results falling below the baseline. In this task, however, it draw our attention the reliability of the LF were similar for Spanish and English, achieving an F1-score of 28.0% for Spanish and 27.21% for English.
Figure 2 depicts the Mutual Information of the linguistic features for the task 2 for the Spanish split compared by label. However, these results must be viewed with caution, due to the limited results of the linguistic features in isolation for the task sexism categorisation. We can observe that lexical words regarding sex appears less frequently in sexism messages categorised as stereotyping and dominance. Out of the di erent sexist traits, we can notice that negative statements (psycholinguistic processes negative ) appear most frequently in documents labelled as misogynist but without sexual violence. The rest of the features does not show signi cant di erences among the sexist traits. 6
In this manuscript we have detailed the participation of the UMUTeam in the EXIST 2021 shared task with three runs that combined linguistic features with state-of-the-art transformers. We are very happy with the opportunity that we have been given to participate in these tasks and we hope to repeat it in the future. We are aware that the results achieved by the LF in isolation are below reliable baselines such as n-grams based on TF-IDF. We are currently evaluating the labels of the test set in order to detect weakness and to improve our pipeline. Moreover, we are currently implementing more advanced ensembles by training deep-learning models that learn from the predictions of each individual model. We also have learned another way to create sentence- xed embeddings from netuned BERT models that are more easy to combine with other kind of features.
For future research directions we observed that misogyny categorisation was performed as multiclass, in which all labels are considered mutually exclusive. However, we consider that it will be interesting to evaluate this sexist speech as a multi-label task. However, we are aware that this proposal would imply to relabel the dataset. Another research direction is to incorporate contextual features to the classi cation in order to provide a context to the documents. 7
This work was supported by the Spanish National Research Agency (AEI) through project LaTe4PSP (PID2019-107652RB-I00/AEI/10.13039/501100011033). In addition, Jose Antonio Garc a-D az has been supported by Banco Santander and University of Murcia through the industrial doctorate programme.
objectification lexical sex stylometry corpus inflesz morphosyntax gender words
feminine stylometry corpus syllabes per
word psycholinguistic processes
negative morphosyntax gender words
masculine stylometry corpus length
errors orthographics morphosyntax affixes suffixes stylometry corpus words with 2ltr 0% 25% 50% 75% 100%