Detecting and tackling sexist messages in social media is important for encouraging better behaviours in our society as well as to contribute to e ective equality between men and women. In this paper we present our participation in the sEXism Identi cation in Social neTworks (EXIST) task at IberLEF'2021 [1]. Our approach to solve the task is based on considering the sexism as a subset of bias. Our work consisted in transferring lexical features commonly associated with bias, and analyzing how well they serve to detect sexism in social networks. The results show that these types of features do not have much statistical correlation with these types of short sexist messages.
The emergence of social networks in recent years has allowed people from di
erent countries and cultural backgrounds to communicate with each other freely
using the same communication channels. This fact, which is a positive
consequence of web technologies, also carries a series of disadvantages. The
disinhibition generated by anonymity makes users feel free to say things that they would
never say in person, including hateful and sexist messages [
For this reason, creating systems capable of automatically detecting sexist messages on social networks is very important for encouraging better behaviors in society and ght against discrimination and inequality. It is also important to analyze and detect sexism as it is presented in this task, identifying not only hateful messages, but also messages that discredit the feminist movement, that deny equality between men and women, or that present women as objects.
On the other hand, the study and interpretation of bias is a wide research eld and it has been studied from many di erent perspectives. The bias in a text is not only found in words choice, but also in what information is omitted and what is commissioned, in what labels are placed on the entities that appear in the text, and it may even be present in who read the text according to its socio-cultural context. In this work, we focus on studying bias from a lexical point of view.
According to the Oxford English Dictionary, sexism may be understood dened as any type of prejudice, stereotype or discrimination, generally against women, that is carried out on the basis of sex. If we compare this de nition with the de nition of bias (inclination or prejudice for or against a person or group, especially in a way considered unfair) we will realize that sexism is a subset of it. Our approach is based on transferring features commonly used in the task of bias and propaganda detection, and studying the applicability of these features with the detection of sexist messages published in social media.
In our approach, we study how well these extracted characteristics model sexism. To do this, we add them to several classic machine learning models based on TF-IDF, and a bidirectional Long Short-Term Memory (Bi-LSTM) model with a word-embeddings layer. The results obtained show that the correlation between the lexical characteristics extracted, and the class of the text (binary or multiclass) is minimal, resulting in very little or inexistant improvements in the learning step performance. The limitations encountered are discussed in Section 4.
The rest of this paper is structured as follows. In Section 2, we explain the data used in our system and the pre-processing done. Section 3 presents the details of the proposed systems. In Section 4 we present and analyze the results obtained in our experiments and in the EXIST competition. Finally, the paper concludes in Section 5 with conclusions and future work. 2
The sEXism Identi cation in Social neTworks (EXIST) task consits on
automatically identi ng sexism content on social networks such as Twitter and Gab.com.
The aim of this task is to detect sexism content in a broad sense, from explicit
and o ensive misogyny to other subtle expressions that involve implicit sexist
expressions and behaviours [
The task is divided into two sub-tasks. The rst sub-task is a binary classi cation in which the system have to classify the given text as \sexist" (it is sexist itself, describes a sexist situation or criticizes a sexist behaviour) or \nonsexist". The second sub-task aims to classify the text, once a message is classi ed as sexist, according to any of the following type of sexism: { Ideological and inequality: the message discredits the feminist movement, rejects inequality between men and women, or presents men as victims of gender-based oppression. { Stereotyping and dominance: the message expresses false ideas about women that suggest they are more suitable to ful ll certain roles, or inappropriate for certain tasks, or claims that men are somehow superior to women. { Objecti cation: the message presents women as objects apart from their dignity and personal aspects, or assumes or describes certain physical qualities that women must have in order to ful ll traditional gender roles. { Sexual violence: the message includes or describes sexual suggestions, requests for sexual favors or harassment of a sexual nature. { Misogyny non-sexual violence: the message expresses hatred and violence towards women.
The dataset provided contains 11,345 instances of text in both English and Spanish. The texts were extracted from the social networks Twitter (tweets) and Gab (gabs). The training set contains 6,977 tweets while the test set contains 3,386 tweets and 982 gabs. The distribution between both languages has been balanced. 3
In this section we present the system pipeline, which consist in a text preprocessing step, feature engineering, and describe a learning step using both traditional ML techniques and novel deep learning approaches. 3.1
Data pre-processing To assist the feature extraction step, and the TF-IDF computation, the following pre-processing has been applied to the text: 1. Converting the text to lowercase. 2. Removing punctuation and digits. 3. Removing hashtags and mentions. 4. Tokenizing the text. 5. Removing stop-words. The NLTK library has been used to obtain the stopwords in both English and Spanish. 6. Lemmatization. The Stanza library has been used, both for English and Spanish texts.
For the Bi-LSTM with a word embedding layer model developed, this preprocessing is not applied, only the text has been converted to lowercase 3.2
Feature engineering
For both languages, a custom lexicon of biased words is used. We have used
the lexicon of sexist words used in [
Also, we have included Hurtlex [
Finally, other features related to the text context have been extracted (i.e. the count of mentions and hashtags for each tweet/gab), along with the sentiment and the PoS tagging. In Table 1 we describe every feature extracted:
Number of men- [0, n] tions Number of hash- [0, n] tags Biased words [0, n]
Report verbs
Hedge words boolean 3.3
Learning step
from the weak subjective words list [
from the strong subjective words list [
from the hedge words list [
We have developed both traditional methods using TF-IDF features and deep
learning based methods using word embeddings. In the following subsections we
describe the classi cation systems in detail:
Traditional Machine Learning methods. We have opted for some classical
ML classi ers such as Logistic Regression (LR), and Support Vector Machine
(SVM), using the skLearn library, since they are widely used for this type of task
[
Bidirectional Long Short-Term Memory (Bi-LSTM). We have
experimented with some Deep Neural Network approaches such as Bi-LSTM that has
been succcesfully used for NLP classi cation before [
We have developed a Bi-LSTM model whose rst layer performs word embeddings. After this rst layer, we add a dropout layer (0.3 dropout rate) and a fully-connected output layer with one neuron per predicted class. The Adam optimizer is used along with binary cross entropy as loss function for task 1, or categorical loss function for task 2. Besides, 10 epochs were executed for training the models for task 1, and 50 epochs for task 2.
Also, another model with a parallel Dense layer for the bias features extracted has been implemented. This model has been designed as showed in Fig. 1.
The word embeddings used in the embedding layers are the following: { GloVe Twitter 27B: 2B tweets, 27B tokens, 1.2M vocab, uncased, 200 dimensions vectors. { GloVe SBWC (Spanish Billion Word Corpus): 0.85M vocab, uncased, 300 dimensions vectors. { FastText SUC (Spanish Unannotated Corpora): 3B tokens, uncased, 300 dimensions vectors. Embeddings layer
Dropout Bi-LSTM
Bias features Input
Dense Concatenate
Dense
Output
The models that have generated the runs sent to the organizers and that are analyzed in the following section have been trained with the GloVe Twitter 27B embedding for English, and the FastText SUC embedding for Spanish. 4
In this section we show the results obtained for both tasks and we do an analysis of them. 4.1
O
cial results A total of 72 runs were submitted for task 1, our approaches were ranked 44th for the Bi-LSTM model with a word embedding layer, 51st for the Bi-LSTM model with a word embedding layer with bias features, and 65th for the Logistic Regression model with only bias features.
Our results are summarized and compared with the best systems and the baseline system in the following table:
A total of 63 runs were submitted for task 2, our approaches were ranked 37th for the Bi-LSTM model with a word embeddings layer and bias features, 39st for the Bi-LSTM model with a word embeddings layer, and 57th for the Logistic Regression model with only bias features.
Our results are summarized and compared with the best systems and the baseline system in the following table:
Majority Class 1 2 3 44 51 52 65 66 1 2 3 37 39 51 57 62
As we can see in both cases, the incorporation of bias features is not statistically signi cant for the improvement or deterioration of the models. Furthermore, the Logistic Regression method with only bias features performs worse than the baseline method, and just above selecting the majority class for each prediction in both tasks. 4.2
Extended results In this subsection we show the results obtained for systems that were not sent to the competition. We also analyze the di erence in classi cation performance according to the language of the texts. All these results have been obtained in the test set.
If we look at the results, in both cases the models that incorporate bias features perform equally or slightly better than their analogues without these characteristics, while for the results in Spanish, the models that incorporate the bias features worsen signi cantly. We believe this may be due to the fact that in the process of translating the lexicons from English to Spanish, we have lost some relevant information. 4.3
Discussion The results obtained show that the correlation between the bias characteristics extracted, and the class of the text (binary or multiclass) do not behave signi cantly better than other models not using such characteristics. We believe that the use of these features is limited by the di erence between the language for which these lexicons were created and the language being analyzed (formal vs. informal / slang), having to literally translate the lexicons from English to Spanish, and the appearance of unintended bias.
Furthermore, as expected, the models based on deep learning techniques perform better than both of the classic ML models implemented (LR and SVM). This improvement is most clearly seen in subtask 2 (multiclass classi cation).
Sexism does not only have to be latent in hateful messages, this is seen in the categories described by the organizers for task 2. In fact, if we look at the following Pearson correlation matrix, we will realize that all the features extracted from Hurtlex have a higher correlation with the label sexual-violence to a greater extent. It is logical since Hurtlex is a lexicon of o ensive, aggressive and/or hateful words; and this label is the one that has the most relationship with this type of words. If we look at the subsets of Hurtlex that most correlate with this category are: AN (animals), ASM and ASF (male and female genitalia), OM (words related to homosexuality), and PR (words related to prostitution).
Furthermore, we believe that our models are heavily a ected by unintended
bias. This problem leads the models to associate unreasonably high sexist scores
to a non-sexist text only because it contains certain terms, called identity terms
[
In this paper, we present the systems we have developed as part of our participation in the EXIST competition. Speci cally, we have participated in both sub-tasks proposed. In order to solve these tasks, we have implemented classical Machine Learning models and novel Deep Learning methods such as a Bi-LSTM, incorporating lexical features from bias detection and o ensive lexicons.
We have found that adding bias features to any model do not make it behave signi cantly better than the same models not using those features. These small performance improvements are canceled in the classi cation of Spanish texts due to the loss of information produced by translating the bias lexicons with an automatic translation system.
Our next steps will focus on exploring more features from others lexicons related to bias understanding (misinformation, hoaxes, propaganda), and study how to translate better the current English lexicons to Spanish. Also, we aim to investigate di erent bias mitigation strategies.
Acknowledgements. This work was supported by the Spanish Ministry of Science and Innovation under Project Misinformation and Miscommunication in Social Media (PGC2018-096212-B-C32).