This paper presents the participation of the MiniTrue team in the EXIST 2021 Challenge on the sexism detection in social media task for English and Spanish. Our approach combines the language models with a simple voting mechanism for the sexist label prediction. For this, three BERT based models and a voting function are used. Experimental results show that our nal model with the voting function has achieved the best results among our four models, which means that this voting mechanism brings an extra bene t to our system. Nevertheless, we also observe that our system is robust to data sources and languages.
Equality, openness, and freedom, as the foundation and pillar of the Internet
spirit, should have made our cyberworld a better place. However, inequality,
prejudice and bias against females still deeply implants in online social networks [
Currently, researches on the sexism detection are frequently related to
misogyny detection. Misogyny and sexism are usually considered as synonyms, but
there are some nuances between these two terms. The de nition of misogyny
emphasizes more on the enmity, hatred and contempt towards female [
Traditionally, technologies of sexism detection are assembled into pipelined
architectures. User and relationship based features (e.g. including followers,
friends, number of tweets, interacting with other sexism users, etc.), as well
as the the text-based features (e.g. TF-IDF, Character n-grams, Token n-grams,
etc.) are extracted from the social network. Then, a number of traditional
machine learning algorithm could be applied for this problem [
However, recent approaches based on end-to-end neural networks o er a
promising alternative. Recurrent Neural Networks (RNN) and Long Short Term
Memory networks are commonly used for Natural Language Processing tasks in
earlier works. [
This work is done as part of the EXIST 2021 (Sexism Identi cation in Social
Networks) shared task [
The note is organized as follows. We present the description of our method in Section 2. Then we group the experimental results in Section 3 before discussing the perspectives and concluding in Section 4. 2
This section presents the architecture and the description of our system. Three basic models and a vote-inference system are described respectively in Subsection 2.1 and Subsection 2.2. 2.1
We develop three independent basic models for label prediction by combining language models with simple neural networks. As before mentioned, three language models are applied in our system, including BERT, Multilingual BERT, and BETO.
Basic Model One: This rst basic model connects the language models with
a feed forward network. In order to gain the multilingual representation for the
input text data, our model uses two di erent pretrained language models, the
BERT and the Multilingual BERT. After tokenization, the text data feeds into
these two language models respectively, and generate two sentence embeddings.
A concentration layer is applied to combine these two embeddings together. After
that, embedded data ows to a feed forward network for the nal inference I1.
Basic Model Two: Research points out that the performance of Multilingual
BERT model is not as good as the language speci c language models [
Basic Model Three: To leverage the sequential information of the input text, we use a Bi-directional LSTM network to obtain the consecutive features after the language model. In this time, we just use a single multilingual BERT, and get the sequential embeddings for each input token. After the Bi-directional LSTM, we use a Sigmoid function to get the nal inference I3. 2.2
As before mentioned, I1, I2, I3 are three inference values predicted by our basic models, and a simple voting mechanism is applied to count the nal output. If two or more models draw the same inference label, this label will be our nal prediction. The nal output is predicted as: f (x) := (0 if I1+I2+I3 <2 1 if I1+I2+I3 >= 2 (1)
The architecture of our model is shown in Figure 1. As we can see, the input text date feeds into each basic model, and then ows to each inference network respectively. Finally, all of this information is combined in the voting function for the prediction. 3
In this section, we present the description of datasets, the setup of our experiments, and the nal results of our experiments. Experimental validation is conducted on the EXIST 2021 training and test corpus. The datasets are described in subsetion 3.1 followed by the analysis of our nal results in subsetion 3.2.
There are two sub-tasks in this shared task, the rst one is Sexism Identication, which is a binary classi cation task, and the system needs to decide whether a given social media text is or is not sexist. The second sub-task is to categorize these sexist texts into di erent prede ned sexism types. This paper focuses on the rst sub-task.
Both of the training and test datasets consist in English and Spanish text data, which are collected from the social media such as Twitter and Gab. Table 1 brie y describes the corpus. As we can see, there are 6977 tweets/gabs in the training set, and 4368 tweets/gabs in the test set, and labels are balanced in this sub-task. 3.2
The evaluation of this sexism identi cation sub-task is mainly based on Accuracy score. Our results are gathered in Table 2, which contains the Accuracy and F-Measure scores for the basic models and the nal systems. Our rst question is whether the voting function brings additional bene ts to our system. As we can see, the our nal system (with a simple voting mechanism) performs the best Accuracy and F-measure metrics among these four systems, which provides a 3% extra performance regarding basic models.
Unlike the training data set, the test set contains the text data from di erent sources, which are the Twitter and Gab. It brings the second question, which is whether our nal model is data source sensitive. So we list the results in di erent data sources for each model. As shown in Table 3, even though the training data comes from Twitter, the results in Gab are better than those in Twitter in the test set, which shows a robust ability of the language models.
The last question about our system is whether it is language sensitive. Table 4 lists the evaluation metrics of our systems by di erent languages. Results show that the performance of our models in di erent languages is similar to each other, and it seems that to introduce the language speci c language models, i.e. the BETO has not brought extra bene ts to our nal system.
In this paper, we implement a classi cation system that combines the language models with a simple voting mechanism. Our system has been evaluated on the EXIST 2021 sub-task one. The evaluation results have shown that our system is datasource and language robust, and the voting function indeed brings in extra bene ts to our nal system. Our experiments prove that the pre-trained language model is also highly adaptable to social media texts, and a simple voting mechanism can highly leverage the predictive ability of the multi-model system.