The presence of misogynistic contents is one of the most crucial problems of social networks. In this paper we present our system for misogyny identification on Twitter. Our approach is based on a convolutional neural network that exploits pretrained word embeddings. We also experimented a comparison among different architectures to understand the effectiveness of our method. The paper also described our submissions to both subtasks A and B to Automatic Misogyny Identification competition at Evalita 2020.
The paper describes our submission to the
Automatic Misogyny Identification task at Evalita 2020
We proposed a convolutional based approach to recognize misogynistic sentences. We grounded our work over a robust model selection technique. In order to confirm our approach we developed other architectures based on state of art models to make a systematic comparison.
Our work is organized as follows. Section 2 briefly describes related work on the proposed task. Section 3 describes our architectures. Section 4 introduces our method. In particular, it describes our approach for model selection and assessment. Section 5 presents the official results obtained in the AMI competition. Section 6 concludes this work. 2
The misogyny identification and classification
approaches are very recent
Finally, it is important to mention that
different deep learning approaches have been proposed
In this section we describe our approach that exploits the intuition of extracting dependencies among words as features from tweets. We also made an analysis about other architectures and we compare them with ours in order to understand strength and weakness of our architecture. Our method consists of the following steps: • normalization of the datasets; • use an effective word embedding representation; • define different state of art architectures to compare them with our model.
This model was trained for 15 epochs using a
batch size of 128.
Out-of-vocabulary words are one of the most
important issues with the use of word embedding,
especially in the context of social networks in which
colloquial language is widespread. In order to
normalize tweets, we pre-processed them using tools
from ekphrasis
First of all we removed punctuation and separated sentences into words. Then we applied the normalization process. This process involves, for example, allcaps annotation (’ABC’ becomes ’allcaps abc allcaps’), elongated words normalization (’vaaaaai’ becomes ’elongated vai elongated’) and emoticons transformation. We manually carried out translations of these keywords to adapt annotations to the Italian language.
We experimented different word embedding pretrained model. After a sequence of considerations we chose the word embeddings presented in Cimino et al. (2018) trained on 46 million Italian tweets. It is a word2vec based model and it encodes each word in a 128-size vector. 3.2
The model used for the AMI competition is represented in Figure 1.
Given a tweet, we firstly apply the pre-processing described in Section 3.1 to normalize and transform it into a sequence of words. Then this sequence is mapped into a fixed real vector domain by the embedding layer.
The embedding layer passes an input feature space
to three 1D Convolutional layers. Each of those
uses 150 filters and a stride of 1 but different
kernel sizes of 1, 2, 4 respectively. These layers are
the most interesting ones. Each layer can indeed
be seen as extractors of n-gram features where n is
equal to the kernel size
The last two layers are dense layers with tanh and softmax activation functions respectively. The final softmax layer maps the sequence received as input to a probability distribution over all possible classes. 4
In the subtask A we split the training set provided into a train set (4250 tweets) and a test set (750 tweets). This internal test set was used only to evaluate our final model. In subtask B we merged raw and synthetic datasets and separated from each of these two test sets.
As explained in Section 3.2 we used as output layer a dense layer with softmax activation function. In order to obtain three different labels for subtask A, misogynous and aggressiveness columns were converted into a single one. We also apply one-hot encoding to the integer representation, otherwise a natural ordering between categories may result in poor performance or unexpected results.
The frequency distribution of these labels turns out to be quite unbalanced, as shown in Table 1. Furthermore for each class we have a very small number of training examples. This could have a strong influence on the overfitting of the model. We indeed avoided to use a deep neural network and we preferred to develop a simple architecture in a robust way as recommended in Zhang and Wallace (2015).
2277 484 1489
386
70
294
We decided to apply a robust model selection
technique to find the best hyperparameters of our
model. We used repeated K-fold cross-validation
In subtask A we used the official AMI score as metric. While in the subtask B we decided to use the AUC metric. In both of them we also took into consideration the standard deviation among different runs.
Model selection phase was divided in 2 mainly stages: • Stage 1 we validate the best hyperparameters for each different model. We report the hyperparameters ranges in Figure 2. In this stage we used a repeated 5-fold with 10 repetitions. • Stage 2 We chose the most promising models according to score and standard deviation metrics. We applied another repeated 5-fold cross-validation increasing the number of repetitions to 15. Then we chose the best model among them using the same metrics as before.
f32, 64, 128g f[100, 100, 100], [150, 150, 150]g f[1, 2, 3], [1, 2, 4]g f0.001, 0.005g f8, 16g
Then we built other architectures to compare
them with ours. In the following we list models
used for these comparisons:
• Convolution-biGRU Based Deep Neural
Network: this architecture allows to capture
long-range dependencies from both
directions of a sentence;
• Convolutional Based Neural Network: deep
neural network based on convolutional layers
that tries to extract different features using a
greater number of layers. It is an extension of
the architecture described in Section 3.2;
• Skipped Convolutional Neural Network
As regards subtask B, we only considered the AUC score and results obtained for both model selection and assessment have proved to be inconclusive.
Run Run 1 Run 2 Run 3 Both run fabsam.r.c.run1 and fabsam.r.c.run1 have outperformed other constrained runs and our best run ranks third in the official leaderboard. This confirms the effectiveness of our approach. During an error analysis we noticed that our model wrongly classifies short sentences and hate speech sentences referred to men. Nevertheless, in our best run the f1 score for misogynous label reaches 0:8038 while the real problem is in the 0:6647 of aggressiveness label. This is probably due to the small number of non-aggressive examples used to fit the model.
Different results of runs reflect the standard deviation observed during the validation phase. While scores obtained are smaller then model selection results. 5.2
In the following we reported our results for the subtask B.
SubtaskB run2 run3 run1 run1 run2 run2 run3 run3 u/c u c c c c c c c We used for subtask B the same model used for the other subtask. We have performed a poor validation approach using as evaluation metric the AUC. We chose to train the model merging raw and synthetic datasets. This choice led to poor performance on unseen datasets. Indeed our model was strongly affected by overfitting when it met identity terms used in training. From an error analysis we noticed that it wrongly classifies lots of sentences from synthetic dataset, while it performs very well on raw dataset. 6
The presence of misogynistic contents in social network is a major problem. A crucial work in this direction is the detection and recognition of this type of contents.
We propose a simple architecture based on convolutional layers. From our experiments we understood that capturing long-term dependencies produces an unstable training and poor performance in this type of subtasks. Performances of the model could be increased focusing its approach on model selection. Lastly, it could be very important to take into consideration data augmentation techniques or other sources of data to solve the unbalanced dataset issue.