English. This paper describes the system that team YNU OXZ submitted for EVALITA 2020. We participate in the shared task on Automatic Misogyny Identification (AMI) and Hate Speech Detection (HaSpeeDe 2) at the 7th evaluation campaign EVALITA 2020. For HaSpeeDe 2, we participate in Task A - Hate Speech Detection and submitted two-run results for the news headline test and tweets headline test, respectively. Our submitted run is based on the pre-trained multilanguage model XLM-RoBERTa, and input into Convolution Neural Network and K-max Pooling (CNN + K-max Pooling). Then, an Ordered Neurons LSTM (ONLSTM) is added to the previous representation and submitted to a linear decision function. Regarding the AMI shared task for the automatic identification of misogynous content in the Italian language. We participate in subtask A about Misogyny & Aggressive Behaviour Identification. Our system is similar to the one defined for HaSpeeDe and is based on the pre-trained multi-language model XLMRoBERTa, an Ordered Neurons LSTM (ON-LSTM), a Capsule Network, and a final classifier.
People use offensive contents in their social media posts to degrade an individual or religion or other organizations in many respects, the identification of such social media posts is a necessity, a
Copyright ⃝c 2020 for this paper by its authors. Use
permitted under Creative Commons License Attribution 4.0
International (CC BY 4.0).
substantial amount of work has been done in
languages like English. However, hate speech and
offensive language identification in other language
scenario is still an area worth exploring. The latest
edition of EVALITA
In recent years, with the acceleration of information dissemination, the identification of hate speech and offense language has become a crucial mission in multilingual sentiment analysis fields and has attracted the attention of a large number of industrial and academic researchers. From an NLP perspective, much attention has been paid to the topic of HS - together with all its possible facets and related phenomena, such as offensive/abusive language, and its identification. This is shown by the proliferation, especially in the last few years, of contributions on this topic (e.g.
Caselli et al. (2020), Jurgens et al. (2019), Fortuna
et al. (2019)), corpora and lexica (e.g. de Pelle
and Moreira (2017),
Unfortunately, nowadays more and more incidents of harassment against women have appeared and misogynistic comments have been found in social media, where misogynists hide behind by anonymity security. Therefore, it is very important to identify misogyny in social media. Pamungkas et al. (2020) conducted extensive and in-depth research on online misogyny, developed a state-ofthe-art model for detecting misogyny in social media and explored the feasibility of detecting misog2https://projects.fzai.h-da.de/iggsa/germeval/ 3https://hasocfire.github.io/hasoc/2020 4http://hitz.eus/sepln2019/ 5http://alt.qcri.org/semeval2020/ yny in a multilingual environment. Aiming at the TRAC-2 shared tasks of Aggression Identification and Misogynistic Aggression Identification, Samghabadi et al. (2020) propose an end-to-end neural model using attention on top of BERT that incorporates a multi-task learning paradigm to address both the sub-tasks simultaneously. Arango et al. (2019) discussed the implications for current research and re-conduct experiments, a closer look at model validation to give a more accurate picture of the current state-of-the-art methods. Recent investigations studied how the misogyny phenomenon takes place, such as Farrell et al. (2019) study this phenomenon by investigating the flow of extreme language across seven online communities on Reddit. Goenaga et al. (2018) automatic misogyny identification using neural networks. Automatic misogyny identification in Twitter has been firstly investigated by Anzovino et al. (2018). 2 2.1
In this part, we describe one of the subtasks HaSpeeDe 2 participating in EVALITA 2020. This task introduces its novelty from three main aspects (Language variety and test of time, Stereotypical communication, Syntactic realization of HS). We participated in Task A - Hate Speech Detection (Main Task), a binary classification task aimed at determining the presence or the absence of hateful content in the text towards a given target (among Immigrants, Muslims or Roma people).
The AMI shared task proposes that misogynous content in Italian is automatic identification in Twitter. It is organized according to two main subtasks, namely subtask A - Misogyny & Aggressive Behaviour Identification and subtask B Unbiased Misogyny Identification. We participate in subtask A, the system must recognize whether the text is misogyny, and if it is misogyny, it must also recognize whether it expresses an aggressive attitude. 2.2
HaSpeeDe 2 task organizer provides a new HS training dataset (binary task) based on Twitter data, accompanied by a test set including both in-domain and out-of-domain data (tweets + news headlines), as well as from different time periods. The HaSpeeDe 2020 new training set already contains the Twitter dataset of HaSpeeDe 2018. The new dataset contains a total of 6,839 tweets (label 0 means NOT HS, label 1 means HS), of which HS contains 2,766, NOT HS contains 4,703, the tweets headlines test set contains 1,263 tweets, and the news headlines test set contains 500 elements. In the experimental run, the data we recommend for this task is the result of combining the Facebook dataset (training set + test set) of HaSpeeDe 2018 with the new training set of HaSpeeDe 2020, this is to analyze the influence of out-of-domain texts in the training set. The two contain a total of 10,839 comments/tweets.
The AMI organizer provided a raw dataset (5,000 tweets) as the training set for participants in subtask A, the raw dataset is a balanced dataset of tweets manually labeled according to two levels: • Misogynous: defines if a tweet is misogynous or not misogynous. Label 0 means Not misogynous tweet, label 1 means Misogynous tweet. • Aggressiveness: denotes the subject of the misogynistic tweet (misogynous tweet is label 1). Label 0 means Non-aggressive tweet, label 1 means Aggressive tweet. Not misogynous tweet (misogynous tweet is label 0) are labeled as 0 by default.
For the test set (1,000 tweets) for subtask A provided by the AMI organizer, only the annotations on the “misogynous” and “aggressiveness” fields in the raw dataset will consider.
As shown in Figure 1, we use stratified sampling technology (StratifiedKFold), using StratifiedKFold cross-validation instead of ordinary kfold cross-validation to evaluate a classifier. The reason is that StratifiedKFold can utilize stratified sampling to divide, which can ensure that the proportion of each category in the generated training set and validation set is consistent with the original training set so that the generated data distribution disorder will not occur. In the experiment, we used 5-fold stratified sampling. For the HaSpeeDe 2 training set (Merged dataset), each of which included the randomly sampled training set (8,671) and validation set (2,168). For the AMI training set (raw dataset), each of which included the randomly sampled training set (4,000) and validation set (1,000). 3
In this part, we introduce our final submission system. Figure 2 shows the overall framework of the system we submitted to HaSpeeDe 2 Task A. We use the pre-trained multi-language model XLM-RoBERTa. We discover the limitations of BERT’s pooler output (P O) and obtained rich semantic information by extracting the hidden state (The last four hidden layers) of XLM-RoBERTa, which is used as input for Convolution Neural Network and K-max Pooling (CNN + K-max Pooling). Then, we input the output of (CNN + K-max Pooling) into the Ordered Neurons LSTM (ONLSTM). Finally, we concatenate the P O and output of ON-LSTM ON-LSTM together and pass it through the Linear layer and Softmax for the final classification.
Figure 3 shows the overall framework of the
system we submitted to AMI subtask A. We
use the pre-trained multi-language model
XLMRoBERTa. We first get pooler output (P O) and
obtained rich semantic information by extracting
the hidden state (The last four hidden layers) of
XLM-RoBERTa, which is input into Ordered
Neurons LSTM (ON-LSTM). Then, we input the
output of ON-LSTM into Capsule Network.Finally,
we concatenate the P O and output of Capsule
togetherand through the Linear layer and Softmax
for the final classification.
Early work in the field of cross-language
understanding has proved the effectiveness of
multilingual masked language model (MLM) in
crosslanguage understanding, but models such as
XLM
In the classification task, the original output of
XLM-RoBERTa is obtained through the last
hidden state of the model. However, the output
usually does not summarize the semantic content of the
input. Recent studies have shown that
abundant semantic information features are learned by the
top hidden layer of BERT
As shown in Figure 2, we input the extracted hidden states of the last four layers of XLMRoBERTa into CNN and K-max Pooling for convolution operations to obtain multiple feature maps. The specific operation: a sentence contains L words, each of which has a dimension of d after the embedding layer, and the representation of the sentence is formed by splicing the L words to form a matrix of L ∗ d. There are several convolution kernels in the convolutional layer, the size of which is N ∗ d, and N is the filter window size. The convolution operation is to apply a convolution kernel to create a new feature in a matrix that is spliced by words. Its formula is as follows:
Cl = f (w ∗ x(l : L + N − 1) + b) (1) where l represents the lth word, Cl is the feature, w is the convolution kernel, b is the bias term, and f is a nonlinear function. After the convolution operation of the whole sentence, a feature map is obtained, which is a vector of size L + N - 1.
Another important idea of CNN is pooling. The pooling layer is usually connected behind the convolution layer. The purpose of introducing it is to simplify the output of the convolutional layer and perform dimensionality reduction on the features of the Filter to form the final feature. Here is the K-max Pooling operation, which takes the value of the scores in Top K among all the feature values, and retains the original order of these feature values, that is, by retaining some feature information for subsequent use. Obviously, K-max Pooling can express the same type of feature multiple times, that is, can express the intensity of a certain type of feature; in addition, because the relative order of these Top K eigenvalues is preserved, it should be said that it retains part of the position information. However, this location information is only the relative order between features, not absolute location information. 3.3
For HaSpeeDe 2, as shown in Figure 2, we input
the output of CNN and K-max pooling into
ONLSTM. For AMI, as shown in Figure 3, We input
the extracted hidden states of the last four layers
of XLM-RoBERTa into ON-LSTM. ON-LSTM is
a new variant of LSTM, which sorts the neurons
in a specific order, allowing the hierarchical
structure (tree structure) to be integrated into the LSTM
to express richer information. The gate structure
and output structure of ON-LSTM are still similar
to the original LSTM. The difference is that the
update mechanism from cbt to ct is different. The
formula is as follows
Among them, −→cs and ←c−s are cumsum() operations in the right and left directions, respectively. the newly introduced fet and iet represent the master forget gate and master input gate respectively. wt represents a vector where the intersection part is 1 and the rest is all 0. In this way, the high-level information remains a considerable long distance, while the low-level information may be updated at each step of input, thereby embedding the hierarchical structure through information grading. 3.4
As shown in Figure 3, we input the output of
ONLSTM into Capsule. In the deep learning
model, spatial patterns are aggregated at a lower
level, which helps to represent higher-level concepts.
We use the Capsule Network
Vj = ∥ Sj ∥2
Sj 1+ ∥ Sj ∥2 ∥ Sj ∥ Sj = ∑ Cij u^jji; i u^jji = Wij ui (6) (7) where Vj is the vector output of capsule j and Sj is its total input, prediction vectors u^jji is by multiplying the output ui of a capsule in the layer below by a weight matrix Wij , the Cij are coupling coefficients that are determined by the iterative dynamic routing process.
The most fundamental difference between the Capsule network and the traditional artificial neural network lies in the unit structure of the network. For traditional neural networks, the calculation of neurons can be divided into the following three steps: 1. Perform a scalar weighted calculation on the input. 2. Sum the weighted input scalars. 3. Nonlinearization from scalar to the scalar. For the Capsule, its calculation is divided into the following four steps: 1. Do matrix multiplication on the input vector. 2. Scalar weighting of the input vector. 3. Sum the weighted vector. 4. Vector-to-vector nonlinearization. The biggest difference between the Capsule network and the traditional neural network is the unit output. The output of the traditional neural network is a value, while the output of the Capsule network is a vector, which can contain abundant features and is more interpretable.
RoBERTa-base6 pre-trained model, which contains 12 layers. We use Binary cross-entropy, Adam optimizer with a learning rate of 5e-5. The batch size is set to 32 and the max sequence length is set to 80. We extract the hidden layer state of XLM-RoBERTa by setting the output hidden States is true. The model is trained in 8 epochs with a dropout rate of 0.1.
2D convolution (nn.Conv2d7). The size of the convolution kernel is set to (3,4,5) and the number of convolution kernels is set to 256.
For the ON-LSTM, we set the hidden units to 128 and num levels to 16.
to 10, dim capsule to 16, routings to 4. 4
put of BERT is P O. In the same way, we just put P O as the output of XLM-RoBERTa.The results are shown in Table 2. We can see that the results are not good when only P O is used as the output of XLM-RoBERTa. We think that just using P O as the output will lose some effective semantic information. So we think that deep and abundant semantic features are effective for this work. We extract the hidden state of XLM-RoBERTa and we also discover that the performance of the model improves with the increase of the semantic layer.
Table 3 shows the performance of our model at different semantic layers. Table 4 shows our results on the test set. In our experiment, we find the limitations of P O for sentiment analysis of hate text in Italian languages. In the classification task, the original outIn this work, we have similar tasks as discussed in Section 4.1, and we consider the influence of P O 6https://huggingface.co/xlm-roberta-base for identifying misogyny content. We conduct ex7https://pytorch.org/docs/stable/generated/torch.nn.Conv2d periments on the AMI subtask A base on the mod
P el in HaSpeeDe 2, and in order to improve the performance, we propose a new method base on this model. Table 5 shows the comparative experimental data of the CNN + K-max Pooling + ON-LSTM method and the ON-LSTM + Capsule method. Table 6 shows the results of our new model for AMI subtask A on the test set. Run 1 only extracts the last four hidden layer states of XLM-RoBERTa and inputs them into ON-LSTM, then through the Capsule Network, and finally performs classification (without using P O). Run 2 is to concatenate the output of the Capsule Network with the obtained P O and input it to the classifier for final classification (using P O). We think that concatenate the P O and the hidden layer will retain richer semantic information and show excellent results.
(The validation set of 1-fold)
CNN + K-max Pooling + ON-LSTM (HaSpeeDe 2 Model) ON-LSTM + Capsule (AMI model)
0.786 In the experiment, we find the limitation of only using pooler output as the XLM-RoBERTa’s output. To obtain deeper and more abundant semantic features, we extract the hidden layer s
Run 2 (using P O) tate of XLM-RoBERTa. The result shows that it is helpful to improve the performance of XLMRoBERTa to obtain more abundant semantic information features by extracting the hidden state of XLM-RoBERTa. We test the effects of using the external dataset (Merged dataset) and not using the external dataset (raw dataset). Our conclusion is that using data from the same social network for training and test is a necessary condition for good performance. In addition, adding data from different social networks can improve results.