English. This paper describes the system we developed for EVALITA 2018, the 6th evaluation campaign of Natural Language Processing and Speech tools for Italian, on Hate Speech Detection (HaSpeeDe). The task consists in automatically annotating Italian messages from two popular microblogging platforms, Twitter and Facebook, with a boolean value indicating the presence or not of hate speech. We propose an Attention-based in Long Short-Term Memory Recurrent Neural Network where the attention layer helps to calculate the contribution of each part of the text towards targeted hateful messages.
Italiano. In questo articolo descriviamo il sistema che abbiamo sviluppato per il task di Hate Speech Detection (HaSpeeDe), presso EVALITA 2018, la sesta campagna di valutazione dellelaborazione del linguaggio naturale. Il task consiste nellannotare automaticamente testi italiani da due popolari piattaforme di microblogging, Twitter e Facebook, con un valore booleano indicando la presenza o meno di incitamento allodio. Il nostro approccio usa una rete neurale ricorrente LSTM attention-based, in cui il layer di attenzione aiuta a calcolare il contributo di ciascuna porzione del testo verso messaggi di odio mirati.
In recent years, Hate Speech (HS) has become a major issue as a hot topic in the domain of social media. Some key aspects (such as virality, or presumed anonymity) that characterize it, distinguish it from offline communication and make it potentially more dangerous and hurtful. Therefore, the identification of HS is an important step for dealing with the urgent need for effective counter measures to this issue.
The evaluation campaign EVALITA 20181
launched this year the HaSpeeDe (Hate Speech
Detection) task2
Deep neural network are greatly studied due
to their flexibility in capturing nonlinear
relationships. Long Short-Term Memory units (LSTM)
We propose a model that consists in a
Bidirectional LSTM neural network (Bi-LSTM) at the
word level as Figure 1 shows. At each time step
t the Bi-LSTM gets as input a word vector xt
with syntactic and semantic information, known
as word embedding
In NLP problems, standard LSTM receives sequentially (left to right order) at each time step a word embedding xt and produces a hidden state ht. Each hidden state ht is calculated as follow: input gatet = (W (i)xt + U (i)ht 1 + b(i)) f orget gatet = (W (f)xt + U (f)ht 1 + b(f)) output gatet = (W (o)xt + U (i)ht 1 + b(o)) new memt = (W (u)xt + U (u)ht 1 + b(u)) f inal memt = it ht = ot ut + ft tanh(ct) ct 1
Where all W ; U and b are parameters to be learned during training. The function is the sigmoid function and stands for element-wise multiplication.
The bidirectional LSTM, on the other hand, makes the same operations as standard LSTM but, processes the incoming text in a left-to-right and a right-to-left order in parallel. Thus, the output is a two hidden state at each time step !ht and ht .
The proposed method uses a Bidirectional LSTM network which considers each new hidden state as the concatenation of these two h^t = [!ht ,ht ]. The idea of this Bi-LSTM is to capture long-range and backwards dependencies. With an attention mechanism we allow the BiLSTM to decide which part of the sentence should “attend”. Importantly, we let the model learn what to attend on the basis of the input sentence and what it has produced so far. Figure 2 shows the general attention mechanism.
Let H 2 R2 Nh Tx the matrix of hidden states [h^1; h^2; :::; h^Tx ] produced by the Bi-LSTM, where Nh is the size of the hidden state and Tx is the length of the given sentence. The goal is then to derive a context vector ct that captures relevant information and feeds it as an input to the next level (Pos-Att-LSTM). Each ct is calculate as follow:
Tx ct = X t;t0 = tanh(Wa [h^t; st 1] + ba)
Where Wa and ba are the trainable attention weights, st 1 is the past hidden state of the PosAtt-LSTM and h^t is the current hidden state. The idea of the concatenation layer is to take into account not only the input sentence but also the past hidden state to produce the attention weights. 2.5
The goal of the Post-Att-LSTM is to predict whether the text is hateful or not. This network at each time step receives the context vector ct which is propagated until the final hidden state sTx . This vector is a high level representation of the text and is used in the final softmax layer as follow: y^ = sof tmax(Wg sTx + bg)
Where Wg and bg are the parameters for the softmax layer. Finally, cross entropy is used as the loss function, which is defined as:
L =
X yi log(y^i)
i yi is the true classification of the i-th text. 3
As run1 in M1 and M3, we first evaluated the model described before which is compound for the Bi-LSTM, the Attention layer and the LSTM (Bi-LSTM+Att+LSTM). Also, a variation in this model originated a new model for analizing the contribution of the Bi-LSTM layer. Therefore, we substituted the Bi-LSTM for a LSTM (LSTM+Att+LSTM).
Then, we processed the training sets to generate resources that we called the hate words dictionaries. For each train set we generated a dictionary of the most common words in the texts labeled as hateful. Taking into account this dictionaries, we added a linguistic characteristic to texts which defines if it contains a word into the correspondent dictionary. Thus, run 2 of the model is obtained considering this linguistic characteristic.
We used a SVM as baseline to compare the results of the different variants of the model and all variants achieved better results than this baseline.
The results show that the original model outperforms the results of the variant where the BiLSTM is not used. It is important to note that this occurs for run2 where the linguistic characteristic is taken into account. In fact, when this feature is not used the results decrease and the original model obtains the worst results in most cases. Therefore, taking into account the run2 of each variant, the results suggest that the best option is to use the Bi-LSTM with the linguistic characteristic.
The HaSpeeDe task was three sub-tasks, based on the dataset used. First, only the Facebook dataset could be used to classify the Facebook test set (HaSpeeDe-FB), where our system takes macro-average F1-score of 0.7147 and 0.7144, reaching the 11th and 10th positions for run1 and run2 of the model respectively. Another subtask was HaSpeeDe-TW, here only the Twitter dataset can be used to classify the Twitter test set, where our system takes scores of 0.6638 and 0.6567, reaching the 12th and 13th positions for run1 and run2 of the model respectively. Finally, two other tasks consisted of using one of the datasets to train and the other to classify (Cross-HaSpeeDe). Here our system takes scores of 0.4544 and 0.5436, reaching places 10th and 7th in Cross-HaSpeeDeFB and scores of 0.4451 and 0.318, for places 10th and 12th in Cross-HaSpeeDe-TW.
We think that these results can be improved with a more careful tunning of the model parameters. In addition, it may be necessary to enrich the system with linguistic resources for the treatment of the Italian language. 4
We propose an Attention-based Long Short-Term Memory Network Recurrent Neural Network for the EVALITA 2018 task on Hate Speech Detection (HaSpeeDe). The model consists of a bidirectional LSTM neural network with an attention mechanism that allows to estimate the importance of each word and then, this context vector is used with another LSTM model to estimate whether a text is hateful or not. The results showed that the use of a linguistic characteristic based on the occurrence of hateful words in the texts allows to improve the performance of the model. In addition, experiments performed on the training sets with 5-fold cross-validation suggest that the use of the Bi-LSTM layer is important when this linguistic characteristic is taken into account.
The work of the fourth author was partially supported by the SomEMBED TIN2015-71147-C21-P research project (MINECO/FEDER).