This paper describes the system we developed for IberEval 2018 on Aggressive detection track of Authorship and Aggressiveness Analysis on Twitter (MEX-A3T)1. The task focuses on the detection of aggressive comments in tweets that come from Mexican users. Systems must be able to determine whether a tweet is aggressive or not. Our approach is an Attention-based Long Short-Term Memory Network Recurrent Neural Network where the attention layer helps to calculate the contribution of each word towards targeted aggressive classes. In particular, we build a Bidireccional LSTM to extract information from the word embeddings over the sentence, then apply attention over the hidden states to estimate the importance of each word and nally feed this context vector to another LSTM model to estimate whether the tweet is aggressive or not. The experimental results show that our model achieves outstanding results.
Recurrent Neural Networks (RNNs) is a type of deep neural network designed
for sequence modeling. These kinds of models are greatly studied due to their
exibility in capturing nonlinear relationships. However, traditional RNNs su er
from problems known as exploding or vanishing gradients and, therefore, have
di culty in capturing long-term dependencies. Long Short-Term Memory
networks (LSTM) [
Moreover, attention models have become an e ective mechanism to obtain
better results [2{6]. In [
In this paper, we propose a similar Attention-based LSTM for the IberEval
2018 track on Authorship and aggressiveness analysis in Twitter (MEX-A3T)
[
The task focuses on the detection of aggressive comments. This is a topic that has not been widely studied in the community. The aim is to determine whether a tweet, which comes from Mexican users, is aggressive or not.
The paper is organized as follows. Section 2 describes our system. Experimental results are then discussed in Section 3. Finally, we present our conclusions with a summary of our ndings in Section 4. 2 2.1
In the preprocessing step, the tweets are cleaned. Firstly, the emoticons are
recognized and replaced by corresponding words that express the sentiment they
convey. Also, we remove all links and urls. Afterwards, tweets are morphologically
analyzed by FreeLing [
We propose a model that consists of a Bidirectional LSTM neural network
(BiLSTM) at the word level. At each time step t the Bi-LSTM gets as input a word
vector wt with syntactic and semantic information, known as word embedding
[
In NLP problems, standard LSTM receives sequentially (left to right order) at each time step a word embedding wt and produces a hidden state ht. Each hidden state ht is calculated as follow: Attention mechanism for aggressive detection 3 it = (W (i)xt + U (i)ht 1 + b(i)) ft = (W (f)xt + U (f)ht 1 + b(f)) ot = (W (o)xt + U (i)ht 1 + b(o)) ut = (W (u)xt + U (u)ht 1 + b(u)) ct = it ht = ot ut + ft tanh(ct)
Where all W ; U and b are parameters to be learned during training. The function is the sigmoid function and stands for element-wise multiplication.
The bidirectional LSTM, on the other hand, makes the same operations as standard LSTM but, processes the incoming text in a left-to-right and a rightto-left order in parallel. Thus, the output is a two hidden state at each time step !ht and ht .
The proposed method uses a Bidirectional LSTM network which considers ! each new hidden state as the concatenation of these two h^t = [ht ; ht ]. The idea of this Bi-LSTM is to capture long-range and backwards dependencies. With an attention mechanism we allow the Bi-LSTM to decide which part of the sentence should \attend". Importantly, we let the model learn what to attend on the basis of the input sentence and what it has produced so far.
Let H 2 R2 Nh Tx the matrix of hidden states [h^1; h^2; :::; h^Tx ] produced by the Bi-LSTM, where Nh is the size of the hidden state and Tx is the length of the given sentence. The goal is then to derive a context vector ct that captures relevant information and feed it as input to the next level (Pos-Att-LSTM). Each ct is calculate as follow:
Tx ct = X t0=1 t;t0 h^t0 t;i =
t;i PTx j=1 t;j = tanh(Wa [h^t; st 1] + ba)
Where Wa and ba are the trainable attention parameters, st 1 is the past hidden state of the Pos-Att-LSTM and h^t is the current hidden state. The idea of the concatenation layer is to take into account not only the input sentence but also the past hidden state to produce the attention weights. 2.5
The goal of the Post-Att-LSTM is to predict whether the tweet is aggressive or not. This network at each time step receives the context vector ct which is propagated until the nal hidden state sTx . This vector is a high level representation of the tweet and is used in the nal softmax layer as follow: 4 y^ = sof tmax(Wg
sTx + bg)
Where Wq and bq are the parameters for the softmax layer. Finally, cross entropy is used as the loss function, which is de ned as:
L =
X yi log(y^i)
i Where yi is the true classi cation of the tweet. (1) 3
Table 1 shows the results obtained by the proposed method on the aggressive
class for two di erent runs (run1 and run2). A variation in the model was realized
for run2, where a linguistic characteristic is added to tweets. This characteristic
is based on the study carried out in the work [