This paper presents a description of our participation in task \MEX-A3T: Authorship and aggressiveness analysis in Twitter: case study in Mexican Spanish" at Iberian Languages Evaluation Forum (IberLEF) 2019. This work is focused on studying the detection of aggressiveness on Spanish tweets. For a ner-grained study of the task, we analyzed three di erent approaches: the rst two approaches consider the design of architectures using convolutional and recurrent neural networks. Meanwhile, the third approach is focused on pay attention to certain important parts of a sentence using self-attention networks.
The shared task \MEX-A3T: Authorship and aggressiveness analysis in Twitter:
case study in Mexican Spanish"[
Considering the systems proposed in the last edition of MEX-A3T(2018)[
This paper is organized into four sections: the rst one presents an introduction to this study. The second section provides a description of the proposed systems. The third one explains the experiments conducted and the results achieved by our models and comments the o cial results. And the last section shows some conclusions. 2
As it is known, CNNs allow us to process a sentence using di erent n-grams
at a time to recognize some patterns or information that can be useful for text
classi cation. Also, LSTM and GRU [
This paper considers three main approaches: the rst one takes into account a bunch of architectures which use CNNs as the base layer to handle n-grams. Basically, this architecture is composed of an input embedding layer, followed by an spatial dropout and a convolutional layer with a set up of 256 lters and kernel size of 2, 3 or 4. Next, a GlobalMaxPooling1D is de ned (or a Maxpooling to add a LSTM or GRU layers) and nally, a dense layer with softmax is used to generate the results.
The second approach takes into account a stack of di erent layers: rstly, three CNNs are de ned, the rst CNN considers bigrams, the second one considers trigrams, and the last one considers quadrigrams. All these CNNs are attened for feeding a dense layer (of 256 nodes). Finally a last dense layer with softmax is used to obtain the predictions.
The third approach considers the use of self-attention networks [
In this section all the experiments carried out in this task are commented as well as the results obtained.
For Aggressive detection, the organizers of MEX-A3T provided a training set of 7700 tweets written in Spanish. (labeled with two possible classes: 0 = \non-aggressive\(NOA), 1 = \aggressive\(AGG)). Regarding the distribution of the classes we can observe that these are not balanced (the NOA class (65%) and the AGG class (35%)). In order to evaluate the performance of the proposed systems, 3156 unlabeled tweets were provided as a test set.
We applied the following text-preprocessing to all the tweets that are the
input to our systems: we remove all URLs, numbers, users, times, dates, emails,
percents [
Furthermore, 5-fold cross-validation sets were used to avoid over- tting in all the experiments. The o cial evaluation metric to evaluate the systems was F1 score for aggressiveness (f1 aggr). However, we show in Table 1 and Table 2 some additional metrics, accuracy (acc.) and macro f1 to have a better interpretation of the results. pared to other architectures that follow this rst focus (see Table 1, rows 1 to 9).
As we mentioned in Section 2. The second approach takes into account a variety of layers. the main idea behind this approach is to vary the number of inputs with its respective CNNs. Also, we want to vary the number of nodes of the next dense layer which process the outputs of the previous CNNs. Firstly, we used only a CNN to process bigrams, then we added a dense layer of 128 nodes plus another dense layer with two nodes with softmax. The next tested architecture kept the same structure, however, we added another input and its CNN to process trigrams. And for the last tested architecture, we added another CNN to process the quadrigrams. Summing up, the system called stacked (row 10 in Table 1) which reached good results use MexE embeddings and it is structured by three inputs and its respective CNNs, a dense layer of 256 nodes and a last dense layer of two nodes.
To work and make experiments with the third approach based on SANs, rstly, we de ned the architecture commented on Section 2. After de ning the system, we set up some parameters to see the importance and impact that those have over the performance reached by the system. For instance, one interesting fact was the de nition of the Dropout in the inputs to sentence-level on the encoder, we tried on a range of 0.2 to 0.8, and we noticed that a dropout of 0.7 achieved the best performance in this particular case, meaning that hiding a huge part of the sentence helps a lot to reach good results. Another important consideration, it was taken into account the unbalanced training dataset, to face this fact we set up the experiments to treat every instance of the class NOA as two instances of the class AGG (minority class), it means that we are assigning higher values to AGG instances for having a balanced training dataset. Therefore, having de ned our system named attention (see row 11 in Table 1) and the parameters tuned, all this combined with the Twitter87 embeddings allow us to reach the highest results for f1 agg. 3.1
O
cial Results
Table 2 shows the o cial results published by the organizers of MEX-A3T-2019.
For the aggressiveness detection subtask, we submitted two runs, both
submissions obtained worst results than the baselines proposed by the organizers. The
Run1 (attention) was the submission which reached the best result, this
system uses SANs as explained on Section 2. Meanwhile, Run2 (cnn b) achieved
low results, all of them below bag-of-words (BoW) [
Some interesting fact that we noticed was the performance of our third approach based on SANs, this system worked really good on the training dataset. But it did not reach good results on the test dataset. We assume that this issue is due to the fact that the content of the test data is too di erent from the training set. It means that our system has trained and learned some patterns that would not be founded on the test data.
Team f1 aggr macro f1 acc.
Baseline (Trigrams) 0,4300 0,6080 0,6688 Our System Run1 0,4081 0,4897 0,5029 Baseline (BoW ) 0,3690 0,5760 0,6777
Our System Run2 0,2921 0,5122 0,6115 4
In this work we have presented our participation in the MEX-A3T-2019 shared task focused on aggressiveness detection. We proposed three approaches: the rst two approaches consider the use of convolutional and recurrent neural networks to compute n-grams and long-term dependencies in tweets. The third approach considers the use of self-attention networks to pay close attention to some words of the tweet. Based on the experiments and the results obtained in this paper, we noticed that the architectures proposed on the rst two approaches reached slightly similar results. But making a comparison between the rst two approaches and the last approach, we see good results reached by selfattention networks. Therefore, we have observed that a deeper study on SANs can help us to improve our results.
This work has been partially supported by the Spanish MINECO and FEDER founds under project AMIC (TIN2017-85854-C4-2-R) . Work of Jose-Angel Gonzalez is also nanced by Universitat Politecnica de Valencia under grant PAID-01-17.