Overview of MEX-A3T at IberLEF 2020: Fake News and Aggressiveness Analysis in Mexican Spanish

Abstract

This paper presents the overview of MEX-A3T 2020, the third edition of this lab under the IberLEF conference. The main purpose of MEX-A3T is to explore diferent methodologies and strategies related to the analysis of social media content in Mexican Spanish. This year edition focuses in the identification of fake news and the detection of aggressive tweets. For this purpose, we provided diferent news from verified web sources and a corpus of tweets from Mexican users.

Keywords

• eol>Fake news detection
• aggressiveness detection
• MEX-A3T
• IberLEF

1. Introduction

tweets in Mexican Spanish. However, for this year, the criteria for identifying aggression have been revised and a new enhanced data set has been created.

Fake News Detection Track: Fake news provide information that aims to manipulate people for diferent purposes: terrorism, political elections, advertisement, among others. In social networks, misinformation extends in seconds among thousands of people, so it is necessary to develop tools that help control the amount of false information on the web. Particularly, fake news detection systems aim to help users to detect and filter out potentially deceptive news. The Fake News Detection Track consists in classifying a given set of news written in Mexican Spanish between true and fake.

The remainder of this paper is as follows: Secti2ocnovers a brief description of the previous edition of MEX-A3T. Section3 presents the evaluation framework used at MEX-A3T 2020. Section4 shows an overview of the participating approaches. Sect5iorneports and analyzes the results obtained by the participating teams. Finally, Sect6iopnresents our conclusions from this evaluation exercise. 2. MEX-A3T 2019 MEX-A3T is a forum for the analysis of social media content in Mexican Spanish. Last year, we organized the second edition of the MEX-A3T shared tas1k],[focusing on the problems of author profiling and aggressiveness detection. A variety of methods were proposed by the participants, comprising content-based (bag of words, word n-grams, term vectors, dictionary words, and so on), stylistic-based features (frequencies, punctuation, POS, Twitter-specific elements, slang words, and so forth), and approaches based on neural networks (CNN, LSTM, and others).

For author profiling, as a novelty of previous year’s edition, it was considered the use of text and images as information sources. Our purpose was to study the relevance and complementarity of multimodal data for profiling social media users. Sadly, participants could not find an efective way to take advantage of both types of information, and did not outperform the baselines proposed.

In the case of the aggressiveness identification, the top-ranked team was UACh2][. This team used two main kinds of features, character n-grams and word embeddings, and employed two diferent classifiers, an SVM and a multilayer perceptron. The main idea of their participation was the inclusion of features for giving context to the text messages, and to explore if people verbally attack diferently depending on their traits and overall environment.

3. MEX-A3T 2020 Evaluation Framework

This section outlines the construction of the two used corpora, highlighting particular properties, challenges, and novelties. It also presents the evaluation measures used for both tasks. Aggressive Non-Aggressive Y por que disculpa? El otro joto comenzó..q.ue tiene de especial la tonta texas Le hubieras dado dos, no mas por joto y metiche Yo voy cualquiera que no seas tu @USUARIO No puedo creer que nos sig@USUARIO . Viejo ladrón HDP. amos matando por tontas ideologías Indios estupidos no saben pa que putas las tontas no van al cielo es mi religión es la pasarela... una vez vi cruzar unas cabras por ahi.. no entiendo como ellas si entienden

3.1. Aggressiveness Detection

Social networks represent a significant threat to users exposed to many risks and potential attacks. One such threat is aggressive comments, which can produce long-term harm to victims, and in some cases, they can lead to suicide. This track focuses on the detection of aggressive comments on Twitter, a topic with little study in the Ibero-American community. Participants have to develop methods to determine whether a tweet is aggressive or not. The track is challenging by the fact that tweets come from Mexican users with a variety of backgrounds and social expressions.

We built a corpus of tweets for the task of aggressiveness detection from Mexican accounts. First, we selected a set of terms that served as seeds for extracting the tweets. We used the words classified as vulgar and non-colloquial inDthicecionario de Mexicanismos de la Academia Mexicana de la Lengua, as well as words and hashtags identified by thIenstituto Nacional de las Mujeres. Tweets were collected considering their geolocation. We considered Mexico City as the center and extracted all tweets that were within a radius of 500 km. We annotated the corpus using the scheme proposed in3[]. The annotation provides a specific criteria to separate a tweet from aggressive, ofensive and vulgar, based on the linguistic characteristics and intent of the message. Table1 shows some examples labeled as aggressive and non-aggressive. As can be intuited, the task of labeling aggressiveness is challenging, especially because in most cases it is necessary to interpret the message in a given context.

The collected corpus consists of more than 10 thousand tweets. For the evaluation exercise, we divided the corpus into two parts, one for training and the other for the test. T2able shows the distribution of this corpus. The non-aggressive class is the majority class in both partitions. For readers interested in more detai4l]sd,[escribes the methodology followed for the construction of the Mexican Aggressiveness Corpus.

3.2. Fake News Detection

The Spanish Fake News Corpus is a collection of news compiled from several web sources: established newspaper websites, media companies websites, special websites dedicated to

Class

Not Aggressive

Aggressive Σ validating fake news, websites designated by diferent journalists as sites that regularly publish fake news. The news were collected from January to July of 2018 and all of them were written in Mexican Spanish 5[]. The assembled corpus has 971 news.

The corpus was manually labeled using two classes (true or fake), and considering the following criteria: • A news report is true if there is evidence that it has been published in reliable sites. • A news report is fake if there are news from reliable sites or specialized websites in the detection of deceptive content that contradict it, or if no other evidence was found about the news besides the given source.

The data collection includes true-fake news pairs of diferent events to have a corpus as balanced as possible. Additionally, in order to avoid topic bias, the corpus covers news from 9 diferent topics: Science, Sport, Economy, Education, Entertainment, Politics, Health, Security, and Society. As can be seen in Tabl3e, the number of fake and true news is balanced; approximately 70% are used as training corpus (676 news), and 30% as the test corpus (295 news). For readers interested in more detail5s], [describes the methodology followed for the construction of the Spanish Fake News corpus.

Training corpus Testing corpus

True Fake True Fake Category Science Sport Economy Education Entertainment Politics Health Security Society Σ 32 45 18 6 48 121 16 11 41 338 30 41 12 9 55 105 16 18 52 338 14 21 6 4 22 54 7 6 19 153 142 13 17 7 3 23 43 7 7 22

Approach

Transformers X Traditional Deep Neural Networks

BoW, n-grams, Stylometrics X

M A U p a i d I

X

s ITAM tsenno C I

D S G s

ITC erA X X X

3.3. Performance Measure

For both tracks, the final score corresponds to th1e-measure for the target class, that is, fake news and aggressive messages respectively.

4. Overview of the Submitted Approaches

At this edition, eleven teams submitted one or more solutions; six teams participated in the fake news detection task, and nine participated in the aggressiveness identification task. This section presents a summary of their approaches regarding preprocessing steps, features, and classification algorithms. In Tabl4e we indicate the general approach used for each team. It can be appreciated that participants used three general approaches: transformers, deep neural networks, and traditional representations like BoW and n-grams feeding a SVM classifier. Following, we briefly describe each of the participating methods.

• Idiap and UAM Participation at MEX-A3T Evaluation Campaign [ 6 ] – Tasks: Fake News Detection; Aggressiveness Detection. – Team name: Idiap-UAM – Summary: The authors used a Supervised Autoencoder (SAE), that is, a neural network that learns a representation (encoding) of input data and then learns to reconstruct the original input. They used three diferent types of features as inputs representation: word n-grams, char n-grams, and BETO encodings. The best performance was obtained when the autoencoder was fed with the combination of the three input representations. • Transformers and Data Augmentation for Aggressiveness Detection in Mexican Spanish [ 7 ] – Tasks: Fake News Detection; Aggressiveness Detection. – Team name: CIMAT – Summary: The authors proposed two diferent strategies for the aggressiveness detection task. The first strategy consisted of an ensemble of diferent BETO models (BERT models trained in Spanish) with majority and weighted voting schemes. The second strategy considered data augmentation, a technique to generate new instances from the original training data. They reported as best strategy the use of 20 ensemble models and adversarial data augmentation, where the model creates a new input for each misclassified sentence. • ITCG’s participation at MEX-A3T 2020: Aggressive Identification and Fake News detection based on textual features for Mexican Spanish [ 8 ] – Tasks: Fake News Detection; Aggressiveness Detection. – Team name: Intensos – Summary: The authors presented a traditional text classification approach, using a combination of binary and tf-idf text representations. They reported that their best result was using a SVM with this representation, without removing stop words. • TecNM at MEX-A3T 2020: Fake News and Aggressiveness Analysis in Spanish Mexican [ 9 ] – Tasks: Fake News Detection; Aggressiveness Detection. – Team name: ITCG-SD – Summary: The authors presented a traditional machine learning approach, using a bag-of-words representation with TF and TF-IDF weights. Their best results were obtained when applying a neural network and a SVM classifier. • UPB at MEX-A3T 2020: Detecting Aggressiveness in Mexican Spanish Social Media Content by Fine-tuning Transformer-Based Models [ 10 ] – Tasks: Aggressiveness Detection – Team name: UPB – Summary: The authors presented diferent approaches to fine-tune pre-trained Spanish, English, and multilingual transformer-based models. The best result they reported was using BETO, a BERT model trained in Spanish, but fine-tuned with the MEX-A3T aggressiveness train set and the HatEval Spanish dataset. • UACh at MEX-A3T 2020: Detecting Aggressive Tweets by Incorporating Author and Message Context [ 11 ] – Tasks: Aggressiveness Detection – Team name: UACh – Summary: The authors explored the idea of using context information, such as message and author metadata. Their proposed approach has two stages. In the first stage, messages were classified considering only their content; this classification was done using BETO. Then, in the second stage, the predictions of the first stage were concatenated with the author and message metadata to form a new representation vector, which was employed by a XGBoost classifier. • GRU with Author Profiling Information to Detect Aggressiveness [ 12 ] – Team name: DeepMath – Summary: The authors presented a bi-directional GRU model using words as inputs. The output of this model was combined with the predictions of gender and occupation of users obtained by a reference model, using a simple concatenation and considering a one-hot-encoding. At the end, the model considered only the gender and Sciences-Student occupation categories; the rest of the categories were discarded by a chi-squared feature selection criterion. • UMUTeam at MEX-A3T’2020: Towards Aggressiveness Identification in Mexican-Spanish tweets with linguistic features and word-embeddings [ 13 ] – Tasks: Aggressiveness Detection – Team name: UMUTeam – Summary: The authors evaluated the characterization of aggressive messages through a set of linguistic attributes and sentence-embeddings. They used two types of classifiers, a support vector machine and two types of deep neural networks. Their best result was obtained by a Bi-LSTM network trained with FastText embeddings and combined with linguistic features. • Detecting Aggressiveness in Mexican Spanish Tweets with LSTM + GRU and LSTM + CNN Architectures [ 14 ] – Tasks: Aggressiveness Detection – Team name: UGalileo – Summary: The authors proposed the use of two diferent architectures based on deep learning models. The first architecture consisted of a Bi-GRU and a Bi-LSTM networks, where the outputs are concatenated and then a prediction layer is added. For the second architecture, the authors used a Bi-LSTM and CNN network, then a concatenation and a prediction layer. Both architectures achieved similar results over the test dataset partition. • Ares Team: No system description paper – Tasks: Fake News Detection – Team name: Ares – Summary: The authors proposed the use of a TF-IDF representation, combined with the capital letter ratio in the article, total number of words in the body of the article, and percentage of coincidence between the words of the body and the headline of the article. The variable selection algorithm is an F-test, and a linear algorithm with training through SGD classification

5. Experimental evaluation and analysis of results

This section summarizes the results obtained by the participants of MEX-A3T 2020, comparing and analyzing in detail the performance of their submitted solutions. For the final phase of the challenge, participants sent their predictions for the test partition, the performance on this data was used to rank them. We used the F1 over the interest class as the main evaluation measure.

For computing the evaluation scores we relied on the EvALL platfo1r5m]. [EvALL is an online evaluation service targeting information retrieval and natural language processing tasks. It is a complete evaluation framework that receives as input the ground truth and the predictive outputs of systems and returns a complete performance evaluation. In the following subsections, we report the results obtained by participants as evaluated by EvALL and an analysis of their results.

As baseline methods, we implemented two popular approaches that have shown to be hard to beat in both tasks:i) a classification model trained on the bag of words (BoW) representation, and ii) a Bi-GRU neural network. Also, we compared the systems’ results against the result from INGEOTEC, the best performing system at the first MEX-A3T edition [ 16 ].

For both classification tasks the BoW approach was applied, in which we used all vocabulary from the corpora, removing stopwords and special characters. The size of the representation of each text was 14,913 for fake news detection, and 5,212 for aggressiveness identification; for classification we used a SVM classifier with linear kernel a nd= 1. On the other hand, we also applied a Bi-GRU neural network in the task of aggressiveness identification. In this approach texts were pre-processed by removing stopwords, special characters, and converting all emojis to words (e.g,. - ‘cara sonriente’). As input features pre-trained Spanish FastT1e7x]t[ embeddings were used, and a fully-connected softmax layer handle the class probabilities.

5.1. Aggressiveness detection results

Table5 presents the results obtained by the teams in the aggressiveness detection task. For this task, we sort the teams by thei r 1 results over the aggressive class. For extra analysis, we also report the accuracy, the mac r1oand the 1 in the non-aggressive class. The approach submitted by the CIMAT team obtained the best performance, outperforming all teams, and the proposed baselines.

To analyze in more detail the participants’ results, we focused on the analysis of the complementariness and diversity of their predictions. To measure the complementarity, we used the Maximum Possible Accuracy (MPA) metric, which is defined as the quotient of the correctly classified instances over the total number of test instances. We considered an instance as correctly classified iaft least one of the participating teams classified it correctly. On the other hand, to measure the diversity we used the Coincident Failure Diversity (CFD) metric [ 18 ], which focuses on calculating the error diversity among the participants predictions. The minimum value of this measure is 0 when all teams simultaneously predict a pattern correctly or wrongly, while the maximum value is 1, when the misclassifications are all unique.

Table6 shows the results of applying the Maximum Possible Accuracy, and the Coincident Failure Diversity metrics over all participating teams and the diferent approaches in the aggressiveness identification task. From these results, it is possible to observe that the MPA

Aggressive Non aggressive 0.7998 0.9195 0.8596 0.7971 0.9205 0.8588 0.7969 0.9107 0.8538 0.7720 0.9042 0.8381 0.7468 0.8933 0.8200 0.7255 0.8886 0.8071 0.7124 0.8841 0.7983 0.7066 0.8953 0.8010 0.7062 0.8861 0.7961 0.7001 0.8544 0.7773 0.6957 0.8537 0.7747 0.6760 0.8780 0.7770 0.6727 0.8706 0.7716 0.6619 0.8752 0.7686 0.6516 0.8771 0.7644 0.6388 0.8208 0.7298 0.6387 0.8430 0.7408 0.6080 0.8820 0.7450 0.5892 0.8430 0.7161 0.3437 0.8463 0.5950 0.2515 0.7664 0.5090 from all teams and from the teams using the diferent types of approaches is considerably greater than the best performance Accuracy obtained by the CIMAT team, suggesting that the participants systems and approaches are complementary to each other. In terms of the diferent approaches, the Transformes approach obtained the greater MPA over the BoW and traditional DNN approaches, which is consistent with the results shown in Tab5,lewhere the top-teams obtained their best performances using transformers. The results obtained with the CFD metric, show that there is a high error diversity in the predictions of the DNN approaches, which is consistent with the performance shown by this approach using the MPA metric. On the other hand, the approach that obtained the best performance results with the MPA metric, showed less error diversity in the participants predictions.

Approach

All teams

Transformers BoW and n-grams Traditional DNN

As a result of applying the MPA metric, it was possible to identify those common errors across all systems. In fact, there are only 10 tweets that no system could classify correctly. All of them are aggressive tweets that were classified as non-aggressive. Below we present some of these tweets, where we can identify ofenses with no vulgar or profane words, the use of out of the training vocabulary words, as well as some ironic comments.

• No se vale estar “buena” por gorda y “flaca” por anoréxica, chingenle al gym y a la dieta. • Viendo a #AnetteMichel con ese vestido en #MasteChefMx se me hace bien gorda. • @USUARIO te quiero meter todo mi camote. • Las fans de odisseo se ven bien bonitas en sus fotos de twitter y estan bien feas en persona. • #InyeccionLetalParaLGBT y para la que me ilusiono, luego me echó la culpa de que le rompi el corazón, me dejó, y volvió con su ex... ardida!

5.2. Fake News Detection results

Table7 shows a summary of the results obtained by each team in the Fake News detection track. As in the previous section, we report th1e in both fake and true classes, the macro 1, and the accuracy. We used th e1 over the fake class to rank participants. In this task, the approach submitted by the Idiap-UAM team outperformed all the other approaches and the baselines. It can be observed that all systems achieved balanced results in both fake and true classes, however, the1 score of the true class is in general slightly better in almost all systems. All participated teams used a machine-learning-based approach relying on style-based features, i.e., neither team used a knowledge base or Web searching to verify the authenticity of the news.

Team

Idiap-UAM-1 Idiap-UAM-2

Ares

CIMAT-1 Baseline (BoW-RF )

Intensos-2

Intensos-1 Baseline (INGEOTEC)

ITCG-SD

The analysis of the complementariness and the diversity of the predictions of the diferent approaches using the MPA and CFD metrics are shown in the Tab8l.eThe table uses the following hierarchy for the participants: Transformers approach considers only the CIMAT team, BoW and n-gram approach considers Ares, Intensos and ITCG-SD teams, Hybrid approaches includes Idiap-UAM teams. The CFD for the Transformers methodologies row could not be calculated because there is only one participant. The MPA for the row of all teams has the highest value, which means that the teams’ approaches complement each other.The best systems (Idiap-UAM 1.2) obtained a lower MPA value, around 9%, compared to that of all teams. On the contrary, the systems with BoW and n-grams approach obtained an MPA value similar to that of all the teams, showing greater complementariness in the proposed approaches. The teams that implemented BoW and n-gram approaches showed a greatest diversity of errors in their predictions, this is consistent with their MPA performance and the heterogeneity of the approaches. The lowest value for the CFD score corresponds to the Idiap-UAM runs, which means that their predictions are alike, this can be explained because both runs use the same core.

Approach

All teams Hybrid (Idiap-UAM 1,2)

Transformers (CIMAT) BoW, n-grams (Intensos 1,2 + ITCG + Ares)

The Table9 shows the results of th e 1 score for the fake class in the diferent topics of the corpus. It can be observed that the Economy category is the most dificult for all the evaluated approaches. On the contrary, there were three systems that correctly classified all the instances in the Education topic, and two systems that achieved perfect scores in the the Security topic. The performance of the systems does not seem related to the number of news each topic has. Politics, Entertainment, Sport and Science are the largest topics, while Education, Health and Security are the less represented groups. However, it seems that the most dificult topic to identify was economy, although it could seem that, having more examples, could help the system to learn better.

We identified the common prediction errors across all the systems and find that there were only 8 news, 7 in the fake class that none of the approaches classify correctly. 1T0asbhloews the classified instances, it can be observed that the 37.5% of the missclassified news belong to the Economy category, while the 25% are included in the group of politics. The groups of society, science and health, show one entry that has not been correctly classified by any team (12% of the total).

6. Conclusions

This paper described the design and results of the MEX-A3T shared task collocated with IberLef 2020. MEX-A3T stands forAuthorship and Aggressiveness Analysis in Mexican Spanish Tweets. Two tasks were proposed, one targeting fake news detection and the other focused on aggressiveness detection.

Regarding aggressiveness detection, this has been the third edition of the task, and this n o i t a c u

Team Ed Idiap-UAM-1 1.00 Idiap-UAM-2 1.00

Ares 1.00 CIMAT-1 0.86

BoW-RF 0.86 Intensos-2 0.67 Intensos-1 0.75 INGEOTEC 0.88 ITCG-SD 0.75 year the results have outperformed the past competitions. Clearly, the use of transformers has achieved the best results, and shows the appropriateness of this method for approaching this key topic in NLP.

Although this has been the first edition of the task in fake news detection, the results that have been achieved are really promising. Contrary to the task of aggressiveness detection, the best results here have been reached by hybrid approaches, using both transformers and BoW-n-grams, or just n-grams. Traditional Deep Neural Networks have not been used in this task.

Summing up, the achievements of these tasks of the IberLef evaluation forum showed how some key topics in NLP using Spanish as a source language have experienced a great development in recent years. Both data compilation and the use of cutting-edge methods, have placed Spanish among the languages with the most accurate applications in the area of natural language processing.

Acknowledgments

Our special thanks go to all of MEX-A3T’s participants. We would like to thank CONACyT for partially supporting this work under grants CB-2015-01-257383, FC-2016-2410, CB-A1-S-27780, the Thematic Networks program (Language Technologies Thematic Network), and UNAM under PAPIIT projects IA401219, TA100520. The first author thanks for doctoral scholarship CONACyT-Mexico 654803 and the second for master scholarship CONACyT-Mexico.

References

[1] M. E. Aragón , M. Á . Álvarez-Carmona , M. Montes-y Gómez , H. J. Escalante , L. VillaseñorPineda, D. Moctezuma, Overview of mex-3at at iberlef 2019 : Authorship and aggressiveness analysis in mexican spanish tweets , in: Notebook Papers of 1st SEPLN Workshop on Iberian Languages Evaluation Forum (IberLEF) , Bilbao, Spain, September, 2019 , p. .

[2] M. Casavantes , R. López , L. C. González , Uach at mex-a3t 2019 : Preliminary results on detecting aggressive tweets by adding author information via an unsupervised strategy , in: In Proceedings of the First Workshop for Iberian Languages Evaluation Forum (IberLEF 2019 ), CEUR WS Proceedings , 2019 , p. .

[3] M.-J. Díaz-Torres, P. A. Moran-Méndez , L. Villasenor-Pineda , M. Montes-y Gomez , J. Aguilera , L. Meneses-Lerin , Automatic detection of ofensive language in social media: Defining linguistic criteria to build a mexican spanish dataset , in: Proceedings of the Second Workshop on Trolling, Aggression and Cyberbullying , 2020 , p. .

[4] M. Á . Álvarez-Carmona , E. Guzmán-Falcón , M. Montes-y Gómez , H. J. Escalante , L. Villaseñor-Pineda , V. Reyes-Meza , A. Rico-Sulayes , Overview of mex-a3t at ibereval 2018: Authorship and aggressiveness analysis in mexican spanish tweets , in: Notebook Papers of 3rd SEPLN Workshop on Evaluation of Human Language Technologies for Iberian Languages (IBEREVAL) , Seville, Spain, September, 2018 , p. .

[5] J.-P. Posadas-Durán , H. Gómez-Adorno , G. Sidorov , J. J. M. Escobar , Detection of fake news in a new corpus for the spanish language , Journal of Intelligent & Fuzzy Systems 36 ( 2019 ) 4869 - 4876 .

[6] E. Villatoro-Tello , G. Ramírez-de-la Rosa , S. Kumar , S. Parida , M. Petr , Idiap and uam participation at mex-a3t evaluation campaign , in: Notebook Papers of 2nd SEPLN Workshop on Iberian Languages Evaluation Forum (IberLEF) , Malaga, Spain, September, 2020 , p. .

[7] M. Guzman-Silverio , A. Balderas-Paredes , A.-P. López-Monroy , Transformers and data augmentation for aggressiveness detection in mexican spanish , in: Notebook Papers of 2nd SEPLN Workshop on Iberian Languages Evaluation Forum (IberLEF) , Malaga, Spain, September, 2020 , p. .

[8] D. Zaizar-Gutierrez , D. Fajardo-Delgado , M. - A. Álvarez Carmona , Itcg's participation at mex-a3t 2020: Aggressive identification and fake news detection based on textual features for mexican spanish , in: Notebook Papers of 2nd SEPLN Workshop on Iberian Languages Evaluation Forum (IberLEF) , Malaga, Spain, September, 2020 , p. .

[9] S. Arce-Cardenas , D. Fajardo-Delgado , M. - A. Álvarez Carmona , Tecnm at mex-a3t 2020 : Fake news and aggressiveness analysis in spanish mexican , in: Notebook Papers of 2nd SEPLN Workshop on Iberian Languages Evaluation Forum (IberLEF) , Malaga, Spain, September, 2020 , p. .

[10] M.-A. Tanase , G.-E. Zaharia , D.-C. Cercel, M. Dascalu , Upb at mex-a3t 2020 : Detecting aggressiveness in mexican spanish social media content by fine-tuning transformer-based models , in: Notebook Papers of 2nd SEPLN Workshop on Iberian Languages Evaluation Forum (IberLEF) , Malaga, Spain, September, 2020 , p. .

[11] M. Casavantes , R. López , L.-C. González, Uach at mex-a3t 2020 : Detecting aggressive tweets by incorporating author and message context , in: Notebook Papers of 2nd SEPLN Workshop on Iberian Languages Evaluation Forum (IberLEF) , Malaga, Spain, September, 2020 , p. .

[12] M.-G. Garrido-Espinosa , A. Rosales-Pérez , A.-P. López-Monroy , Gru with author profiling information to detect aggressiveness , in: Notebook Papers of 2nd SEPLN Workshop on Iberian Languages Evaluation Forum (IberLEF) , Malaga, Spain, September, 2020 , p. .

[13] J.-A. García-Díaz , R. Valencia-García , Umuteam at mex-a3t' 2020 : Towards aggressiveness identification in mexican-spanish tweets with linguistic features and word-embeddings , in: Notebook Papers of 2nd SEPLN Workshop on Iberian Languages Evaluation Forum (IberLEF) , Malaga, Spain, September, 2020 , p. .

[14] V. Peñaloza , Detecting aggressiveness in mexican spanish tweets with lstm + gru and lstm + cnn architectures , in: Notebook Papers of 2nd SEPLN Workshop on Iberian Languages Evaluation Forum (IberLEF) , Malaga, Spain, September, 2020 , p. .

[15] E. Amigó , J. Carrillo-de Albornoz , M. Almagro-Cádiz , J. Gonzalo , J. Rodríguez-Vidal , F. Verdejo , Evall: Open access evaluation for information access systems , in: Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval , ACM, 2017 , pp. 1301 - 1304 .

[16] M. Graf , S. Miranda-Jiménez , E. S. Tellez , D. Moctezuma , V. Salgado , J. Ortiz-Bejar , C. N. Sánchez , Ingeotec at mex-a3t: Author profiling and aggressiveness analysis in twitter using tc and evomsa , in: In Proceedings of the Third Workshop on Evaluation of Human Language Technologies for Iberian Languages (IberEval 2018 ), CEUR WS Proceedings , 2018 , p. .

[17] E. Grave , P. Bojanowski , P. Gupta , A. Joulin , T. Mikolov , Learning word vectors for 157 languages , in : Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018 ), 2018 , p. .

[18] E. K. Tang , P. N. Suganthan , X. Yao , An analysis of diversity measures , Mach. Learn . 65 ( 2006 ) 247 - 271 . URL:https://doi.org/10.1007/s10994-006-9449- 2. doi: 10 .1007/ s10994- 006- 9449- 2.