SINAI at IberLEF-2021 DETOXIS task: Exploring Features as Tasks in a Multi-task Learning Approach to Detecting Toxic Comments

Abstract

This paper describes the participation of the SINAI research group at DETOXIS (DEtection of TOxicity in comments In Spanish) shared task at IberLEF 2021. The proposed system follows a Multitask Learning approach where multiple tasks related to toxic comments identi cation are learned in parallel while using a shared representation. Speci cally, we use the dataset features provided by the organizers as tasks along with the combination of polarity classi cation, emotion classi cation and o ensive language detection tasks to explore if they help in the identi cation of toxic comments. Our proposal ranked rst in both DETOXIS subtasks, toxicity detection and toxicity level detection.

Keywords

• Multi-Task Learning
• BERT
• Toxic Features
• Sentiment Analysis

-

Toxic comment classi cation is a eld of research that has attracted increasing interest in the Natural Language Processing (NLP) community in recent years. In this task, the organizers de ned a toxic comment as \a comment that denigrates, hates or vili es, attacks, threatens, insults, o ends or disquali es a person or group of people based on characteristics such as race, ethnicity, nationality, political ideology, religion, gender and sexual orientation, among others". Therefore, toxicity term will be used as an umbrella term to include di erent de nitions used in the literature to describe hate speech [ 5, 4 ], abusive [ 20 ], aggressive [ 14 ], and o ensive language [ 28 ]. In fact, these di erent terms address di erent aspects of toxic language [ 24 ].

Detecting online toxicity may be di cult, as it can be expressed in di erent ways: explicitly (through insult, mockery or inappropriate humor) or implicitly (through sarcasm). Another aspect to take into account is the the presence of di erent levels of intensity in toxicity (from rude and o ensive comments to more aggressive ones, the latter being those that incite hatred or even physical violence).

In this paper, we present the systems we developed as part of our participation in the DETOXIS (DEtection of TOxicity in comments In Spanish) shared task [ 26 ], at IberLEF 2021 [ 19 ] in both subtasks. The aim of DETOXIS is the detection of toxicity in comments posted in Spanish in response to di erent online news articles related to immigration. The DETOXIS task is divided into two related classi cation subtasks: (1) toxicity detection and (2) toxicity level detection. The rst subtask consists of detecting whether or not a comment is toxic while the second one aims to categorize the comment according to four levels of toxicity (0: not toxic, 1: mildly toxic, 2: toxic, and 3: very toxic).

The rest of the paper is structured as follows. In Section 2 we explain the data used in our experiments. In Section 3, we describe our proposal to address the task. In Section 4 and 5, we present the experiment setup and results, respectively. Finally, the conclusion is presented in Section 6. 2

Corpora

To run our experiments, we used the Spanish dataset provided by the organizers of the DETOXIS task at IberLEF 2021. The DETOXIS dataset was collected from the NewsCom-TOX dataset. This dataset consists of 4,357 comments (approximately) posted in response to di erent articles extracted from Spanish online newspapers (ABC, elDiario.es, El Mundo, NIUS, etc.) and discussion forums (such as Meneame) from August 2017 to July 2020. These articles were manually selected taking into account their controversial subject matter, their potential toxicity and the number of comments posted (minimum 50 comments). A keyword-based approach was used to search for articles primarily related to immigration. Comments were selected in the same order in which they appear in the web timeline. The author (anonymous), date and time the comment was posted are also retrieved. The number of comments ranged from 65 to 359 comments per article. On average, approximately 30% of the comments are toxic. Each comment was annotated into two categories \toxic" and \non-toxic", and subsequently those annotated as \toxic" were assigned with di erent toxicity levels (non-toxic, slightly toxic, toxic, and very toxic). In addition, the following characteristics were also annotated: argumentation, constructiveness, stance, target, stereotype, sarcasm, mockery, insult, improper language, aggressiveness and intolerance. All of these characteristics (or categories) have a binary classi cation, except for the level of toxicity. Each comment was annotated by three annotators and, once all comments for each item were annotated, an inter-annotator agreement test was performed.

In addition, we used in our experiments other corpora corresponding to tasks that could be related to detection of toxicity from social media including polarity classi cation (InterTASS), emotion classi cation (EmoEvent and Universal Joy), HS identi cation (HatEval and HaterNet), and aggressiveness detection (MEXA3T). The datasets are described below: { International TASS Corpus (InterTASS) was released in 2017 [ 17 ] with Spanish tweets and updated in 2018 with texts written in three di erent variants of Spanish from Spain, Costa Rica and Peru [ 16 ]. In 2019, InterTASS was enlarged with new texts written in two new Spanish variants: Uruguayan and Mexican [ 10 ] and nally, it was completed with Chilean-Spanish Tweets in 2020 [ 13 ]. The corpus released in 2019 is the one used in this paper. At least three annotators annotated each tweet with its level of polarity, which could be labeled as positive, negative, neutral and none. { EmoEvent [ 23 ] is a multilingual emotion dataset based on events that took place in April 2019. It focuses on tweets in the areas of entertainment, catastrophes, politics, global commemoration and global strikes. For the creation of the corpus, the authors collected Spanish and English tweets from the Twitter platform. Then, each tweet was labeled with one of seven emotions, six Ekman's basic emotions plus the \neutral or other emotions" label. Focusing on the Spanish language, a total of 8,409 were labeled by three Amazon Mechanical Turkers. { Universal Joy[ 15 ] is a new data set of over 530k anonymized public Facebook posts across 18 languages. It was collected in October 2014 by searching for public Facebook posts with a Facebook \feelings tag", and labeled with ve di erent emotions: anger, anticipation, fear, joy, and sadness. There is a wide variety in the amount of data per language, ranging from 284,265 posts for English, the most frequent language, to 869 posts for Bengali. We used the 31,326 Spanish posts. { HatEval was provided by organizers in SemEval 2019 Task 5 [ 5 ]. The task consisted in detecting hateful content in Twitter posts, against two targets: women and immigrants. For the creation of the corpus, the data was collected using a di erent time frame. The majority of tweets against women were derived from an earlier collection made in the context of two earlier challenges on misogynistic speech identi cation, whose collection phase began on July 2017 and ended on November 2017 [ 12, 11 ]. The remaining tweets were collected from July to September 2018. The dataset contains tweets composed of an identi er, the text of the tweet and the mark of HS, which is 0 if the text is not hateful and 1 if the text is hateful speech against women or immigrants. { HaterNet [ 22 ] was built for the intelligent system of the same name, used by the National O ce against Hate Crimes of the Spanish Secretary of State for Security. For the creation of this corpus, over 2 million tweets originated in Spain on di erent random dates between February 2017 and December 2017 were collected. Subsequently, the tweets were ltered using six HS dictionaries and one dictionary containing generic insults. After this, only 6000 tweets were selected due to time restrictions, to be manually labeled by four experts with di erent backgrounds and in case of a tie a fth person, cast the deciding vote. Finally, out of the 6000 tweets, 1,567 were labeled as hateful and 4,433 as non-hateful. { MEX-A3T [ 3 ]. It was provided by the organizers in IberEval 2018: Authorship and aggressiveness analysis in Mexican Spanish tweets [ 1 ]. They built a corpus of tweets to detect aggressiveness from Mexican accounts collected from August to November of 2017. In order to extract the tweets, they selected a set of terms that served as seeds. Then, they used both words non-colloquial in the Dictionary of Mexicanisms and classi ed as vulgar. The hashtags were related to sexism, homophobia, politics and discrimination. They used Mexico City as the center and extracted all tweets that were within a radius of 500 km. Finally, two people labeled the collected tweets. The dataset contains tweets composed of an identi er, the text of the tweet, and the mark of aggressiveness, being 0 if the tweet is not-aggressive and 1 if the tweet is aggressive. 3

System overview

In this section, we describe the systems developed for the DEtection of TOxicity shared task in Spanish comments at IberLEF 2021.

We propose a Multi-Task Learning (MTL) system using the well-known Transformer-based model BERT which has been proven to be very successful in many natural language processing tasks.

In the MTL scenario, the goal is to learn multiple tasks simultaneously instead of learning them separately in order to improve performance on each task [ 7 ]. These tasks are usually related, although they may have di erent data or features. By sharing representations across related tasks, we can allow our model to better generalize to our original task. In this study, we used tasks related to the toxicity comment detection task. These tasks include hate speech detection, offensive language identi cation, polarity classi cation, and emotion classi cation, sharing the same type of source: social media platforms (Twitter and Facebook). Moreover, we consider each of the features provided in the DETOXIS dataset (constructiveness, argumentation, mockery, sarcasm, positive stance, negative stance, target person, target group, stereotype, insult, improper language, aggressiveness, intolerance) as speci c tasks to train our system. From now, we will refer to all these speci c tasks (the ones provided in the DETOXIS dataset) as tasks related to toxicity comments features.

To develop the MTL system, we follow the most widely used technique in neural networks introduced by [ 7 ], the hard parameter sharing approach. It consists of a single encoder that is shared and updated between all tasks, while keeping a few task-speci c layers to specialize in each task. [ 25 ].

The general architecture of the MTL-BERT model is shown in Figure 1. The shared layers are based on BERT [ 9 ]. Following Devlin et al., 2018, in the rst step, all the inputs are converted to WordPieces [ 27 ], two additional tokens are added at the start ([CLS]) and end ([SEP]) of the input sequence, respectively. In the shared layers, the BERT model rst converts the input sequence to a sequence of embedding vectors. This semantic representation is shared across all tasks. Then, on top of the shared BERT layers, the task-speci c output heads are created for each task, and task heads are attached to a common sentence encoder. Finally, the layers are ne-tuned according to the given set of downstream tasks.

Experimental setup

Dataset preprocessing

We perform a social media speci c data cleaning in the corpora related to Twitter and Facebook (InterTASS, EmoEvent, Universal Joy, HatEval, HaterNet, MEXA3T) before including the texts in the models. The following practices to prepare the text for deep learning experiments have been carried out using the ekphrasis module [ 6 ]: { URLs, emails, users' mentions, percentages, monetary amounts, time and date expressions, and phone numbers are normalized. { Hashtags are unpacked and split to their constituent words. { Elongated words and repeated characters in words are annotated and reduced. { Emojis are converted to its alias.

As the DETOXIS task dataset provided by the organizers includes responses to di erent articles extracted from Spanish online newspapers, we performed a di erent data cleaning that includes the following steps: { Remove URLs, hashtags and users' mentions. { Reduce words with more than 4 repeated characters to 3 repetitions. { Remove multiple spaces.

{ Remove texts with only numbers. 4.2

System settings

All the models were implemented using PyTorch, a high-performance deep learning library [ 21 ] based on the Torch library. The experiments were run on a single Tesla-V100 32 GB GPU with 192 GB of RAM.

During the evaluation phase, we train the model on the training set provided by the organizers, then we evaluate it on the test set.

Regarding our participation, we submitted ve runs using the proposed MTLbased system. The details of the modules and the di erences of the ve settings are described below.

{ Run 1. In order to establish a baseline in our study and compare the results with the MTL scenario, the rst run correspond to our baseline, a singletask learning approach which involves only the DETOXIS dataset. For this setting, we use the well-known Transformer BERT. { Run 2. In this setting, our goal is to train the MTL system on the tasks which are related to the identi cation of toxicity comments. Speci cally, HS identi cation, o ensive language detection and the toxicity comments features (constructiveness, argumentation, mockery...) explained in Section 3. Our assumption is that all these tasks are related to the inappropriate behavior on the web, therefore the knowledge share during training among these tasks may bene t to the task of toxic comments identi cation even if the texts correspond to di erent language registers from social media and newspapers. { Run 3. This con guration includes run 2 but with the addition of a new task: polarity classi cation. Our goal is to leverage on the sentiment expressed in the posts to aid in the classi cation of toxic comments. Our assumption is that toxicity is associated with a negative polarity, then the knowledge share can help to detect easily toxic comments. For the polarity classi cation task, we use the InterTASS dataset. { Run 4. This con guration includes run 2 but with the addition of a new task: emotion classi cation. In this setting, our goal is to leverage in the identi cation of emotion categories to aid in the classi cation of toxic comments. Our assumption is that negative emotions such as anger, fear, sadness and disgust could be related to toxicity while positive emotions are not. For the emotion analysis task, we use the EmoEvent and Universal Joy datasets. { Run 5. In this setup, we have included the polarity and emotion classi cation tasks in run 2. Therefore, in this setting the MTL system is trained on the di erent tasks explainded above. We expect that the combination of all the tasks helps to identify toxic comments.

In all the runs, the DETOXIS training set has also been used to train the MTL system, and then we have evaluated the shared task using the DETOXIS test set.

Since the DETOXIS dataset is composed of Spanish texts, while training the MTL system we use the BETO model [ 8 ] trained on Spanish texts. We employ the following hyperparameters in the ve runs: learning rate as 2e-05, batch size as 16, dropout probability as 0.01, the optimization algorithm Adamw, and maximum epoch as 3. 5

Results

In this section we present the results obtained by the di erent runs we have explored in both subtasks of the competition. In order to evaluate them we use the o cial competition metrics for subtask 1 (F-measure) and subtask 2 (Closeness Evaluation Metric (CEM) [ 2 ]). In addition, for the level detection subtask, the organizers has provided evaluation results with Rank Biased Precision (RBP) [ 18 ], Pearson coe cient, and Accuracy (Acc).

We evaluated our ve runs on subtasks 1 and 2 of the DETOXIS shared task. The results obtained are shown in Table 1 and 2, respectively. As can be seen, in subtask 1, the di erent settings of the MTL system have outperformed our baseline BETO (Run 1). It should be noted that the best setting in both subtasks is Run 5 in which all tasks related to toxicity detection in comments are combined. Speci cally, in subtask 1, run 5 outperforms with a substantial margin (3,77%) our baseline BETO. For subtask 1 it can also be observed that Run 3 achieves remarkable results, and Run 4 surpassed the baseline BETO, therefore we can con rm our hypothesis that sentiment analysis helps the task of detecting toxicity in comments. Regarding subtask 2, runs 4 and 5 surpasses the baseline BETO in terms of CEM score, which means that emotion analysis along the combination of HS identi cation, o ensive language detection and the tasks related to toxicity comments features (argumentation, constructiveness, sarcasm, mockery...) could bene t the detection of toxicity comments.

It should be remarked that although the datasets of some of the tasks explored (sentiment analysis, hate speech detection and o ensive language identication) include posts from social media, the MTL seems to be able to transfer the knowledge to a di erent language register employ in the comments posted in response to di erent articles extracted from online newspapers (DETOXIS dataset).

Finally, our results in the competition for both subtasks among the participants (Table 3 and Table 4) show the success of our proposed model achieving the rst place in the ranking in both subtasks. The representations computed by the encoder embed the a ective knowledge and the knowledge related to toxicity detection tasks (o ensive language, constructiveness, sarcasm, among others) allows the MTL model to identify toxic comments more accurately. This paper presents the participation of the SINAI research group at the DEtection of TOxicity in comments in Spanish shared task at IberLEF 2021. Our proposal explores how transferred knowledge from tasks related to the identication of toxicity language (polarity classi cation, emotion classi cation, hate speech detection, o ensive language detection, constructiveness, argumentation, sarcasm, mockery, etc.) may help in a text classi cation task like DETOXIS. Experiments conducted show the e cacy of our proposed approach in achieving convincing performance in both subtasks. Further exploration on how and which of the features we use in our MTL approach (constructiveness, argumentation, sarcasm, mockery, etc.) helps to the identi cation of toxic comments are left as future work, and we welcome the community to contribute.

Acknowledgement

This work has been partially supported by a grant from European Regional Development Fund (FEDER), LIVING-LANG project [RTI2018-094653-B-C21], and Ministry of Science, Innovation and Universities (scholarship [FPI-PRE2019089310]) from the Spanish Government.

References

1. Alvarez-Carmona , M.A. , Guzman-Falcon , E. , Montes- y-Gomez, M. , Jair-Escalante , H. , Villasen~or- Pineda , L. , Reyes-Meza , V. , Rico-Sulayes , A. : Overview of MEXA3T at ibereval 2018: Authorship and aggressiveness analysis in mexican spanish tweets . In: Proceedings of the Third Workshop on Evaluation of Human Language Technologies for Iberian Languages (IberEval 2018 ) co-located with 34th Conference of the Spanish Society for Natural Language Processing (SEPLN 2018 ), Sevilla, Spain, September 18th , 2018 . CEUR Workshop Proceedings , vol. 2150 , pp. 74 { 96 . CEUR-WS.org ( 2018 )

2. Amigo , E. , Gonzalo , J. , Mizzaro , S. , Carrillo-de Albornoz , J.: An e ectiveness metric for ordinal classi cation: Formal properties and experimental results . arXiv preprint arXiv: 2006 . 01245 ( 2020 )

3. Aragon , M.E. , Jarqu n-Vasquez, H.J. , Montes- y-Gomez, M. , Escalante , H.J. , Villasen~or- Pineda , L. , Gomez-Adorno , H. , Posadas-Duran , J.P. , Bel-Enguix , G. : Overview of MEX-A3T at IberLEF 2020: Fake News and Aggressiveness Analysis in Mexican Spanish . In: Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2020 ) co-located with 36th Conference of the Spanish Society for Natural Language Processing (SEPLN 2020 ), Malaga, Spain, September 23th , 2020 . CEUR Workshop Proceedings , vol. 2664 , pp. 222 { 235 . CEUR-WS.org ( 2020 )

4. Plaza-del Arco , F.M. , Molina-Gonzalez , M.D. , Uren~a- Lopez , L.A. , Mart nValdivia , M.T.: Comparing pre-trained language models for spanish hate speech detection . Expert Systems with Applications 166 , 114120 ( 2021 )

5. Basile , V. , Bosco , C. , Fersini , E. , Nozza , D. , Patti , V. , Rangel Pardo , F.M. , Rosso , P. , Sanguinetti , M.: SemEval-2019 task 5: Multilingual detection of hate speech against immigrants and women in Twitter . In: Proceedings of the 13th International Workshop on Semantic Evaluation . pp. 54 { 63 . Association for Computational Linguistics, Minneapolis, Minnesota, USA (Jun 2019 ). https://doi.org/10.18653/v1/ S19 -2007

6. Baziotis , C. , Pelekis , N. , Doulkeridis , C. : Datastories at semeval -2017 task 4: Deep lstm with attention for message-level and topic-based sentiment analysis . In: Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval2017) . pp. 747 { 754 . Association for Computational Linguistics, Vancouver, Canada ( August 2017 )

7. Caruana , R.: Multitask learning . Machine learning 28(1) , 41 { 75 ( 1997 )

8. Can~ete, J., Chaperon , G. , Fuentes , R. , Ho , J.H. , Kang , H. , Perez , J.: Spanish pretrained bert model and evaluation data . In: PML4DC at ICLR 2020 ( 2020 )

9. Devlin , J. , Chang , M.W. , Lee , K. , Toutanova , K. : Bert: Pre-training of deep bidirectional transformers for language understanding ( 2018 )

10. D az- Galiano , M.C. , Garc a-Vega, M. , Casasola , E. , Chiruzzo , L. , Garc aCumbreras , M.A. , Mart nez-Camara, E. , Moctezuma , D. , Montejo-Raez , A. , Sobrevilla-Cabezudo , M.A. , Sadit-Tellez , E. , et al.: Overview of TASS 2019 : One More Further for the Global Spanish Sentiment Analysis Corpus . In: IberLEF@ SEPLN. pp. 550 { 560 ( 2019 )

11. Fersini , E. , Nozza , D. , Rosso , P. : Overview of the evalita 2018 task on automatic misogyny identi cation (ami) . EVALITA Evaluation of NLP and Speech Tools for Italian 12 , 59 ( 2018 )

12. Fersini , E. , Rosso , P. , Anzovino , M. : Overview of the task on automatic misogyny identi cation at ibereval 2018 . In: Rosso, P. , Gonzalo , J. , Mart nez , R., Montalvo , S., de Albornoz, J.C. (eds.) Proceedings of the Third Workshop on Evaluation of Human Language Technologies for Iberian Languages (IberEval 2018 ). CEUR Workshop Proceedings , vol. 2150 , pp. 214 { 228 . CEUR-WS.org ( 2018 )

13. Garc a-Vega, M., D az- Galiano , M.C. , Garc a-Cumbreras, M.A. , Plaza-del- Arco , F.M. , Montejo-Raez , A. , Jimenez-Zafra , S.M. , Mart nez-Camara, E. , et al.: Overview of TASS 2020 : Introducing emotion detection . In: Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2020 ) co-located with 36th Conference of the Spanish Society for Natural Language Processing (SEPLN 2020 ), Malaga, Spain, September 23th , 2020 . CEUR Workshop Proceedings , vol. 2664 , pp. 163 { 170 . CEUR-WS.org ( 2020 )

14. Kumar , R. , Ojha , A.K. , Malmasi , S. , Zampieri , M. : Evaluating aggression identi cation in social media . In: Proceedings of the Second Workshop on Trolling, Aggression and Cyberbullying . pp. 1 { 5 ( 2020 )

15. Lamprinidis , S. , Bianchi , F. , Hardt , D. , Hovy , D. : Universal joy a data set and results for classifying emotions across languages . In: Proceedings of the Eleventh Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis . pp. 62 { 75 ( 2021 )

16. Mart nez-Camara, E. , Almeida-Cruz , Y. , D az- Galiano , M.C. , Estevez-Velarde , S. , Garc a-Cumbreras, M.A. , Garc a-Vega, M. , Gutierrez , Y. , Montejo-Raez , A. , Montoyo , A. , Mun~oz, R., Piad-Mor s , A. , Villena-Roman , J. : Overview of TASS 2018 : Opinions, health and emotions . In: Proceedings of TASS 2018: Workshop on Semantic Analysis at SEPLN, TASS@SEPLN 2018. CEUR Workshop Proceedings , vol. 2172 , pp. 13 { 27 . CEUR-WS.org ( 2018 )

17. Mart nez-Camara, E., D az- Galiano , M.C. , Garc a-Cumbreras, M.A. , Garc a-Vega, M. , Villena-Roman , J. : Overview of TASS 2017 . Proceedings of TASS pp. 13 { 21 ( 2017 )

18. Mo at, A., Zobel , J.: Rank-biased precision for measurement of retrieval e ectiveness . ACM Transactions on Information Systems (TOIS) 27(1) , 1 { 27 ( 2008 )

19. Montes , M. , Rosso , P. , Gonzalo , J. , Aragon , E. , Agerri , R. , Alvarez Carmona , M. , Alvarez Mellado , E. , Carrillo-de Albornoz , J., Chiruzzo , L. , Freitas , L. , Gomez Adorno , H. , Gutierrez , Y. , Jimenez-Zafra , S.M. , Lima , S. , Plaza-del- Arco , F.M. , Taule , M. (eds.): Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2021 ) ( 2021 )

20. Nobata , C. , Tetreault , J., Thomas , A. , Mehdad , Y. , Chang , Y. : Abusive language detection in online user content . In: Proceedings of the 25th international conference on world wide web . pp. 145 { 153 ( 2016 )

21. Paszke , A. , Gross , S. , Massa , F. , Lerer , A. , Bradbury , J. , Chanan , G. , Killeen , T. , Lin , Z. , Gimelshein , N. , Antiga , L. , et al.: Pytorch: An imperative style, highperformance deep learning library . In: Advances in neural information processing systems . pp. 8026 { 8037 ( 2019 )

22. Pereira-Kohatsu , J.C. , Quijano-Sanchez , L. , Liberatore , F. , Camacho-Collados , M. : Detecting and monitoring hate speech in twitter . Sensors 19 ( 21 ), 4654 ( 2019 )

23. Plaza-del- Arco , F. , Strapparava , C. , Uren ~ a- Lopez , L.A. , Martin-Valdivia , M.T.: EmoEvent: A Multilingual Emotion Corpus based on di erent Events . In: Proceedings of the 12th Language Resources and Evaluation Conference . pp. 1492 { 1498 . European Language Resources Association , Marseille, France (May 2020 ), https://www.aclweb.org/anthology/2020.lrec- 1 . 186

24. Poletto , F. , Basile , V. , Sanguinetti , M. , Bosco , C. , Patti , V. : Resources and benchmark corpora for hate speech detection: a systematic review . Language Resources and Evaluation pp. 1 { 47 ( 2020 )

25. Ruder , S. : Neural transfer learning for natural language processing . Ph.D. thesis , NUI Galway ( 2019 )

26. Taule , M. , Ariza , A. , Nofre , M. , Amigo , E. , Rosso , P. : Overview of the detoxis task at iberlef-2021: Detection of toxicity in comments in spanish . Procesamiento del Lenguaje Natural 67 ( 2021 )

27. Wu , Y. , Schuster , M. , Chen , Z. , Le , Q.V. , Norouzi , M. , Macherey , W. , Krikun , M. , Cao , Y. , Gao , Q. , Macherey , K. , et al.: Google's neural machine translation system: Bridging the gap between human and machine translation . CoRR abs/1609 .08144 ( 2016 )

28. Zampieri , M. , Nakov , P. , Rosenthal , S. , Atanasova , P. , Karadzhov , G. , Mubarak , H. , Derczynski , L. , Pitenis , Z. , Coltekin, C.: Semeval-2020 task 12: Multilingual o ensive language identi cation in social media (o enseval 2020) . arXiv preprint arXiv: 2006 . 07235 ( 2020 )