Transformers Pipeline for O ensiveness Detection in Mexican Spanish Social Media

Abstract

In this paper, we describe the methodology proposed for participating in the MeO endEs@IberLEF 2021 competition for the Subtask 3: Non-contextual binary classi cation for Mexican Spanish, which consists in the classi cation of tweets as o ensive or non-o ensive. We proposed a Transformers-based pipeline, consisting on a series of preprocessing steps and the use of an extended corpus, followed by an ensemble of BERT models. The proposed strategy obtained the best results on this task by ranking rst place.

Keywords

• O ensiveness Detection
• Mexican Spanish
• Transformers

Introduction

In the last years, there have been many initiatives in the NLP and Machine Learning community, to guide research e orts towards solutions in the automatic detection of threats and risks to the users of social networks. Those threats include aggressiveness, hate speech, harassment, racism, misogyny, among many others. For spanish language, those e orts have been promoted by academic competitions in speci c tasks, such as the events organized by TASS [ 6,10 ], PAN [ 5 ] and particularly, MEX-A3T@IberLEF [ 1,2,7 ] and MeO endEs@IberLEF [ 12, 13 ], which includes a track for aggresiveness and o ensiveness identi cation task respectively for tweets in Mexican spanish.

Detection of o ensive comments or posts in social media is not easy, because it is not depending on the presence or absence of speci c words. As an example, consider the next tweets taken from MEX-A3T 2020 training corpus: "No se si guardar dinero para salir contigo o gastarlo en pendejadas a la verga" "Ya no saben que verga decir, consigan una vida y sufran o algo"

In the rst case, the tweet is non-o ensive, even when it contains vulgar and rude language. The second tweet is o ensive, although the language is less vulgar than the rst one. Based on that, we argue that it is necessary to take into account the context in which words are used.

There are many proposals to tackle o ensive and aggressive content detection in social media for spanish language, with document representations based on n-grams (word and character level), and word embeddings, with classi ers based on standard machine learning and deep learning approaches [ 1, 3, 7 ]. However, in the last year, there is a visible trend in the use of contextualized representations of words, such as Bi-LSTM, Bi-GRU, and Transformers-based models, such as BETO [ 8 ] with and without ne tunning [ 2, 4, 16, 17 ]. State of the art results for Mexican spanish on this task has been reached with a bagging-like scheme, by combining di erent BERT models trained on di erent augmented datasets [ 11 ].

Similar to [ 11 ], we propose an ensemble of BERT models, but also, we use a pre-processing step in order to obtain valuable text descriptions of specialized language used in tweets, followed by an extension of the training corpus. The empirical evaluation shows that the proposed approach obtained the best results in the challenge by ranking rst place. In the following sections, our proposal is explained in detail.

This document is organized as follows: Section 2 describes the dataset and the experimental settings. Section 3 describes the proposed pipeline, and Section 3.4 the experimental results. Finally, Section 4 outlines the conclusions. 2

Dataset and model settings

O endMex corpus consists of a training set of 5060 tweets and a validation set of 76 tweets, from the total, 80% was used for training and 20% for evaluation purposes. The dataset has a length mean of 24.11 and a maximum of 60 tokens, with an unbalanced ratio of 2.66 and the o ensive class as the minority.

For this task, a pre-trained BERT model on Spanish was used [ 8 ], and for the ne-tuning step for small datasets (less than 100,000), we used the exhaustive search over the recommended hyperparameters [ 9 ] and we choose the best one on the evaluation set. As a result, we used a BERT model with a training batch size of 16 for 4 epochs, and an Adam optimizer with a learning rate of 3e-5. 3

Pipeline

In this section, the proposed pipeline is described: the corpus pre-processing step, second, the extended corpus step, and nally, the BERT ensemble step. 3.1

Step 1: Pre-processing From other classi cation tasks such as irony or sentiment classi cation has been proved that adding the tweet jargon like hashtags, emojis, and emoticons as text descriptions improves tweet classi cation tasks through deep learning models like BERT [ 14, 15 ].

Our procedure is the following: { Hashtags are split into words (see Figure 1) using the python word ninja library (https://github.com/keredson/wordninja) with a Spanish dictionary made with the Spanish fasttext vocabulary (https://fasttext.cc/docs/en/crawlvectors.html) . { Emojis are replaced with their text meaning in Spanish (see Figure 1) given by the python library emoji (https://github.com/carpedm20/emoji/). { Emoticons are replaced with a text representation in Spanish similar to the emojis meanings (see Table 1). { Words out of vocabulary are replaced with the corresponding words (see Table 1).

After pre-processing step, the maximum corpus length increases twice (see Figure 2); this is the reason why the maximum sequence length of BERT model is 128. It was demonstrated [ 16 ] that increasing the training corpus examples with other corpus labeled with a related task such as hate speech could improve model performance. In this pipeline, we choose to use hate speech and negative sentiment from the HatEval 2019 in Spanish [ 4 ] and TASS 2019 for Mexican Spanish [ 10 ] corpus, respectively.

The methodology proposed to add examples from other corpus as a way to improve model performance and reduce the unbalanced ratio is shown in Figure 3, and consists on the following three steps. In the rst step the corpus must be preprocessed by the method described in section 3.1, the second step consists on training with the O endMex 2021 corpus by the method described in section 3.3, and make inference on the HatEval and TASS corpus, and then, we select only those examples whose weights in the classi er are greater or equal to 0.95, which are added to the O endMex corpus as o ensive examples. Finally, the step three consists of training from scratch the model again. The intuitive idea of this step is to augment the training data only with those instances that could improve the classi cation score. In order to alleviate BERT instability of ne-tuning on small samples and unbalanced datasets, it was shown [ 11 ] that using single BERTs as weak models and through an ensemble of 20 BERTs and a weighted voting scheme, which means that accumulating the softmax layer outputs and selecting the class with the maximum weight makes a more robust model (see Figure 4). After following the proposed pipeline described in Section 3, we obtained the results shown in Table 2, where it can be seen that each step on the proposed pipeline helps to improve the model performance, as we expected. The best result achieved a F1 score on the evaluation set of 71.07 with a 20 BERT ensemble, pre-processing, and nally adding more examples to the training corpus. This work presented a pipeline of three steps for o ensiveness detection on Mexican Spanish social media that e ectively achieved rst place on the MeOffendEs@IberLEF 2021 subtask 3 competition with a F1 score of 0.7026 on the test set. Our experimental results on the evaluation set shown that each step on the pipeline improves the model performance. We thought this pipeline could be implemented quickly and successfully in other related tasks such as aggressiveness detection.

Acknowledgements References

Gomez-Espinosa thanks CONACYT for the scholarship for Master degree studies with number: 1002761.

References

1. Aragon , M.E. , Carmona , M.A.A. , Montes- y-Gomez, M. , Escalante , H.J. , Pineda , L.V. , Moctezuma , D. : Overview of MEX-A3T at iberlef 2019: Authorship and aggressiveness analysis in mexican spanish tweets . In: Proceedings of the Iberian Languages Evaluation Forum co-located with 35th Conference of the Spanish Society for Natural Language Processing , IberLEF@SEPLN 2019 , Bilbao, Spain, September 24th , 2019 . CEUR Workshop Proceedings , vol. 2421 , pp. 478 { 494 . CEURWS.org ( 2019 )

2. Aragon , M.E. , Jarqu n-Vasquez, H.J. , Montes- y-Gomez, M. , Escalante , H.J. , Pineda , L.V. , Gomez-Adorno , H. , Posadas-Duran , J.P. , Bel-Enguix , G. : Overview of MEX-A3T at iberlef 2020: Fake news and aggressiveness analysis in mexican spanish . In: Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2020 ) co-located with 36th Conference of the Spanish Society for Natural Language Processing (SEPLN 2020 ), Malaga, Spain, September 23th , 2020 . CEUR Workshop Proceedings , vol. 2664 , pp. 222 { 235 . CEUR-WS.org ( 2020 )

3. Aragon , M.E. , Lopez-Monroy , A.P. : Author pro ling and aggressiveness detection in spanish tweets: MEX-A3T 2018 . In: Proceedings of the Third Workshop on Evaluation of Human Language Technologies for Iberian Languages (IberEval 2018 ) co-located with 34th Conference of the Spanish Society for Natural Language Processing (SEPLN 2018 ), Sevilla, Spain, September 18th , 2018 . CEUR Workshop Proceedings , vol. 2150 , pp. 134 { 139 . CEUR-WS.org ( 2018 )

4. Basile , V. , Bosco , C. , Fersini , E. , Nozza , D. , Patti , V. , Rangel Pardo , F.M. , Rosso , P. , Sanguinetti , M.: SemEval-2019 task 5: Multilingual detection of hate speech against immigrants and women in Twitter . In: Proceedings of the 13th International Workshop on Semantic Evaluation . pp. 54 { 63 . Association for Computational Linguistics, Minneapolis, Minnesota, USA (Jun 2019 )

5. Bevendor , J. , Chulvi , B. , Sarracen , G.L.D.L.P. , Kestemont , M. , Manjavacas , E. , Markov , I. , Mayerl , M. , Potthast , M. , Rangel , F. , Rosso , P. , Stamatatos , E. , Stein , B. , Wiegmann , M. , Wolska , M. , Zangerle , E.: Overview of pan 2021: Authorship veri cation, pro ling hate speech spreaders on twitter, and style change detection . In: Hiemstra, D. , Moens , M.F. , Mothe , J. , Perego , R. , Potthast , M. , Sebastiani , F . (eds.) Advances in Information Retrieval. pp. 567 { 573 . Springer International Publishing, Cham ( 2021 )

6. Camara , E.M. , Almeida-Cruz , Y. , D az- Galiano , M.C. , Estevez-Velarde , S. , Cumbreras , M.A.G. , Vega , M.G. , Gutierrez , Y. , Montejo-Raez , A. , Montoyo , A. , Mun~oz, R., Piad-Mor s , A. , Villena-Roman , J. : Overview of TASS 2018 : Opinions, health and emotions . In: Proceedings of TASS 2018: Workshop on Semantic Analysis at SEPLN, TASS@SEPLN 2018 , co-located with 34nd SEPLN Conference (SEPLN 2018 ), Sevilla, Spain, September 18th , 2018 . CEUR Workshop Proceedings , vol. 2172 , pp. 13 { 27 . CEUR-WS.org ( 2018 )

7. Carmona , M.A.A. , Guzman-Falcon , E. , Montes- y-Gomez, M. , Escalante , H.J. , Pineda , L.V. , Reyes-Meza , V. , Sulayes , A.R. : Overview of MEX-A3T at ibereval 2018: Authorship and aggressiveness analysis in mexican spanish tweets . In: Proceedings of the Third Workshop on Evaluation of Human Language Technologies for Iberian Languages (IberEval 2018 ) co-located with 34th Conference of the Spanish Society for Natural Language Processing (SEPLN 2018 ), Sevilla, Spain, September 18th , 2018 . CEUR Workshop Proceedings , vol. 2150 , pp. 74 { 96 . CEURWS.org ( 2018 )

8. Can~ete, J., Chaperon , G. , Fuentes , R. , Ho , J.H. , Kang , H. , Perez , J.: Spanish pretrained bert model and evaluation data . In: PML4DC at ICLR 2020 ( 2020 )

9. Devlin , J. , Chang , M.W. , Lee , K. , Toutanova , K. : BERT: Pre-training of deep bidirectional transformers for language understanding . In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , Volume 1 (Long and Short Papers). pp. 4171 { 4186 . Association for Computational Linguistics, Minneapolis, Minnesota (Jun 2019 )

10. D az- Galiano , M.C. , Vega , M.G. , Casasola , E. , Chiruzzo , L. , Cumbreras , M.A.G. , Camara , E.M. , Moctezuma , D. , Montejo-Raez , A. , Cabezudo , M.A.S. , Tellez , E.S. , Gra , M. , Miranda-Jimenez , S. : Overview of TASS 2019: One more further for the global spanish sentiment analysis corpus . In: Proceedings of the Iberian Languages Evaluation Forum co-located with 35th Conference of the Spanish Society for Natural Language Processing , IberLEF@SEPLN 2019 , Bilbao, Spain, September 24th , 2019 . CEUR Workshop Proceedings , vol. 2421 , pp. 550 { 560 . CEUR-WS.org ( 2019 )

11. Guzman-Silverio , M. , Balderas-Paredes , A. , Lopez-Monroy , A.P. : Transformers and data augmentation for aggressiveness detection in mexican spanish . In: Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2020 ) co-located with 36th Conference of the Spanish Society for Natural Language Processing (SEPLN 2020 ), Malaga, Spain, September 23th , 2020 . CEUR Workshop Proceedings , vol. 2664 , pp. 293 { 302 . CEUR-WS.org ( 2020 )

12. Montes , M. , Rosso , P. , Gonzalo , J. , Aragon , E. , Agerri , R. , Alvarez-Carmona , M.A. , Alvarez Mellado , E. , Carrillo-de Albornoz , J., Chiruzzo , L. , Freitas , L. , Gomez Adorno , H. , Gutierrez , Y. , Jimenez-Zafra , S.M. , Lima , S. , Plaza-de Arco , F.M. , Taule , M. (eds.): Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2021 ) ( 2021 )

13. Plaza-del- Arco , F.M. , Casavantes , M. , Escalante , H. , Martin-Valdivia , M.T. , Montejo-Raez , A. , Montes- y-Gomez, M. , Jarqu n-Vasquez, H. , Villasen~or- Pineda , L. : Overview of the MeO endEs task on o ensive text detection at IberLEF 2021 . Procesamiento del Lenguaje Natural 67 ( 0 ) ( 2021 )

14. Pota , M. , Ventura , M. , Catelli , R. , Esposito , M.: An e ective bert-based pipeline for twitter sentiment analysis: A case study in italian . Sensors 21 ( 1 ) ( 2021 )

15. Singh , A. , Blanco , E. , Jin , W. : Incorporating emoji descriptions improves tweet classi cation . In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , Volume 1 (Long and Short Papers). pp. 2096 { 2101 . Association for Computational Linguistics, Minneapolis, Minnesota (Jun 2019 )

16. Tanase , M. , Zaharia , G. , Cercel , D. , Dascalu , M. : Detecting aggressiveness in mexican spanish social media content by ne-tuning transformer-based models . In: Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2020 ) co-located with 36th Conference of the Spanish Society for Natural Language Processing (SEPLN 2020 ), Malaga, Spain, September 23th , 2020 . CEUR Workshop Proceedings , vol. 2664 , pp. 236 { 245 . CEUR-WS.org ( 2020 )

17. Zampieri , M. , Nakov , P. , Rosenthal , S. , Atanasova , P. , Karadzhov , G. , Mubarak , H. , Derczynski , L. , Pitenis , Z. , Coltekin, C.: SemEval-2020 task 12: Multilingual o ensive language identi cation in social media (O ensEval 2020) . In: Proceedings of the Fourteenth Workshop on Semantic Evaluation . pp. 1425 { 1447 . International Committee for Computational Linguistics, Barcelona (online) (Dec 2020 )