Evaluation of Intermediate Pre-training for the Detection of O ensive Language

Abstract

This paper presents an evaluation of intermediate pretraining for the task of o ensive language identi cation. We leverage recent advances in multilingual contextual representation and ne-tuning of pre-trained language models. We compare the performance of a pretrained language model adapted for the social media domain and another that was further trained on multilingual sentiment analysis data. We found that the intermediate pre-training steps prior to ne-tuning on the target task yield performance gains. The best submissions by our team, NLP-CIC, achieved rst and second place on the non-contextual Spanish (Subtask 1) and Mexican Spanish (Subtask 3) subtasks of the MeO endEs-IberLEF 2021 shared task respectively.

Keywords

• XLM-RoBERTa
• Social Media
• Spanish
• Mexican Spanish
• O ensive Language Identi cation
• Sentiment Analysis

Introduction

The purpose of social media is for information exchange. This involves interactions among users on the various social media platforms. During these interactions, users often show unhealthy and anti-social behaviour such as insults and personal attacks. This kind of behaviour hampers meaningful conversations at the least and can cause harm to individuals, groups, and the society at large. Natural language processing research can help in identifying o ensive language to help reduce incidences of unacceptable behaviour. Research into this problem has gained attention especially in English language. This is attributable to availability of labeled data and pre-trained word embeddings and language models. Recently, there has been a number of shared tasks with a focus on languages other than English. One example is the IberLEF shared task series on o ensive language identi cation in Mexican Spanish. For the 2021 edition [ 8 ] the tasks include MeO endEs [ 11 ], a track on o ensive language identi cation on several social media platforms. The challenge includes datasets for Spanish and Mexican Spanish.

Multilingual language models (LM) are becoming a popular area of focus with the transformer architecture [ 13 ] which makes it possible to combine text written in di erent languages to learn a single multilingual representation. There are several successful instances of this approach in multilingual BERT [ 6 ], XLMRoBERTa [ 5 ], and recently multilingual T5 [ 15 ]. However, these pre-trained multilingual LMs mainly cover domains with text written in consistent and formal style in contrast to social media text which are noisy, irregular and informal. To adapt LMs to a speci c domain, the authors of [ 4, 7 ] showed that using the transformer architecture and its pre-trained weights, a domain-speci c model can be derived by continuing pre-training on text speci c to the domain of interest. For the social media domain, [ 9 ] and [ 3 ] are examples of this adaptation in the monolingual English setting. For the multilingual case, the authors of [ 2 ] introduced an adaptation of XLM-RoBERTa to multilingual twitter text. XLM-RoBERTa was further trained with the masked language modeling objective on twitter text (about 12GB) in over 30 languages. Furthermore, this LM was trained on a uni ed collection of sentiment analysis data in eight languages with the goal of demonstrating the e ectiveness of the multilingual LM trained on twitter text.

In this paper, our focus is to evaluate the e ectiveness of the pre-trained LMs on the identi cation of o ensive language in tweets written in Spanish and Mexican Spanish. We examine the e ect that intermediate pre-training on sentiment analysis, a la [ 10, 12 ], has on o ensive language identi cation. 2

Methodology

Task. We address the non-contextual classi cation of o ensive language in tweets written in Spanish (Subtask 1) and Mexican Spanish (Subtask 3). For Subtask 1, the task is to classify comments written in Spanish using only the textual content into one of four categories: O ensive where the target is a person (OFP); O ensive where the target is a group of people (OFG); non-o ensive, but with inadequate language (NOM); non-o ensive (NO). This subtask also assess the agreement between the con dence of model predictions and the con dence of human annotators. Subtask 3 is a binary classi cation of tweets written in Mexican Spanish. It requires predicting whether a comment is o ensive or not. Data. The MeO endEs 2021 shared task [ 11 ] provides two corpora, O endEs and O endMEX, which are collections of messages on social media platforms in Spanish and Mexican Spanish annotated with labels indicating o ensiveness. The generic Spanish data consist of labeled comments focusing on popular young Spanish in uencers collected from di erent social media platforms (YouTube, Instagram, and Twitter). The Mexican Spanish dataset was collected from Twitter and manually labeled for o ensiveness. In addition, metadata for each comment is provided for the classi cation in the contextual tracks of the competition. We only participate in the non-contextual tracks in both languages. Table 1 provides details of the Mexican Spanish dataset. Also, the details of the Spanish dataset is in Table 2.

We perform minimal pre-processing of the data in our experiments as it was reported in [ 1 ] that extensive pre-processing tends to hurt performance of pretrained LMs. Hence, we normalize the text by converting user mentions and web links to @USER and URL. We also replace multiple consecutive whitespaces with a single one and punctuation marks are surrounded by a single whitespace character on both sides. Then, the sequence of text is tokenized with the subword tokenizer provided with the XLM-RoBERTa model, which is a Sentence Piece model (using a unigram language model) with a vocabulary size of 250K [ 5 ]. We set the maximum sequence length to 128 subword tokens.

Fine-tuning. We use the huggingface transformers library [ 14 ] for the experiments. We add a linear prediction layer on top of the pooled output of the last transformer layer and optimize this layer jointly with the pre-trained layers. We optimize the model using Adam (without bias correction) with a batch size of 128 on a single Nvidia V100 GPU (32GB) and a maximum learning rate within the range of 1e-5 to 5e-5. We use a warmup ratio of 0.1 and set the maximum number of epochs to 10 with earlystopping on the validation performance metric (micro F1) using a patience of 2 evaluation runs. We evaluate the performance of the model every 20 steps on the validation set. Furthermore, we employ three regularization techniques: weight decay with a factor of 0.01, dropout applied to the pooled output of the last transformer layer with a probability of 0.2, and label smoothing with a factor of 0.1. Our submissions vary in their use of the regularization approaches. Details of the settings for each submission are in Table 3. The con gurations are based on XLM-twitter 1 and XLM-twittersentiment 2 pre-trained models introduced in [ 2 ]. For all experiments, we set the random seed to 42. On average, the ne-tuning procedure takes about 900 seconds (wall time) for the Spanish task and approximately 600 seconds (wall time) for the Mexican Spanish task. 1 https://huggingface.co/cardi nlp/twitter-xlm-roberta-base 2 https://huggingface.co/cardi nlp/twitter-xlm-roberta-base-sentiment The scores of our submissions on the development and test sets for Subtask 1 (generic Spanish) are in Table 4. Three submissions are allowed for this subtask. Submission-II which is based on the XLM-RoBERTa model trained on both multilingual twitter text and sentiment analysis dataset achieved our best submission out of the three. The model also has the least mean squared error, an indication of greater agreement with the con dence of human annotators. The overall ranking showed that this system is the best on the competition for the non-contextual classi cation in Spanish. Figure 1 shows the confusion matrix of the best model (Submission II) on the validation dataset for Subtask 1. It can be observed that most of the mistakes on the validation data occurs where the model predicts non-o ensive (NO) when the comments are actually o ensive to a person (OFP). Also, the model performs poorly on the o ensive to a group category (OFG). It makes the correct prediction on 1 out of 4 examples in the validation set.

Table 5 presents the results received by the NLP-CIC team on the leaderboard on the unseen test set for Subtask 3 (Mexican Spanish). The maximum number of submissions for this task is ve. It can be observed that the XLMRoberta model that has been further trained on twitter data and a collection of sentiment analysis datasets in eight languages (Submission-I) that we netune on Subtask 3 dataset has the highest score out of our four submissions. Also, the results show that label smoothing was bene cial for this task. On the overall ranking for the competition, this system is in second place. In Figure 2, the confusion matrix provides an overview into the best model performance (Submission I) across the two classes. The model predicts the non-o ensive label (NO) on 8 examples when the true label is o ensive (OFF) compared to the converse where the model predicts the o ensive label on 1 example when the true label is non-o ensive. It shows that the model is relatively better at identifying the non-o ensive category on the validation dataset. This can be linked to the number of examples for the non-o ensive category which is about three times more than the o ensive category in the training set.

We observed that overall, the scores on the Spanish dataset is far higher than the Mexican Spanish dataset even though it's a binary classi cation task. We think that the amount of data available for the Spanish task is a factor for this di erence in performance. The consistent performance of the model that includes sentiment analysis as part of the pre-training for both tasks con rms our hypothesis that sentiment analysis can be bene cial for detecting o ensiveness. 4

Conclusion

We address the task of o ensive language identi cation in Spanish and Mexican Spanish using a pre-trained language model adapted for the twitter domain. We found that a further training on multilingual sentiment analysis is bene cial to the task. In addition, label smoothing proved useful on the Mexican Spanish dataset. The best systems submitted by our team, NLP-CIC, achieved rst place on the non-contextual Spanish task and second place on the non-contextual Mexican Spanish task.

In the future, we will like to examine whether a model trained on Spanish data can be seamlessly transferred to Mexican Spanish for this task and vice versa. Our models only use textual content, it is very likely that the addition of metadata can improve their performance.

Acknowledgements

Thanks to the competition organizers for their support. The authors thank CONACYT for the computer resources provided through the INAOE Supercomputing Laboratory's Deep Learning Platform for Language Technologies.

References

1. Aroyehun , S.T. , Gelbukh , A. : NLP-CIC at HASOC 2020: Multilingual O ensive Language Detection using All-in-one Model . In: FIRE (Working Notes) . pp. 331 { 335 ( 2020 ), http://ceur-ws. org/ Vol- 2826 / T2 -31.pdf

2. Barbieri , F. , Anke , L.E. , Camacho-Collados , J. : XLM-T: A Multilingual Language Model Toolkit for Twitter . arXiv preprint arXiv:2104.12250 ( 2021 )

3. Barbieri , F. , Camacho-Collados , J. , Espinosa Anke , L. , Neves , L.: TweetEval: Uni ed benchmark and comparative evaluation for tweet classi - cation . In: Findings of the Association for Computational Linguistics: EMNLP 2020 . pp. 1644 { 1650 . Association for Computational Linguistics, Online (Nov 2020 ). https://doi.org/10.18653/v1/ 2020 . ndings-emnlp. 148 , https://www.aclweb.org/anthology/2020. ndings-emnlp. 148

4. Beltagy , I. , Lo , K. , Cohan , A. : SciBERT: A pretrained language model for scienti c text . In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) . pp. 3615 { 3620 . Association for Computational Linguistics, Hong Kong, China (Nov 2019 ). https://doi.org/10.18653/v1/ D19 -1371, https://www.aclweb.org/anthology/D19-1371

5. Conneau , A. , Khandelwal , K. , Goyal , N. , Chaudhary , V. , Wenzek , G. , Guzman , F. , Grave , E. , Ott , M. , Zettlemoyer , L. , Stoyanov , V. : Unsupervised cross-lingual representation learning at scale . In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics . pp. 8440 { 8451 . Association for Computational Linguistics, Online (Jul 2020 ). https://doi.org/10.18653/v1/ 2020 .aclmain. 747 , https://www.aclweb.org/anthology/2020.acl-main. 747

6. Devlin , J. , Chang , M.W. , Lee , K. , Toutanova , K. : BERT: Pre-training of deep bidirectional transformers for language understanding . In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , Volume 1 (Long and Short Papers). pp. 4171 { 4186 . Association for Computational Linguistics, Minneapolis, Minnesota (Jun 2019 ). https://doi.org/10.18653/v1/ N19 -1423, https://www.aclweb.org/anthology/N19-1423

7. Gururangan , S. , Marasovic , A. , Swayamdipta , S. , Lo , K. , Beltagy , I. , Downey , D. , Smith , N.A. : Don't stop pretraining: Adapt language models to domains and tasks . In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics . pp. 8342 { 8360 . Association for Computational Linguistics, Online (Jul 2020 ). https://doi.org/10.18653/v1/ 2020 .acl-main. 740 , https://www.aclweb.org/anthology/2020.acl-main. 740

8. Montes , M. , Rosso , P. , Gonzalo , J. , Aragon , E. , Agerri , R. , Alvarez-Carmona , M.A. , Alvarez Mellado , E. , Carrillo-de Albornoz , J., Chiruzzo , L. , Freitas , L. , Gomez Adorno , H. , Gutierrez , Y. , Jimenez-Zafra , S.M. , Lima , S. , Plaza-de Arco , F.M. , Taule , M. (eds.): Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2021 ) ( 2021 )

9. Nguyen , D.Q. , Vu , T. , Tuan Nguyen , A. : BERTweet: A pre-trained language model for English tweets . In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations . pp. 9 { 14 . Association for Computational Linguistics, Online (Oct 2020 ). https://doi.org/10.18653/v1/ 2020 .emnlp-demos.2, https://www.aclweb.org/anthology/2020.emnlp-demos. 2

10. Phang , J. , Calixto , I. , Htut , P.M. , Pruksachatkun , Y. , Liu , H. , Vania , C. , Kann , K. , Bowman , S.R. : English intermediate-task training improves zeroshot cross-lingual transfer too . In: Proceedings of the 1st Conference of the Asia-Paci c Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing . pp. 557 { 575 . Association for Computational Linguistics, Suzhou, China (Dec 2020 ), https://www.aclweb.org/anthology/2020.aacl-main. 56

11. Plaza-del- Arco , F.M. , Casavantes , M. , Jair Escalante , H. , Mart n-Valdivia, M.T. , Montejo-Raez , A. , Montes- y-Gomez, M. , Jarqu n-Vasquez, H. , Villasen~or- Pineda , L. : Overview of the MeO endEs task on o ensive text detection at IberLEF 2021 . Procesamiento del Lenguaje Natural 67 ( 0 ) ( 2021 )

12. Pruksachatkun , Y. , Phang , J. , Liu, H. , Htut , P.M. , Zhang , X. , Pang , R.Y., Vania , C. , Kann , K. , Bowman , S.R. : Intermediate-task transfer learning with pretrained language models: When and why does it work? In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics . pp. 5231 { 5247 . Association for Computational Linguistics, Online (Jul 2020 ). https://doi.org/10.18653/v1/ 2020 .acl-main. 467 , https://www.aclweb.org/anthology/2020.acl-main. 467

13. Vaswani , A. , Shazeer , N. , Parmar , N. , Uszkoreit , J. , Jones , L. , Gomez , A.N. , Kaiser , L. , Polosukhin , I. : Attention is all you need . In: Guyon, I. , Luxburg , U.V. , Bengio , S. , Wallach , H. , Fergus , R. , Vishwanathan , S. , Garnett , R . (eds.) Advances in Neural Information Processing Systems 30 , pp. 5998 { 6008 . Curran Associates, Inc. ( 2017 ), http://papers.nips.cc/paper/7181-attention -is-all-you-need .pdf

14. Wolf , T. , Debut , L. , Sanh , V. , Chaumond , J. , Delangue , C. , Moi , A. , Cistac , P. , Rault , T. , Louf , R. , Funtowicz , M. , Davison , J. , Shleifer , S. , von Platen, P. , Ma, C. , Jernite , Y. , Plu , J. , Xu , C. , Le Scao , T. , Gugger , S. , Drame , M. , Lhoest , Q. , Rush , A. : Transformers: State-of-the-Art Natural Language Processing . In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations . pp. 38 { 45 . Association for Computational Linguistics, Online (Oct 2020 ). https://doi.org/10.18653/v1/ 2020 .emnlp-demos.6, https://www.aclweb.org/anthology/2020.emnlp-demos. 6

15. Xue , L. , Constant , N. , Roberts , A. , Kale , M. , Al-Rfou , R. , Siddhant , A. , Barua , A. , Ra el , C.: mt5: A massively multilingual pre-trained text-to-text transformer . arXiv preprint arXiv: 2010 . 11934 ( 2020 )