This paper describes the system submitted to Dravidian-Codemix-HASOC2021: Hate Speech and Ofensive Language Identification in Dravidian Languages (Tamil-English and Malayalam-English). This task aims to identify ofensive content in code-mixed comments/posts in Dravidian Languages collected from social media. Our approach utilizes pooling the last layers of pretrained transformer multilingual BERT for this task which helped us achieve rank nine on the leaderboard with a weighted average score of 0.61 for the Tamil-English dataset in subtask B. After the task deadline, we sampled the dataset uniformly and used the MuRIL pretrained model, which helped us achieve a weighted average score of 0.67, the top score in the leaderboard. Furthermore, our approach to utilizing the pretrained models helps reuse our models for the same task with a diferent dataset. Our code and models are available in GitHub 1 CEUR Workshop Proceedings
In the era of the internet, people from various age groups engage in social media, it has become
a one-stop shop for all activities from learning to entertainment, but it is also filled with
ofensive and disturbing content, which is potentially harmful to everyone [
Dravidian languages are under-resourced in natural language processing [10]. Dravidian
name was derived from Tamil, Dravidian means Tamil [11], Dravidian languages are Tamili
languages [12]. Tamil is a language spoken by Tamils in Tamil Nadu, India, Sri Lanka, and
the Tamil diaspora worldwide, with oficial recognition in India, Sri Lanka, and Singapore
[13, 14, 15]. Current Tamil script was developed from the Tamili script, the Vatteluttu alphabet,
and the Chola-Pallava script. There are 12 vowels, 18 consonants, and 1 ytam in this word
1https://github.com/seanbenhur/tanglish-offensive-language-identification
https://seanbenhur.github.io/ (S. Benhur)
(voiceless velar fricative) [
HASOC2021: Hate Speech and Ofensive Content Identification is a competition that helps increase research in ofensive language identification in code mixed languages such as TamilEnglish and Malayalam-English [6]. The dataset consists of comments/posts that were collected from Youtube and social media. Each comment/post is annotated with an ofensive language label at the comment/post level. This dataset also has class imbalance problems depicting real-world scenarios.
In this paper, we present our system developed for HASOC 2021; the rest of the paper is organized as follows. Section 2 discusses the research work on ofensive language identification and natural language processing in under-resourced languages. Following this, in section 3, we present the methodology of our work, from preprocessing the dataset, our model architecture, and training procedures. In section 4, we discuss our results. Finally, in section 6, we conclude with a summary and future work.
Ofensive Language identification has been widely made across many people in multiple
languages. Shared tasks like HASOC-19 [
Researchers have used a wide variety of techniques for the identification of ofensive language. There have been previous work [32] in using classical machine learning models with eficient feature generation. Other researchers in [ 33] [34] have used an ULMFit model [35] and pretrained XLM-Roberta model with translated and transliterated texts for this task.
This section briefly describes our methodology for this task, including data preparation, model architecture, and training strategies. For this HASOC 2021 competition, we only use the datasets that were provided for the HASOC task. Table 1 shows the statistics of the train and dev distribution.
The dataset given for subtask, Ofensive Language Identification in Tamil-English, consists of Youtube comments, present in code-mixed data containing text written in both native and roman scripts in English.
We use pretrained transformer models with custom pooled output for this task of identifying ofensive content. We have used mBERT and MuRIL pretrained models from huggingface checkpoints. In this section, we describe our pooling operations on the pretrained models and the pretrained models.
Attention Pooler: In this method, the attention operation described in the below equation is applied to the last hidden state of the pretrained transformer; empirically, this helps the model learn the contribution of individual tokens. Finally, the returned pooled output from the transformer is further passed to a linear layer for the prediction of the label. where ℎ and are learnable weights.
= ℎ
(ℎ = (
)ℎ + )
Mean Pooler: In this method, the average of the last layer of the pretrained transformer is taken. This acts like a pooling layer in a convolutional neural net. An alternative to this method is to use max pooling, but max-pooling selects only the words with essential features rather than utilizing all the words. Since our dataset is code-mixed and the spelling of the tokens are not precise, we choose to go with mean pooling approach.
mBERT Multilingual models of BERT [36]. This model was pre-trained using the same pretraining strategy that was employed to BERT, which is Masked Language Modeling (MLM) and Next Sentence Prediction (NSP). It was pretrained on the Wikipedia dump of top 104 languages. To account for the data imbalance due to the size of Wikipedia for a given language, exponentially smoothed weighting of data was performed during data creation and word piece vocabulary creation. This results in high resource languages being under-sampled while low resourced languages being over-sampled.
MuRIL MuRIL [37], pretrained model is trained on 16 diferent Indian Languages; the model was pretrained on Masked Language Modeling(MLM) and Translated Language Modelling(TLM). This model outperforms mBERT on all the tasks in XTREME [38] (1) (2)
Though finetuning transformers gives better results and is dominant across leaderboards of various NLP competitions. Transformer models are still unstable due to catastrophic forgetting [39]. For this ofensive language identification task, we carefully choose our hyperparameters for experimentation. We finetune our custom models with binary-cross-entropy loss and AdamW optimizer, which decouples the weight decay from the optimization step. Linear scheduler for learning rate scheduling with 2e-5 as an initial step is used with this training strategy. The training hyperparameters are listed in Table 2.
In this HASOC 2021 competition, the teams were ranked by the weighted F1-score of their classification system. This section discusses our experimental results; since we have used both training and dev sets for training, the train set in the dataset distribution refers to the concatenated given train and dev sets. The W-Precision, W-Recall, and W-F1-Score refer to the Weighted precision, weighted recall, and weighted F1-Score. Table 3 shows our results obtained before the task deadline using Attention Pooler and mBERT without sampling the dataset. After the task deadline, we uniformly sample our dataset and run or experiments on MuRIL and mBERT with Attention Pooling and Mean Pooling. The results are provided in Table 4 and Table 5. The W-precision, W-Recall and W-F1 Score stands for Weighted Precision, Weighted Recall and Weighted F1-Score.
From the above results, we conclude that the pretrained model MuRIL with MeanPooler performs best than others. Also, one can infer that the diference between training and test scores shows that the model is sufering from overfitting, and also sampling the dataset uniformly is a crucial step to increasing the score.
In this paper, we have presented our solution for the Ofensive Language Identification system, which uses pretrained transformers mBERT and MuRIL. As a result, we achieve Rank 9 on the leaderboard and a 0.67 f1-score after the task deadline. For future research, we will consider improving the results by using any external dataset and other pretrained models and reducing the generalization error of the model. B. R. Chakravarthi, Benchmarking multi-task learning for sentiment analysis and offensive language identification in under-resourced Dravidian languages, arXiv preprint arXiv:2108.03867 (2021). [6] B. R. Chakravarthi, P. K. Kumaresan, R. Sakuntharaj, A. K. Madasamy, S. Thavareesan, P. B, S. Chinnaudayar Navaneethakrishnan, J. P. McCrae, T. Mandl, Overview of the HASOC-DravidianCodeMix Shared Task on Ofensive Language Detection in Tamil and Malayalam, in: Working Notes of FIRE 2021 - Forum for Information Retrieval Evaluation, CEUR, 2021. [7] R. Priyadharshini, B. R. Chakravarthi, S. Thavareesan, D. Chinnappa, T. Durairaj, E. Sherly, Overview of the DravidianCodeMix 2021 Shared Task on Sentiment Detection in Tamil, Malayalam, and Kannada, in: Forum for Information Retrieval Evaluation, FIRE 2021, Association for Computing Machinery, 2021. [8] B. R. Chakravarthi, R. Priyadharshini, V. Muralidaran, N. Jose, S. Suryawanshi, E. Sherly, J. P.
McCrae, DravidianCodeMix: Sentiment Analysis and Ofensive Language Identification Dataset for Dravidian Languages in Code-Mixed Text, arXiv preprint arXiv:2106.09460 (2021). [9] B. R. Chakravarthi, K. Soman, R. Ponnusamy, P. K. Kumaresan, K. P. Thamburaj, J. P.
McCrae, et al., DravidianMultiModality: A Dataset for Multi-modal Sentiment Analysis in Tamil and Malayalam, arXiv preprint arXiv:2106.04853 (2021). [10] B. R. Chakravarthi, Leveraging orthographic information to improve machine translation of under-resourced languages, Ph.D. thesis, NUI Galway, 2020. [11] D. Chinnappa, P. Dhandapani, Tamil lyrics corpus: Analysis and experiments, in: Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages, Association for Computational Linguistics, Kyiv, 2021, pp. 1–9. URL: https://aclanthology.org/2021.dravidianlangtech-1.1. [12] J. J. Andrew, JudithJeyafreedaAndrew@DravidianLangTech-EACL2021:ofensive language detection for Dravidian code-mixed YouTube comments, in: Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages, Association for Computational Linguistics, Kyiv, 2021, pp. 169–174. URL: https://aclanthology.org/2021. dravidianlangtech-1.22. [13] K. Sarveswaran, G. Dias, M. Butt, ThamizhiMorph: A morphological parser for the Tamil language, Machine Translation 35 (2021) 37–70. [14] B. Bharathi, A. S. A, SSNCSE_NLP@DravidianLangTech-EACL2021: Ofensive language identification on multilingual code mixing text, in: Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages, Association for Computational Linguistics, Kyiv, 2021, pp. 313–318. URL: https://aclanthology.org/2021. dravidianlangtech-1.45. [15] B. Bharathi, A. S. A, SSNCSE_NLP@DravidianLangTech-EACL2021: Meme classification for Tamil using machine learning approach, in: Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages, Association for Computational Linguistics, Kyiv, 2021, pp. 336–339. URL: https://aclanthology.org/2021. dravidianlangtech-1.49. [16] S. Thavareesan, S. Mahesan, Word embedding-based Part of Speech tagging in Tamil texts, in: 2020 IEEE 15th International Conference on Industrial and Information Systems (ICIIS), in Tamil, in: Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages, Association for Computational Linguistics, Kyiv, 2021, pp. 126–132.
URL: https://aclanthology.org/2021.dravidianlangtech-1.16. [32] B. Lakshmanan, S. K. Ravindranath, Theedhum nandrum@dravidian-codemix-fire2020: A sentiment polarity classifier for youtube comments with code-switching between tamil, malayalam and english, 2020. a r X i v : 2 0 1 0 . 0 3 1 8 9 . [33] C. Vasantharajan, U. Thayasivam, Hypers@DravidianLangTech-EACL2021: Ofensive language identification in Dravidian code-mixed YouTube comments and posts, in: Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages, Association for Computational Linguistics, Kyiv, 2021, pp. 195–202. URL: https://aclanthology.org/2021.dravidianlangtech-1.26. [34] S. Sai, Y. Sharma, Towards ofensive language identification for Dravidian languages, in: Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages, Association for Computational Linguistics, Kyiv, 2021, pp. 18–27. URL: https: //aclanthology.org/2021.dravidianlangtech-1.3. [35] J. Howard, S. Ruder, Fine-tuned language models for text classification, CoRR abs/1801.06146 (2018). URL: http://arxiv.org/abs/1801.06146. a r X i v : 1 8 0 1 . 0 6 1 4 6 . [36] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, 2019. a r X i v : 1 8 1 0 . 0 4 8 0 5 . [37] S. Khanuja, D. Bansal, S. Mehtani, S. Khosla, A. Dey, B. Gopalan, D. K. Margam, P. Aggarwal, R. T. Nagipogu, S. Dave, S. Gupta, S. C. B. Gali, V. Subramanian, P. Talukdar, Muril: Multilingual representations for indian languages, 2021. a r X i v : 2 1 0 3 . 1 0 7 3 0 . [38] J. Hu, S. Ruder, A. Siddhant, G. Neubig, O. Firat, M. Johnson, Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization, 2020. a r X i v : 2 0 0 3 . 1 1 0 8 0 . [39] M. Mosbach, M. Andriushchenko, D. Klakow, On the stability of fine-tuning bert: Misconceptions, explanations, and strong baselines, 2021. a r X i v : 2 0 0 6 . 0 4 8 8 4 .