This paper describes our system submitted to HASOC-Dravidian-CodeMix task at FIRE 2020. The goal of the task was to detect hate speech and ofensive content for Dravidian Languages. With the advent of various social media platforms, a large number of people from various communities engage online, on a daily basis. Hence, it is essential to develop robust mechanisms to tackle hate speech and ofensive contents posted online as it can potentially impact individuals and communities. In India, most people use both English, and their mother tongue interchangeably, which results in code-mixing and script-mixing of local Indian languages with English. This shared task aims to deal with challenges arising due to code-mixing and script mixing of Tamil and Malayalam language with English, for the hate speech detection task. We employ an ensemble of multilingual BERT models for this task and devise a novel training strategy involving data augmentation using random transliteration. We achieve an F-score of 0.95 for hate speech and ofensive content detection on the Malayalam code-mixed YouTube comments test data in task 1. In task 2, we achieve F-scores of 0.86 and 0.72 respectively for hate speech and ofensive content detection on Tamil and Malayalam code-mixed Twitter test data. We have made our system publicly available at GitHub.
In recent years, the growth of the Internet has been exponential, which derived the boom of
social media platforms. With more than half of the population of the earth having access to
the Internet, it has become practically impossible to manually track and monitor the content
posted online on various social media platforms. Most of the social media platforms have a
very loose identity check system which makes it easy for someone to post hate speech and
ofensive content online without revealing their actual identity. Therefore the need for robust
and automatic hate speech and ofensive content detection systems is now stronger than ever.
Hence this area of research has caught the attention of both industry and academia in recent
years. HASOC-Dravidian-CodeMix [
© 2020 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
In the previous version of HASOC [
HASOC-Dravidian-CodeMix is a subt-rack of HASOC: FIRE 2020 which consist of two shared tasks of detecting hate and ofensive text in Youtube and Twitter data. The use of script and code mixed language is very common for Indian users, on social media platforms. These users tend to mix local Indian languages with English. The dataset provided in the task also depicts this real-world scenario of code and script mixing, which is an interesting NLP problem. Two tasks proposed in HASOC-Dravidian-CodeMix are the following: • Task 1: Hate speech and ofensive content detection in Youtube comments in Malayalam language (Code-mixed, Script-mixed) • Task 2: Hate speech and ofensive content detection in Youtube or Twitter dataset in
Malayalam languages (Code-mixed: Manglish and Tanglish)
We proposed a system based on an ensemble of multilingual transformers [10] and achieved very good results in both tasks. A novel training strategy involving data augmentation by randomly transliterating the text during training resulted in a robust system for task 1. In section 2 we explain our system and training strategy in detail. In section 3 we report our performance in both tasks of the competition.
This section details the dataset, pre-processing steps, network architecture and training strategy for both shared tasks.
We use the datasets provided by the organizers, for both the tasks mentioned above. Task 1 has Youtube comments in the Malayalam language as text which are categorized into hate or ofensive and normal. The text in this task is both code and script mixed. It has both Dravidian and Roman script along with a mixed grammatical structure. This type of mixed structure allowed us to devise a data augmentation strategy in which we randomly transliterate words in Roman script to Malayalam with 0.5 probability during training. Task 2 has text from Youtube or Twitter in Malayalam and Tamil. The text follows the grammatical structure of both Indian languages and English but the text is majorly in Roman script. Here also, each text instance is labelled as hate or ofensive and normal. Table 1 shows the dataset sizes and the distribution of the positive and negative class samples. In task 1, the dataset is imbalanced, and only 16.50% of the text is labelled as hate or ofensive text which depicts real word scenarios where most of the text on social media is not be hate or ofensive text. In task 2, the datasets are fairly balanced between the two classes.
The core of our system is multilingual BERT [10] which is used to obtain vector representation corresponding to each sentence. We use the final hidden state vector of special [CLS] token as an aggregate representation of the entire input sentence [11]. This vector is then passed through 2 fully connected layers and a softmax layer to obtain the final prediction. Finally, we use an ensemble of multilingual BERT models trained with diferent setups to obtain the final prediction. We explain these setups in more detail in section 3.
We also perform some basic pre-processing steps to clean the text. The following information was removed from the text, which is not much important for hate speech and ofensive content detection: • @mention from and RT from tweets • website URLs • digits • replacing multiple spaces with a single space
The original BERT paper [11] suggested updating the weights of the entire model for fine-tuning it on new tasks. Since our dataset was relatively small, this strategy of updating all weight of the model at once resulted in instability in the convergence of the loss function. Therefore, we employed gradual unfreezing of network layers one by one. For the first ten epochs, we trained only the last fully connected and softmax layers using cross-entropy loss and froze the parameters of all BERT layers. In the next ten epochs, we unfreeze the last (12th) BERT layer and update its weights during training. We kept repeating the process of unfreezing one layer at a time after every 10 epochs. This strategy of unfreezing BERT layers one by one resulted in smooth convergence of loss function. We trained our system for 150 epochs with appropriate values of the batch size, learning rate and weight decay which were obtained after extensive hyper-parameter tuning for each task individually.
We also employ a data augmentation strategy during the training of the multilingual BERT model. Analysis of the training and test data reveals that input text from Youtube and Twitter contains grammatical structures of both English and Indian languages. In task 1 there are data samples that contain Dravidian and Roman scripts in either transliterated or original form. We exploited this peculiarity of the dataset to devise a data augmentation strategy by randomly transliterating the text during training with a 0.5 probability. For transliteration, the Indic NLP library [12] was used which converted the input text to either Roman or English script completely. This data augmentation strategy proved to be very useful and improved the overall performance of the system.
In this section, we shed light on various experiments conducted by us for the given tasks and selection of our final systems for the competition. We also report the oficial results of our system on the test set.
We ran multiple experiments with diferent setups and selected the best performing setups for creating a hard vote based ensemble of transformers. The best-performing setups selected for the ensemble were the following: • Fine-tuning multilingual BERT with binary cross-entropy loss and assisted by data augmentation using random transliteration. • Transliterating the entire dataset in the Dravidian script and fine-tuning the multilingual
BERT on this common script dataset. • Fine-tuning multilingual BERT with weighted binary cross-entropy loss, to overcome the class imbalance problem, for task 1.
We participated in all categories of shared tasks. In Table 2, we report oficial results of our submitted systems published by the organizers of HASOC-Dravidian-CodeMix FIRE 2020. Precision, recall and f-score metrics were used to assess the performance of the submitted system. In validations sets, the performance of the system was similar to oficial results. On average, performance on the validation set was 0.02 absolute value more than the oficial results. Due to data augmentation using the transliteration step in task 1, the f1-score of the system was improved from 0.91 to 0.95 on the validation set. Given the complexity of the task and various challenges involved, these results are pretty encouraging and prove the efectiveness of our end-to-end system.
The need to have robust hate speech and ofensive content detection systems which can also deal with the problems arising due to multilinguality has become very obvious in recent years. We presented an end-to-end system that efectively handled multilinguality and gave very competitive results in hate speech and ofensive content detection tasks. Our approach also establishes the eficacy of data augmentation by randomly transliterating the input sentences during training.
In the future, we aim to develop a single system for hate speech and ofensive content detection system for multiple languages and multiple tasks instead of one model per task and language. The multilingual BERT model is quite powerful in learning complex mappings from input to output, and sometimes it overfits when training data size is small. Joint training of multiple tasks across multiple languages may produce a regularization efect which will help the model to generalize better.
We thank all the organizers of HASOC-Dravidian-CodeMix FIRE 2020 for presenting us with the opportunity to tackle a very interesting and relevant problem and providing their timely support during the competition. Natural Language Processing (KONVENS 2018), Vienna, Austria – September 21, 2018, Austrian Academy of Sciences, Vienna, Austria, 2018, pp. 1 – 10. URL: http://nbn-resolving. de/urn:nbn:de:bsz:mh39-84935. [6] V. Basile, C. Bosco, E. Fersini, D. Nozza, V. Patti, F. M. Rangel Pardo, P. Rosso, M. Sanguinetti, SemEval-2019 task 5: Multilingual detection of hate speech against immigrants and women in Twitter, in: Proceedings of the 13th International Workshop on Semantic Evaluation, Association for Computational Linguistics, Minneapolis, Minnesota, USA, 2019, pp. 54–63.
URL: https://www.aclweb.org/anthology/S19-2007. doi:1 0 . 1 8 6 5 3 / v 1 / S 1 9 - 2 0 0 7 . [7] M. Zampieri, S. Malmasi, P. Nakov, S. Rosenthal, N. Farra, R. Kumar, SemEval-2019 task 6: Identifying and categorizing ofensive language in social media (OfensEval), in: Proceedings of the 13th International Workshop on Semantic Evaluation, Association for Computational Linguistics, Minneapolis, Minnesota, USA, 2019, pp. 75–86. URL: https: //www.aclweb.org/anthology/S19-2010. doi:1 0 . 1 8 6 5 3 / v 1 / S 1 9 - 2 0 1 0 . [8] B. R. Chakravarthi, V. Muralidaran, R. Priyadharshini, J. P. McCrae, Corpus creation for sentiment analysis in code-mixed Tamil-English text, in: Proceedings of the 1st Joint Workshop on Spoken Language Technologies for Under-resourced languages (SLTU) and Collaboration and Computing for Under-Resourced Languages (CCURL), European Language Resources association, Marseille, France, 2020, pp. 202–210. URL: https://www. aclweb.org/anthology/2020.sltu-1.28. [9] B. R. Chakravarthi, N. Jose, S. Suryawanshi, E. Sherly, J. P. McCrae, A sentiment analysis dataset for code-mixed Malayalam-English, in: Proceedings of the 1st Joint Workshop on Spoken Language Technologies for Under-resourced languages (SLTU) and Collaboration and Computing for Under-Resourced Languages (CCURL), European Language Resources association, Marseille, France, 2020, pp. 177–184. URL: https://www.aclweb.org/anthology/ 2020.sltu-1.25. [10] T. Pires, E. Schlinger, D. Garrette, How multilingual is multilingual BERT?, in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Florence, Italy, 2019, pp. 4996–5001. URL: https://www. aclweb.org/anthology/P19-1493. doi:1 0 . 1 8 6 5 3 / v 1 / P 1 9 - 1 4 9 3 . [11] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, BERT: Pre-training of deep bidirectional transformers for language understanding, in: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Association for Computational Linguistics, Minneapolis, Minnesota, 2019, pp. 4171–4186. URL: https://www.aclweb.org/ anthology/N19-1423. doi:1 0 . 1 8 6 5 3 / v 1 / N 1 9 - 1 4 2 3 . [12] A. Kunchukuttan, The IndicNLP Library, https://github.com/anoopkunchukuttan/indic_ nlp_library/blob/master/docs/indicnlp.pdf, 2020.