This paper describes the system submitted to Dravidian-Codemix-HASOC2020: Hate Speech and Ofensive Content Identification in Dravidian languages (Tamil-English and Malayalam-English). The task aims to identify ofensive language in code-mixed dataset of comments/posts in Dravidian languages collected from social media. We participated in both Sub-task A, which aims to identify ofensive content in mixed-script (mixture of Native and Roman script) and Sub-task B, which aims to identify ofensive content in Roman script, for Dravidian languages. In order to address these tasks, we proposed pretraining ULMFiT on synthetically generated code-mixed data, generated by modelling code-mixed data generation as a Markov process using Markov chains. Our model achieved 0.88 weighted F1-score for code-mixed Tamil-English language in Sub-task B and got 2nd rank on the leader-board. Additionally, our model achieved 0.91 weighted F1-score (4th Rank) for mixed-script Malayalam-English in Sub-task A and 0.74 weighted F1-score (5th Rank) for code-mixed Malayalam-English language in Sub-task B.
languages collected from social media. Each comment/post is annotated with
ofensive/notofensive label at the comment/post level. The data set has been collected from YouTube
comments and tweets. It has 2 sub-tasks. Sub-task A is a message-level label classification
task in which given a YouTube comment in Code-mixed (Mixture of Native and Roman Script)
Malayalam, systems have to classify it into ofensive or not-ofensive. Sub-task B is also a
message-level label classification task in which given a tweet or YouTube comment in Tanglish
or Manglish (Tamil and Malayalam written using Roman Script), systems have to classify it into
ofensive or not-ofensive [
In this competition, we participated in both sub-task A and sub-task B. We pre-trained
ULMFiT [
Lot of researchers and practitioners from industry and academia have been attracted towards
the problem of automatic identification of hateful and ofensive speech. Authors in [
There have been attempts at synthetically generating code-mixed data based on linguistic theory [10]. But in this paper, we use a novel technique to generate synthetic code-mixed data and pre-train ULMFiT model for Dravidian languages.
In this section we discuss data preparation for pre-training ULMFiT language model, get insights into the Dravidian code-mix HASOC classification train, dev and test sets and discuss modeling details.
Data preparation for pre-training ULMFiT language model. Firstly, we obtain parallel sets of Wikipedia articles in native, transliterated and translated form. This gives us one to one mapping between sentences in native, transliterated and translated form. This allows us to model code-mixed data generation process as a Markov Process with 3 statneast; i v e , t r a n s l a t e d and t r a n s l i t e r a t e d with p 1 , p 2 , p 3 , q 1 , q 2 , q 3 , r 1 , r 2 , r 3 as state transition probabilities as shown in Figure1. We sample from this Markov process from timeT = 0 to time T = N , where
={native, transliterated, transliterated, native, translated, .......} depending upon the values of state transition probabilities. Usi ng ℎ sentence at = from ’s ℎ state. Example of state transition probabilities used for , we form code-mixed data by picking synthetically generating Malayalam code-mixed data are shown in T1a.ble
Dravidian code-mix HASOC classification dataset.
Sub-task A Sub-task B Malayalam mixed-script Malayalam Test 400 64.5% 1 8 186
3.2.1. Model Architecture
Weight-dropped AWD-LSTM variant1[
tokenization, our models can handle diferent spellings in transliteration or spelling errors in test set. Table4 shows vocabulary size of the models trained for each one of the sub-tasks.
ULMFiT Language model pre-training. Our model is based on the Fasta4iimplementation of ULMFiT. Table5 shows perplexity of the trained models on validation set. Pre-trained models along with datasets can be downloaded and used for several other downstream classification tasks from the GitHub repository.
Preprocessing of Dravidian code-mix HASOC dataset. Basic preprocessing was done by lower-casing, removing @username mentions etc. Exact reproduction details are available in the GitHub repository.
Classifier training. Classifier was trained by first fine-tuning the pre-trained ULMFiT language model and then training the classifier.
We did basic pre-processing detailed in the previous section. For each one of the sub-tasks, we split the training data into standard train-validation splits in the ratio of 80:20, in case explicit validation set was not given. Using this split, we perform experiments to identify the best set of hyperparameters, shown in Tabl6e. Gradual unfreezing epochs and learning rates are accessible from notebooks in GitHub repository. Backpropagation through time (BPTT) was set to 70. For language model fine-tuning, dropout multiplicity was set to 0.3, batch size was set to 64, the model was trained for 1 epoch with learning rate 1e-2 with only last layer unfreezed and 5 epochs with learning rate 1e-3 after unfreezing all layers.
In this competition, teams were ranked by the weighted F1 score of their classification system. Tables7, 8 and 9 show Top-5 ranked teams in the competition with their weighted Precision, weighted Recall and weighted F1-score for each one of the sub-tasks. Diference between top
Task Sub-Task A Sub-Task B
Malayalam Mixed Script Malayalam
Tamil
Dropout Multiplicity
Batch Size
Epochs
Model Unfreezed
Learning Rate 0.5 ranked team’s weighted F1-score and our weighted F1-score is less than 0.04 in all the sub-tasks. Perplexity of ULMFiT language model after fine-tuning was 57.53 for mixed-script Malayalam in sub-task A, 71.02 for code-mixed Tamil and 89.12 for code-mixed Malayalam in Latin script in sub-task B.
In this paper, we presented a Markov model to facilitate synthetic generation of code-mixed data and used it to pre-train ULMFiT language model on Dravidian languages. Using transfer learning with the pre-trained ULMFiT model on downstream task of hate speech detection, we achieved Rank 2 for Tamil in Sub-task B, Rank 4 for Malayalam in Sub-task A and Rank 5 for Malayalam in Sub-task B. In future research, we will consider improving synthetically generating code-mixed data by allowing code-mixing within a single sentence and using Transformer based models. [10] A. Pratapa, G. Bhat, M. Choudhury, S. Sitaram, S. Dandapat, K. Bali, Language modeling for code-mixing: The role of linguistic theory based synthetic data, 2018, pp. 1543–1553. doi:10.18653/v1/P18- 1143. [11] S. Merity, N. S. Keskar, R. Socher, Regularizing and optimizing LSTM language models,
CoRR abs/1708.02182 (2017). URL: http://arxiv.org/abs/1708.0218.2arXiv:1708.02182. [12] S. Hochreiter, J. Schmidhuber, Long short-term memory, Neural Comput. 9 (1997) 1735–1780. URL: https://doi.org/10.1162/neco.1997.9.8.1735. doi:10.1162/neco.1997.9.8. 1735. [13] F. A. Gers, J. A. Schmidhuber, F. A. Cummins, Learning to forget: Continual prediction with lstm, Neural Comput. 12 (2000) 2451–2471. URLh:ttps://doi.org/10.1162/ 089976600300015015. doi:10.1162/089976600300015015. [14] T. Kudo, J. Richardson, Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing, CoRR abs/1808.06226 (2018). URL: http://arxiv.org/abs/1808.0622.6arXiv:1808.06226.