To tackle the conundrum of detecting ofensive comments/posts which are considerably informal, unstructured, miswritten and code-mixed, we introduce two inventive methods in this research paper. Ofensive comments/posts on the social media platforms, can afect an individual, a group or underage alike. In order to classify comments/posts in two popular Dravidian languages, Tamil and Malayalam, as a part of the HASOC - DravidianCodeMix FIRE 2021 shared task, we employ two Transformer-based prototypes which successfully stood in the top 8 for all the tasks. The codes for our approach can be viewed and utilized1.
despite understanding that they are infringing on their right to free expression. This may have a negative and adverse impact on user’s mental health. Various social media platforms restrict and minimize the profane comments by employing new rules and techniques.
Online hate speech and ofensive content produces challenges to the society. The detection of hate speech is quite a daunting task, as the precise understanding of the speech largely depends on the circumstances it is being used in. Due to the extreme enormity of the internet and the growing number of online users, as well as the obscurity of the users, manually detecting and removing hate speech and profane information is a time-consuming and challenging job.
Code-mixing often entails the use of two languages to produce a third language that
incorporates aspects from each in a functionally comprehensible manner. Low-resource languages
such as Tamil and Malayalam are gaining significant attention alongside English on social
networking platforms. The majority of the data on social media for these under-resourced
languages is code-mixed. Tamil is a Dravidian language spoken by Tamils in India and Sri Lanka,
as well as the Tamil community worldwide [
This paper presents our work for the shared task on ofensive language detection of codemixed text in Dravidian languages (Malayalam-English and Tamil-English) at HASOC - DravidianCodeMix FIRE 2021. The rest of the paper is summarized as follows, 2 presents a discussion on the previous works on Ofensive Language Detection in Dravidian Languages. 3 entails a detailed task description and analysis of the datasets for Tamil, Malayalam, and Kannada. In 4 we present a description of the models used for the tasks.
There has been a tremendous advancement in the research of ofensive language detection over
the past few years. On social media, hate speech in the form of racist and sexist statements is quite
commonplace. In Waseem and Hovy, the authors provided a dataset of 16k tweets annotated for
hate speech and analysed the features that help detect hate speech in the corpus. The authors
of Davidson et al. used logistical regression to extract N-gram TF-IDF features from tweets
and categorize each tweet into hate, ofensive, and non-ofensive categories. For identifying
abusive language, the authors of Hassan et al. experimented with Support Vector Machines
(SVMs) [
In Liu et al., the authors experimented with various classifiers which includes linear model
with features of word unigrams, word2vec, and Hatebase; word-based Long Short-Term Memory
(LSTM) [
The main aim of the HASOC Shared task is to identify ofensive content in the code-mixed
comments/posts in the Dravidian languages collected from social media [
We are provided with two diferent datasets for the two subtasks. The training dataset provided for task 1 comprised of 5877 YouTube comments in Tamil classified into ofensive or not-ofensive. The task 1 was limited to Tamil language. The dataset is also observed to be imbalanced. The training dataset provided for task 2 comprised of 4000 tweets in code mixed Tamil and 3999 tweets in code mixed Malayalam. The training and validation datasets provided for task 2 are well-balanced.
Language not ofensive ofensive Total
Tamil (Task 1) 4,724 1,153 5,877
Tamil (Task 2) 2,020 1,980 4,000
Malayalam (Task 2) 2047 1952 3,999
4.1. Task 1 We fine tune transformer based language models for this task. Firstly, emoticons and flags were cleaned from the dataset. Then the sentences were converted to lower case as some of the samples contains English text between them. We then pass the sequences through two Feed Forward Layer Global Average Pooling pre-trained models namely XLM-R and DistilBERT extracting the embeddings from both. These embeddings were then concatenated before being passed through the BiLSTM layers [20], eventually being pooled on a global average scale. These are fed to some Fully Connected layers and an Activation Function of sigmoid to get the probability scores as shown in Figure 1. By concatenating the embeddings, we expect that the model we created can benefit from knowledge of both the NLP models employed for the task, helping to distinguish better among the classes. 4.2. Task 2 For this task, we first transliterate [ 21] the Tamil sentences library in the English script to the native Tamil script by usage of “indic-transliteration” library1. We then translate these sentences to English using the Google Translate API[22]. We then clean this parallel corpora of sentences by removing punctuations, stripping unwanted spaces at the end and converting the English sentences to a lower case. After the preprocessing, we tokenize these sequences using a tokenizer of a multilingual model, XLM-R. These tokens of Tamil and English are fed through the same XLM-R model and then passed through BiLSTM layers and a pooling layer at the end. Then we compute the weighted average of the Tamil and English vectors, with weights of 0.7 for Tamil and 0.3 for English. An Activation Function of sigmoid [23] is also applied at the end, deriving the probability scores required to classify a sentence. The entire architecture is shown
in Figure 2. The same technique is done for Malayalam sentences as well, here the sentences being transliterated to Malayalam and the weights being 0.6 for Malayalam and 0.4 for English. These weights were set upon experimentation with various values and then selecting the best from all of them. Refer table 2 for parameters used in this task.
Parameters Number of LSTM layers Number of LSTM units Batch Size Max Length Optimizer Learning Rate Activation Function Loss Function
XLM-R [24] is a multilingual language model that achieved state-of-the-art results in all the multiple cross lingual benchmarks. One of the reason for the unparalleled performance is that it was trained on a mammoth 2.5 TB of CommonCrawl data [25]. It was trained with MLM loss as it’s objective on 100 diferent languages, and it shares similar training routine as the one employed for RoBERTa [26] which is the reason the model is called XLM RoBERTa. XLM Roberta is fine-tuned for both of these tasks in a diferent architecture employed for the specific task. For this task we use xlm-roberta-base which consists of 12-layers, 768-hidden-state, 8-heads and a parameter size of 270M. The reason for selecting XLM-R is it outperformed other models in all the multilingual benchmarks.
DistilBERT [27] is BERT’s distilled version. It employs a triple loss language model that combines language modelling, distillation and cosine-distance losses. The two distillation losses in the triple loss have a significant influence on model performance. We fine tune distilbert-basemultilingual-cased, which is distilled from the mBERT checkpoint, for our cause in Task 1. It is known to have 40% less number of parameters than mBERT and runs 60% faster than it. The model consists of 6 layers, 768 dimensions, and 12 Attention heads, with a total of around 134 million parameters. We chose DistilBERT due to the less size of the model making it extract the word embeddings quicker.
6.1. Task 1 The system description model on the Tamil language for this task gave a promising F1 score of 0.810. Embeddings from two of the most eficient multilingual pretrained models, XLM-R and DistilBERT were concatenated to extract their significant features. Further, the use of BiLSTM layers improves the accuracy as the information being fed doubles. The BiLSTM layers contains two LSTM’s which take the input from the forward and backward directions and, thus, enhancing the context. However, one of the drawbacks which causes an impediment is the class imbalance between the “ofensive” and “not-ofensive” sentences in the test dataset. Also, XLM-R and DistilBERT follow BERT-based architectures and hence, the embeddings produced don’t generate huge variations. Employment of pretrained models belonging to other architectures could produce higher accuracies. Increasing the number of models also might enhance the quality of embeddings produced, and thus, boosting the F1 scores further to 0.810. 6.2. Task 2 It was found out that our model gave an F1 score of 0.612 for Tamil and 0.670 for Malayalam. The dataset contains Tamil and Malayalam comments written in the Roman script, which is hard for Task 1 the multilingual pretrained models trained on the native scripts to comprehend. Transliterating these sentences to the native language can prove to increase the F1 scores. Furthermore, we know that the models like XLM-R is trained on a large corpus of English sentences. Thus, the English translations of these transliterated dataset plays a huge role in further fine-tuning of the model. With the test dataset again containing sentences in native languages written in the Roman script, it was essential to give a higher precedence to the transliterated tokens over the translated tokens [28]. BiLSTM once again plays its role in ameliorating the results by increasing the information being fed.
However, we can never be definite of the accuracy in the transliterations and translations. Reduced quality if these sentences can afect the fine-tuning of the model significantly, and hence, lowering the F1 scores. Class imbalance prevails in this dataset too. With the ratio of not ofensive to ofensive in the range of 2:1, the model seems to find it arduous to predict the P P P P
Overall Overall Overall Overall
R R R R
F F F F
Acc ofensive sentences eficiently. It is observed that the F1 scores between the not ofensive and ofensive difer by 0.15 to 0.3. This also impacts the overall F1 score for this task. However, as we can see in Table 4, the diference between the F1-scores of the “ofensive” and “not-ofensive” labels on the validation dataset didn’t seem to vary much. Table 3 tabulates the detailed weighted F1 scores for the test dataset.
Few of the notable sentences where we felt that the sentences were misclassified have been discussed in this section. In task 1, 528 sentences were classified correctly in Tamil, while 126 failed to be classified well. There can be several reasons for this misclassification. Sometimes, the presence of ofensive words doesn’t ensure that the sentence is “Ofensive” and vice-versa. Comments are also filled with sarcasms, puns and typographical errors which have high probability of getting classified wrongly. Second task in Tamil has 645 sentences classified correctly and 356 wrongly. In Malayalam, 713 are correct and 238 wrong. We have discussed few of the Tamil sentences,
adey kirukka nalla paru„,google unaku theriyuma„„ 2rs eppadi ellam pesura„,Sanghis This sentence is tagged as “not ofensive”, but it is directed towards North Indians and probably as a reply to another comment/post.
Inta treylar kuta parkkira matiri illai.. Itai tiyettar la poy parkkanuma The sentence is classified as ofensive, but it is just a review which states, “this trailer itself isn’t good. Does someone have to go to the theatres too to watch this?”.
Another major drawback was the poor quality of translations by the Google API. The accuracy of classifications would have been better if the translations were of good quality in all the cases. For example,
There is no other way is a very good translation of the Malayalam sentence and the model is able to learn and predict well.
Sentence 2: aga surya um jothikaum etho plan pani taga pola,not,Aga Surya Uma Jyotika is like something Plan Bani Daka Aga Surya Um Jyotikaum Something like Plan snow is an example of how some sentences get partially got converted to English.
Sentence 3: aaiiii jolly yellam onnah polam onnah polam oannaaa polam update app to view IEE Jolly Yellam Onnah Bolam Onnah Bolam Oannaa Polam Uptade App To Viev is the supposed to be the English translation of the above sentence. We can see that the quality of the translation is very bad.
Ofensive language detection in social media posts presents to be a significant task due to social and marketing rationale. For the task of ofensive language detection in code-mixed Dravidian languages Tamil and Malayalam, we introduce our research in this paper. For task 1, we extract the embeddings from XLM-R and DistilBERT, we concatenate them and pass them through BiLSTM layers. This model managed to give an F1 score of 0.810. Similarly, for task 2 we transliterate the dataset in which the Dravidian language is written in the Roman script and then, translate them into English and fine-tune the XLM-R model on it. This model gives us F1 scores of 0.612 for Tamil and 0.670 for Malayalam. Neglecting the fact that the English translations were of a poor quality, the model achieves very decent F1 scores for both the languages and, thus, opening a gateway for more research in this field. P. B, S. Chinnaudayar Navaneethakrishnan, J. P. McCrae, T. Mandl, Overview of the HASOC-DravidianCodeMix Shared Task on Ofensive Language Detection in Tamil and Malayalam, in: Working Notes of FIRE 2021 - Forum for Information Retrieval Evaluation, CEUR, 2021. [20] G. Xu, Y. Meng, X. Qiu, Z. Yu, X. Wu, Sentiment analysis of comment texts based on bilstm,
Ieee Access 7 (2019) 51522–51532. [21] K. Regmi, J. Naidoo, P. Pilkington, Understanding the processes of translation and transliteration in qualitative research, International Journal of Qualitative Methods 9 (2010) 16–26. [22] Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey, J. Klingner, A. Shah, M. Johnson, X. Liu, Łukasz Kaiser, S. Gouws, Y. Kato, T. Kudo, H. Kazawa, K. Stevens, G. Kurian, N. Patil, W. Wang, C. Young, J. Smith, J. Riesa, A. Rudnick, O. Vinyals, G. Corrado, M. Hughes, J. Dean, Google’s neural machine translation system: Bridging the gap between human and machine translation, 2016. arXiv:1609.08144. [23] X. Yin, J. Goudriaan, E. A. Lantinga, J. Vos, H. J. Spiertz, A flexible sigmoid function of determinate growth, Annals of botany 91 (2003) 361–371. [24] A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, V. Stoyanov, Unsupervised cross-lingual representation learning at scale, arXiv preprint arXiv:1911.02116 (2019). [25] J. Smith, H. Saint-Amand, M. Plamadă, P. Koehn, C. Callison-Burch, A. Lopez, Dirt cheap web-scale parallel text from the common crawl, in: Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2013, pp. 1374–1383. [26] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, V. Stoyanov, Roberta: A robustly optimized bert pretraining approach, arXiv preprint arXiv:1907.11692 (2019). [27] V. Sanh, L. Debut, J. Chaumond, T. Wolf, Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, arXiv preprint arXiv:1910.01108 (2019). [28] K. Puranik, A. Hande, R. Priyadharshini, T. Durairaj, A. Sampath, K. Thamburaj, B. R.
Chakravarthi, Attentive fine-tuning of transformers for translation of low-resourced languages @loresmt 2021, 2021.