We present the results and main findings of the HASOC-Ofensive Language Identification on code mixed Dravidian languages. The task featured two tasks. Task 1 is about ofensive language identification in Malayalam language where the comment were written in both native script and Latin script. Task 2 is about ofensive language identification in Tamil and Malayalam languages where the comments were written in Latin script (non-native script). For both the task, given a comment the participants should develop a system to classify the text into ofensive or not-ofensive. In total 96 participants participated and 12 participants submitted the papers. In this paper, we present the task, data, the results and discuss the system submission and methods used by participants.
considered as under-resourced [
The goal of this task is to identify ofensive language from a code-mixed dataset of comments/posts in Dravidian Languages (Malayalam-English and Tamil-English) collected from social media. The comment/post may contain more than one sentence but the average sentence length of the corpora is 1. Each comment/post is annotated with ofensive language label at the comment/post level. The task-1 dataset also has class imbalance problems depicting real-world scenarios. The participants were provided with development, training and test dataset.
Task1:
This is a message-level label classification task. Given a YouTube comment in code-mixed Malayalam, systems have to classify it into ofensive or not-ofensive.
Task2:
This is a message-level label classification task. Given a tweet or Youtube comments in Tanglish and Manglish (Tamil and Malayalam using written using Roman Characters), systems have to classify it into ofensive or not-ofensive.
3.1. Task 1 data
For the Task 1, we downloaded data from YouTube comments. The comments were downloaded from
movie trailers during 2019. All the comment from those movie trailers were downloaded using a
YouTube comment scrapper 1. We utilized these comments to make a dataset for ofensive language
identification classification dataset. The dataset contains all types of code-mixing such as mixing the
scripts of Malayalam script and Latin script, mixing at the word level, mixing at inter-sentential and
intra-sential [
The baseline system for Task 2 for Manglish and Tanglish data used Support Vector Machine (SVM) classifier [15] with Term Frequency - Inverse Document Frequency (TD-IDF) features. This system didn’t undergo any preprocessing and analyzed the data at the word level. The SVM classifier was trained using a linear kernel and a regularization parameter (C) value = 1. Comments are evenly distributed in both classes, and hence the baseline model didn’t consider any approach to make classifier a balanced one. The baseline model for Malayalam achieved the F-score of 0.68, precision score of 0.69 and the recall score of 0.68 and Tamil achieved the F-score of 0.89, precision and recall of 0.89.
All teams were allowed a total of 3 submission per Task. Participants had to submit prediction for Tasks in language of their choosing. A submission’s final score for each Task was computed as the highest score of the submission. The systems were evaluated on precision, recall and F1-score. This takes into account the varying degrees of importance of each class in the dataset. We used a classification report tool from Scikit learn2.
Precision =
Recall = + +
Precision ∗ Recall F-Score = 2 ∗ Precision + Recall (1) (2) (3)
We received a total of 12 submissions for Task 1 for Malayalam, 13 submission for Task 2 for Malayalam, and 12 submission for Task 2 for Tamil. The systems were evaluated based on F1 scores and a rank list was prepared. Table 1, Table 2 and Table 3 show the rank lists of Malayalam and Tamil tasks. We briefly describe below the methodologies used by the participating teams who submitted the paper.
• SivaSai@BITS [16]: The authors proposed a novel and flexible approach of selective translation and transliteration to be able to reap better results out of fine-tuning and ensemble multilingual transformer networks like XLM-RoBERTa and mBERT. • CENmates [19]: The participants used TF-IDF vectors along with character level n-grams as features to the proposed system for system development. They developed and evaluated four systems consisting of logistic regression, XGBoost, long short-term memory networks, and attention networks. They noted that simple TD-IDF approach with character level n-gram features using machine learning classifiers was producing good results almost the same as the results obtained with deep learning-based classifiers. 2https://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html
NIT-AI-NLP [20] YUN [21] Zyy1510 [22]
WLV-RIT [24]
• CUSATNLP [28]: The participants did extensive preprocessing and removed the hashtags, URLs, and emojis. They used the simple LSTM layers, a recurrent dropout (0.2). The system results were very low compared to other participants. • KBCNMUJAL [26]: The participants as well used the common preprocessing techniques to simplify the text messages in addition to that they also remove the special tags. They performed elaborate analysis with classical machine learning model with char n-gram and word n-gram as features. They also reported results for ensemble models. They ranked second for Malayalam and third for Tamil in Task 2. • Gauravarora [23]: Participants proposed pre-training ULMFiT on synthetically generated codemixed data, generated by modelling code-mixed data generation as a Markov process using Markov chains. Their model achieved 0.88 weighted F1-score for code-mixed Tamil-English
Zyy1510 [22]
CFILT [27] YUN [21] NIT-AI-NLP [20]
Ajees [25] • SSNCSE [18]: The participants used char n-gram, TFIDF and fine-tuned BERT in combination with machine learning models such as MLP, Random Forest and Naive Bayes. • NITP-AI-NLP [20]: The authors explored deep learning models such as attention-based Long Short Term Memory (LSTM), Convolution Neural Network (CNN), and machine learning models such as support vector machine, Logistic regression, Random forest, and Naive Bayes. They have hown that the use of character N-gram Term Frequency-Inverse Document Frequency (TF-IDF) features plays a promising role in identifying ofensive social media posts. • YUN [21]: The authors proposed an ensemble model which makes full use of the information of rich sequential patterns. More precisely, the proposed model contains a self-attention based on the BiLSTM and the sub-word representation learning. Experimental results of their model on the Malayalam-English of Task 1, Tamil-English and Malayalam-English of Task 2 have achieved the F1 values of 0.93, 0.85 and 0.67, respectively, and ranked 3rd, 5th, 9th, respectively. • Zyy1510 [22]: ensemble model combines with diferent models to improve the F-1 value of the framework. The ensemble model is a combination of a BiLSTM (Bidirectional LSTM), an LSTM+Convolution, and a CNN (Convolution Neural Network) model. The proposed model have achieved an F-1 of 0.93 (ranked 3 rd ) in Malayalam-English of task1, and F-1 of 0.87 (ranked 3 rd ) and 0.67 (ranked 9 th ) in Tamil-English and Malayalam-English of task2, respectively • Ajees [25]: Mainly three types of machine learning models were experimented using two types of word embedding techniques. The first architecture was a simple MLP classifier using two hidden layers. The second one was a combination of CNN-BiLSTM network with four convolutional layers. And the third one was a BiLSTM stack with two hidden layers. In order to represent the individual words in the comments, a simple CountVectorizer as well as BERT was used. CountVectorizer is used to convert text data to a vector of token counts. It also provides the pre-processing of text data before generating the vector representation. This capability makes it highly flexible towards feature representation for text processing. On the other hand, BERT is a pre-trained NLP model that provides contextualized word embeddings. Static word embedding techniques provide the same vector for polysemous words without proper consideration of their context. Whereas the dynamic embedding techniques like ELMo and BERT can consider the context of words before generating their embeddings. BERT word representations take the entire input sentence into the equation for calculating the word embeddings. • WLV-RIT [24]: applying cross-lingual contextual word embeddings and transfer learning to make predictions to Malayalam data. They further improve the results using various fine tuning strategies. • IIITG-ADBU [17] paper presents the results obtained by our SVM and XLM-RoBERTa based classifiers in the shared task. • CFILT[27] presents an ensemble of multilingual BERT models for this task and devise a novel training strategy involving data augmentation using random transliteration. They achieve an F-score of 0.95 for hate speech and ofensive content detection on the Malayalam code-mixed YouTube comments test data in task 1. In task 2, they achieve F-scores of 0.86 and 0.72 respectively for hate speech and ofensive content detection on Tamil and Malayalam code-mixed Twitter test data.
In Malayalam Task 1, teams SivaSai@BITS and IIITG-ADBU shared the first position with an F1-score of 0.95. These two systems achieved precision and recall score of 0.95. Teams from CFILT-IITBOMBAY and SSNCSE-NLP achieved the second position with an F-score of 0.94. The top four teams attained Fscore higher than 0.90. The diference between the evaluation scores of top teams is minuscule. Table 1 presents the results of the Malayalam Task 1. Teams placed in the first and second position utilized transformer-based model for classification of YouTube comments into OFF and NOT. Transliteration of Romanized text into the native script is also found to be efective in this method. Another important fact visible from the results is the Support Vector Machine classifiers with TF-IDF features also reach top positions. Other systems submitted to the task use deep learning models using Bidirectional LSTM, LSTM, CNN and ULMFiT.
In Malayalam Task 2, CENmates reached the first position with an F-score of 0.78. Teams SivaSai@BITS and KBCNMUJAL bagged the second place, and their F-score was 0.77. The scores of the top five teams are close. Team CENmates used TF-IDF features with character n-gram as features for classification using machine learning algorithms. SivaSai@BITS used the same approach followed for Malayalam Task 1 for this task also. Team KBCNMUJAL used character and word n-grams features for with machine learning classifiers. Other teams used transformer-based models and Deep Learning-based models. Table 2 presents the result of Malayalam Task 2.
In Tamil Task 2, team SivaSai@BITS placed in the first position with an F-score of 0.90. They used the transformer-based model for this task also. Team SSNCSE-NLP grabbed the second position, and they scored an F-score of 0.88. They used TF-IDF with character n-gram features for classification. Gauravarora, who also came second position followed a pre-trained ULMFiT for the classification. Three teams reached third position with an F-score of 0.87. These three teams used entirely diferent features and classifiers for the prediction task. Team KBCNMUJAL uses character n-gram and word ngram features for representing text with ensemble models for classification. Team IIITG-ADBU used the same model used for other tasks for this task also. Team Zyy1510 used an ensemble of BiLSTM, LSTM+Convolution and a Convolution for the classification of social media texts into OFF and NOT. Table 3 presents the results of Tamil Task 2.
When we analyse the models submitted to the Tasks, most of them used either transformer-based models or conventional machine learning classifier with TF-IDF features. The performance of the deep learning models such as Bidirectional LSTM, LSTM, and CNN was not up to the mark. Transformerbased models used BERT for generating the embedding.
Figure 1 shows the box-plots of the performance of the systems for Malayalam Task 1, Malayalam Task 2 and Tamil Task 2, respectively. From the Figures, it is clear that Median of the F-score of all the submissions is close to the top score in Malayalam Task 1 and Malayalam Task 2.
We presented the overview and results of the shared task on HASOC-Ofensive language detection in Dravidian languages. A wide range of systems were evaluated on two task in two languages relying on a thoroughly annotated dataset. The task setup provided an opportunity to test models on codemixed Dravidian languages with mixed scripts. NLP for code-mixed texts is very challenging and under-resourced setting also make it even more challenging. We found that many systems were based on transformers and pre-trained embedding based systems. We that this shared task makes a lasting contribution to the research in Dravidian languages.
This publication is the outcome of the research supported in part by a research grant from Science Foundation Ireland (SFI) under Grant Number SFI/12/RC/2289_P2 (Insight_2), co-funded by the European Regional Development Fund as well as by the EU H2020 programme under grant agreements 825182 (Prêt-à-LLOD), and Irish Research Council grant IRCLA/2017/129 (CARDAMOM-Comparative Deep Models of Language for Minority and Historical Languages).
We would like to thank Mrs.Sharmila Devi V, PhD scholar, Karpagam College of Engineering, Coimbatore for sharing the task 2 Tamil code-mixed ofensive dataset. guage Technologies for Under-resourced languages (SLTU) and Collaboration and Computing for Under-Resourced Languages (CCURL), European Language Resources association, Marseille, France, 2020, pp. 177–184. URL: https://www.aclweb.org/anthology/2020.sltu-1.25. [13] B. R. Chakravarthi, V. Muralidaran, R. Priyadharshini, J. P. McCrae, Corpus creation for sentiment analysis in code-mixed Tamil-English text, in: Proceedings of the 1st Joint Workshop on Spoken Language Technologies for Under-resourced languages (SLTU) and Collaboration and Computing for Under-Resourced Languages (CCURL), European Language Resources association, Marseille, France, 2020, pp. 202–210. URL: https://www.aclweb.org/anthology/2020.sltu-1. 28. [14] A. Hande, R. Priyadharshini, B. R. Chakravarthi, KanCMD: Kannada codemixed dataset for sentiment analysis and ofensive language detection, in: Proceedings of the Third Workshop on Computational Modeling of People’s Opinions, Personality, and Emotions in Social Media, Barcelona, Spain, 2020. [15] K. Soman, R. Loganathan, V. Ajay, Machine learning with SVM and other kernel methods, PHI
Learning Pvt. Ltd., 2009. [16] S. Sai, Y. Sharma, Siva@HASOC-Dravidian-CodeMix-FIRE-2020: Multilingual Ofensive Speech
Detection in Code-mixed and Romanized Text, in: FIRE (Working Notes), 2020. [17] A. Baruah, K. A. Das, F. A. Barbhuiya, K. Dey, IIITG-ADBU@HASOC-Dravidian-CodeMixFIRE2020: Ofensive Content Detection in Code-Mixed Dravidian Text, in: FIRE (Working Notes), 2020. [18] N. N. Balaji, B. Bharathi, SSNCSE-NLP@HASOC-Dravidian-CodeMix- FIRE2020: Ofensive Language Identification on Multilingual Code Mixing Text, in: FIRE (Working Notes), 2020. [19] V. P V, P. Ramanan, R. Devi G, CENMates@HASOC-Dravidian-CodeMix-FIRE2020: Ofensive Language Identification on Code-mixed Social Media Comments, in: FIRE (Working Notes), 2020. [20] S. Kumar, Abhinav adn Saumya, J. P. Singh, NITP-AI-NLP@HASOC-Dravidian-CodeMixFIRE2020: A Machine Learning Approach to Identify Ofensive Languages from Dravidian CodeMixed Text, in: FIRE (Working Notes), 2020. [21] K. Dong, YUN@HASOC-Dravidian-CodeMix-FIRE2020: A Multi-component Sentiment Analysis Model for Ofensive Language Identification, in: FIRE (Working Notes), 2020. [22] Y. Zhu, X. Zhou, Zyy1510@HASOC-Dravidian-CodeMix-FIRE2020: An Ensemble Model for
Ofensive Language Identification, in: FIRE (Working Notes), 2020. [23] G. Arora, Gauravarora@HASOC-Dravidian-CodeMix- FIRE2020: Pre-training ULMFiT on Synthetically Generated Code-Mixed Data for Hate Speech Detection, in: FIRE (Working Notes), 2020. [24] T. Ranasinghe, M. Zampieri, WLV-RIT @ HASOC 2020: Ofensive Language Identification in
Code-switched Texts, in: FIRE (Working Notes), 2020. [25] A. A P, Ajees@HASOC-Dravidian-CodeMix-FIRE2020, in: FIRE (Working Notes), 2020. [26] V. Pathak, M. Joshi, P. Joshi, M. Mundada, T. Joshi, KBCNMUJAL@HASOC-Dravidian-CodeMixFIRE2020: Using Machine Learning for Detection of Hate Speech and Ofensive Codemix Social Media text, in: FIRE (Working Notes), 2020. [27] P. Singh, P. Bhattacharyya, CFILT IIT Bombay@HASOC-Dravidian-CodeMix FIRE 2020: Assisting ensemble of transformers with random transliteration, in: FIRE (Working Notes), 2020. [28] S. Renjit, CUSAT-NLP@HASOC-Dravidian-CodeMix-FIRE2020: Identifying Ofensive Language from Manglish Tweets, in: FIRE (Working Notes), 2020.