The immense growth in social media forums does increase the spread of ofensive language. We detect and examine the challenges faced by automatic approaches for ofensive language detection in the TamilEnglish language. Among these dificulties are subtleties in code-mixed Tamil language, identifying what constitutes ofensive, and handling the imbalanced data under the low resource language. This paper presents our work in the shared task of HASOC-Dravidian-CodeMix-FIRE 2021, where we explore diferent machine learning algorithms, deep learning techniques, and transfer learning models. We also explore various feature extraction techniques and utilize ofensive features to perform this task. Our team SSN_NLP_MLRG has participated in task1 and classifies the code-mixed Tamil textual content into ofensive or not-ofensive. Our team best model is Multilingual BERT, and submission had a macro F1-score 0.84 of task1 of Tamil code-mixed language. Our team achieved the 3 rank on the final test results in task1 for the Tamil code-mixed language.
Social media is one of the platforms for the public to communicate with each other, share ideas,
express thought, and their emotions freely without considering others [
Code-mixing plays a critical role in a multilingual community. The code-mixed texts are
the mixture of native scripts and non-native scripts like text are in Tamil language but written
nEvelop-O
in roman script, and mixing of both the Tamil and English languages. The challenging part
is to train a system on monolingual data at diferent linguistic levels in the code-mixed text
because of the complexity of code-switching [
Ofensive language 2 such as threatening, harassment, violence, defrauding, sexual comments,
gender-specific comments, racial slurs, any content that could seriously ofend someone or
group. Based on their age, religion, political beliefs, marital, parental status, physical features,
national origin, and disability. Ofensive language 3 is the ofense of using curse language
in a way that could ofend a reasonable person in, near, or within hearing or view of public
forums or schools. More users have been experiencing online harassment. Depending on the
circumstances, this ofense is a punishable ofense by the Court. So the direct and indirect
ofensive content like sarcasm [
This paper presents a description of our Team SSN_NLP_MLRG submission runs to the shared
task of ofensive language identification of code-mixed text in Dravidian languages
(TamilEnglish and Malayalam-English). This task is a part of the Forum for Information Retrieval
Evaluation (FIRE 2021) workshop [
The goal of task1 is to identify and classify the social media comments are ofensive or not ofensive language in the Tamil-English code-mixed Language. We explore various approaches like machine learning techniques, neural networks, and pre-trained models to detect the offensive language in Tamil code-mixed language4. This paper contains the following sections. Section 2 has the related works. Section 3 has the experimented data and task description. Section 4 has the technique of our models. Section 5 has conducted research and presented its ifndings. Finally, Section 6 summarises our findings and suggests ways to improve our work. 1https://en.wikipedia.org/wiki/Tamil_language 2https://www.lawinsider.com/dictionary/ofensive-content 3https://www.primelawyers.com.au/criminal-law/public-order-ofences/ofensive-language/ 4https://github.com/kalaiwind/Dravidian-2021
The novel selective translation and transliteration approach for pre-processing and trained
the system by using ensemble XLM-RoBERTa to detect ofensive language identification in
Dravidian languages [
The researchers used the ALBERT model with the cross-lingual translation to detect hate
speech and ofensive language in English, Tamil, and German languages [
The overview shared task of HASOC: Hate Speech and Ofensive Content Identification in
Indo-European Languages has two sub-tasks for the three languages are English, German, and
Hindi (code-mixed) [
Most of the work in ofensive language identification from social media comments was done
in high-resource languages like English. We still face the problem of handling the dataset in
lowresource languages like the Dravidian language [
This section presents the description of Tamil code-mixed data, task, and data preparation and pre-processing techniques.
The organizers provided the HASOC 2021 Dravidian dataset for Tamil and Malayalam codemixed languages, which ofer comments from the social media forums. Our team SSN_NLP_MLRG participated in the task1 Tamil code-mixed dataset. The Tamil code-mixed HASOC 2021 dataset consists of 5880 posts for the train system and 654 posts for testing the model system. The task1 of Tamil shared task is a multi-class classification task and aims to classify the posts into three classes, namely Ofensive (OFF): the posts contain the curse, profane, ofense, threatening words. Not ofensive (NOT): The comments do not have ofense words. Not-Tamil: The comments do not intend in the Tamil language.
The Tamil train dataset contains 6534 comments. We have removed the duplication of posts from the training dataset. After removal, task1 of the Tamil language contains 4664 not ofensive posts, 1145 ofensive posts, and three not-Tamil posts. We have used 5812 posts to build the system. Table 1 shows the description for the Tamil code-mixed dataset. We separately collected the vocabulary of ofense content from various sources. First, we have to identify the languages that contain the maximum number of comments from the training dataset. In our case, the Tamil language has the maximum number in the shared task1 of the HASOC-Dravidian-CodeMix-FIRE 2021. We detected the other language and performed two actions. If the text is in the roman script of the Tamil language then, transliterates it into the Tamil language, and the text is in other languages, translate them into the Tamil language by using google API. Figure 1 shows the statistics of the training Tamil code-mixed dataset.
The data pre-processing is important in order to clean the comments from the unnecessary noisy content and transform it into a coherent form, which can be portable for Tamil code-mixed language. We used the NLTK libraries for data cleaning. We remove @ symbol with string and the hashtag symbol with string denoted as the user’s name because it does not have any expressions and afects the performance of the model. We removed the punctuation, numerals, symbols, and emojis. After that, we converted the upper case text into small case text. We replaced the misspelling ofense words by using the collected ofense data. We corrected the repeated letters and then translate the words by using google API. Finally, we replaced the * words into appropriate matched words presents in the collected vocabulary words.
This section presents the diferent approaches and models experimented with for the Tamil code-mixed data.
We experimented with traditional machine learning algorithms namely support vector machine classifier (SVM), Naive Bayes classifier (NB), random forest classifier (RF), and Extreme gradient boosting ensemble classifier (XGB), and used to predict the ofensive content in the given code-mixed posts. We used the scikit-learn library 5 implementation of these above-mentioned traditional classifiers. First, the data was pre-processed and extracted the Ngram, character level, word-level features by using Term frequency-inverse document frequency (TF-IDF) vectorization. We used sklearn CountVectorizer which helps to build vocabulary for known words and also tokenize the collected text documents. For the Naive Bayes classifier, we created a count vectorizer object to transform the training and validation data and we extracted the Ngram TF-IDF, character level TF-IDF, word-level TF-IDF features.
In FastText, the pre-trained vectors for 157 languages were trained on common crawl and Wikipedia. We used the FastText pre-trained word embedding vectors for the Tamil language namely Wikipedia Tamil vectors (wiki.ta.vec) and common crawl Tamil vectors (cc.ta.300.vec.gz). For the SVM Classifier, we used TF-IDF Vectorization to extract the features of Ngram TF-IDF vectors. For the Random Forest classifier, we used the count vectorizer and TF-IDF vectorizer for extracting the count vectors and word-level vectors respectively. For the XGBoost classifier, we extracted the features of count vectors, word level, and character level vectors.
5https://scikit-learn.org/stable/modules/classes.html
The ofensive language identification of code-mixed language by the following models, namely neural network (NN), Convolutional neural network (CNN), and recurrent neural network (RNN) with LSTM (Long short term memory) layer. The architecture of a neural network consists of 1 input layer, 1 hidden layer, and 1 output layer. The input is word-level embedding vectors which were extracted by using FastText pre-trained word embedding vectors. We have set the dense is 100 with the activation as relu in the input layer. The output layer has a dense of 1 with the activation as sigmoid and used the Adam optimizer and binary loss cross-entropy.
The architecture of a convolutional neural network consists of a 1D convolutional layer followed by a 1D max-pooling layer, then followed by the 3 output layers. The word-level representation is generated through a 1D convolutional layer with the activation as relu, dense as 50, and drop out of 0.25 in the output layer 1, sigmoid activation in the output layer 2, and used Adam optimizer and binary loss cross-entropy. We obtained the most prominent features by a 1-D maximum pooling layer.
The architecture of a recurrent neural network consists of a 1D convolutional layer followed by a 1D max-pooling layer, then an LSTM layer, and followed the 3 output layers. LSTM has the ability to process its sequences and retain all the information. LSTM has a dropout of 0.25 with activation as relu and sigmoid and optimizers as Adam and set the loss as binary cross-entropy.
Transfer learning plays a turning point in the computer science field and it’s led to major improvements and breakthroughs. For the past two decades, the introduction of pre-trained language models namely Universal language model fine-tuning for text classification (ULMFit) and Bidirectional Encoder Representation from Transformers (BERT) led to a revolution in the Natural Language Processing world. Most of the researchers were used BERT-based models and they also achieved state-of-the-art results in many tasks in NLP.
We used the MBERT (Multilingual BERT), ALBERT (A Lite BERT for self-supervised learning of language representations), DistilBERT (Distilled version of BERT) with the ktrain, and ULMFiT [20] with the Fastai to build the system to identify the ofensive content in the Tamil code-mixed language. We used the Average-SGD Weight-Dropped LSTM (AWD-LSTM) architecture model for the binary classification task to predict the ofensive content or Not-ofensive.
Fastai has functions for creating language and classification model data bunches, as well as setting the batch size to 32, the learning rate to 3e-02, 3e-03, 1e-03, 5e-04, and the epoch to 15, 3, 2, and 5 for training. For all the BERT-based models, we take 20 % of the data from the training data for the validation process. We analyzed the trained model to set the batch size to 6, 32 and learning rates as1e-5, 2e-5, 3e-5, and the epochs to 5, 6, and 10. With the pre-trained weights, we fine-tune the classifier. Finally, we have used the MBERT model to predict the ofensive content and got a weighted-average F1-score of 0.84 with the epochs 10 and the learning rate as 2e-5 for the task1 Tamil code-mixed language.
This section presents the analysis of diferent models and provides the details of results that were experimented with in the Tamil code-mixed data.
We experimented with the diferent models and compared the scores based on the evaluation metrics of weighted precision, weighted recall, and weighted average F1 score. Table 2 presents the validation results of diferent approaches of models. Based on the performance of the validation process, the NB model with the Character level vector, the NB model with the ngram TF-IDF vectors, XGB classifier with the character level vectors got an accuracy of 0.83 which is close to the MBERT.
MBERT model achieved an accuracy of 0.84 and Precision, Recall and an F1score of 0.83, 0.84, and 0.83 respectively which is compared with the performance of the other machine learning approaches, deep learning approaches, and pre-trained language models. The Precision, Recall, and F1score for the Not-ofensive comments are 0.89, 0.92, and 0.90 respectively. The Precision, Recall, and F1score for the ofensive comments are 0.58, 0.40, and 0.53 respectively. Table 3 presents the test results of diferent approaches of models. In the machine learning techniques, the NB model with the ngram features achieved the Precision, Recall, and an F1score of 0.83, 0.84, and 0.84 respectively. NB model with the character level vectors achieved the Precision, Recall, and an F1score of 0.83, 0.83, and 0.84 respectively. So, NB ngram and character level performed well which compared with other performance of the machine learning techniques. For BERT-based Models, MBERT performed well with the Precision, Recall, and an F1score of 0.84, 0.85, and 0.84 respectively. We observed that the performance of the MBERT model achieved good results compared to the other models.
This task is part of a shared competition organized in HASOC-Dravidian-CodeMix-FIRE 2021, where we participated as the SSN_NLP_MLRG team. For task1, we submitted the best performing model and the category-wise results are shown in Table 4. For the Tamil code-mixed task, we submitted the MBERT model which achieved a macro F1-score of 0.84 on the test set. We ranked 3rd in task1 shared of ofensive language identification in Tamil code-mixed language.
For further analysis, we used the confusion matrix to represent the performance of a classiifcation model on test data for that the true values are known. Figure 2 shows the confusion matrix of the MBERT model and Figure 3 shows the confusion matrix of the DistilBERT model.
For ALBERT Model, The Precision, Recall, and F1score for the Not-ofensive comments are 0.82, 1.00, and 0.90 respectively. But the F1 score of the ofensive content is 0.00 because the number of not-ofensive comments is higher in the overall Tamil code-mixed dataset. We observed the performance of the ULMFiT is the same as like ALBERT model.
From the confusion matrix, we observed that due to an unbalanced dataset many test cases were classified as Not-ofensive. For DistilBERT, The Precision, Recall, and F1score for the Notofensive comments and ofensive comments are 0.88, 0.91, 0.90, and 0.52, 0.54, 0.58 respectively. From table Table 4, we observed the F1score for the ofensive comments in the MBERT model is 0.55 which comparatively lowers the DistilBERT. So, MBERT performs well in the Not-Ofensive category and DistilBERT performs well in the Ofensive category. Figure 4 shows the confusion matrix of the ALBERT model and Figure 5 shows the confusion matrix of the ULMFiT model. From the confusion matrix in figure 4 and figure 5, we observed that due to an unbalanced dataset many test cases were classified as Not-ofensive.
This paper presents the submitted runs for the ofensive language identification for Dravidian Languages in Code-Mixed data in the Forum for Information Retrieval Evaluation (FIRE) 2021. The results show that the Not-ofensive class in each dataset receives the highest F1 scores, regardless of the model. This is due to the maximum number of the same as compared to the rest of the class. Comments that were not in the particular Tamil language of their dataset do not receive any classification in the test data. We experimented with diferent approaches such as machine learning techniques, deep learning approaches, and pre-trained BERT-based models. Based on the evaluation, MBERT performs well. Our team submission had a macro F1-score of 0.84 and achieved the 3rd rank on the final test data in task1 for the Tamil codemixed language. For future work, we will handle the imbalanced dataset and extend this work into other languages. Further, we will detect the sarcastic feature which helps to avoid misclassification. Dravidian languages, in: Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages, Association for Computational Linguistics, Kyiv, 2021, pp. 119–125. URL: https://aclanthology.org/2021.dravidianlangtech-1.15. [20] A. Kalaivani, D. Thenmozhi, SSN_NLP_MLRG@Dravidian-CodeMix-FIRE2020: Sentiment Code-Mixed Text Classification in Tamil and Malayalam using ULMFiT, in: FIRE (Working Notes), 2020, pp. 528–534.