The concept of content moderation is as old as the online social media itself, the goal is to prevent any hate speech, comments etc to happen on the platform so as to keep the online social environment friendly and sane. With an exponentially increasing number of people on social media content moderation is a dificult task as such in the modern era we make use of specialised tools such as AI and NLP. In non-native English spoken countries, social media texts are mostly in code mixed form. This paper discusses the work put by SSNCSE_NLP in HASOC ofensive language identification on multilingual codemixed text tasks of FIRE 2021. In this paper we have put a detailed comparison on the performance of several RNN's based models with transformers based BERT architecture by varying the essential hyperparameters when training on a smaller dataset for tasks like sentiment analysis. We achieved an F1 score of 72.47% in task 1 and 69.2% ,61.5% in task2 Tamil and Malayalam respectively on the test set from our best evaluated model.
Every small or big brand wants to put their product into as many hands as possible and easily
accessible in their own native languages this, in combination with the reach of internet has
resulted into massive expanse of rich diverse user groups, online content moderation in those
native languages along with the mixed languages that the user group speaks is hence necessary.
As codemixed languages consist of bilingual, trilingual or more languages in combination with
symbols and emojis it’s dificult to train eficient models. With recent developments in sequence
processing models and Transformer[
The paper is organized as follows: The dataset descriptions are given in Section2.1 Section 2.3 details the experimental setup and various features used for this task. Section 3 provides a subjective analysis and comparison of the performance of various models on the development and test data. Finally, Section 4 concludes the paper.
The primary goal of this shared task is to detect ofensive language of the code-mixed dataset of
comments/posts in Dravidian Languages (Malayalam-English and Tamil-English) collected from
social media [
The datasets consisted of Dravidian languages Tamil and Malayalam codemixed with English words, symbols and emojis. The dataset was parsed to generate word level tokens then generate characters to separate out non-UTF-8 charset also emojis were removed from the dataset obtained from the charset, later the word level tokens were directly parsed into LSTM based networks while the clean text were parsed separately to generate BERT-tokens for training the transformer architecture this was done so as to create proper tokens for Tamil in English(Tanglish) and Malayalam in English(Manglish) datasets provided in task-2 while pre-trained embeddings from Indic BERT[11] was used for task-1.
For natural language modelling several LSTM architectures including ELMo[
ELMo is a LSTM based architecture that leverages bi-directionality[11] of natural language models by using two separate LSTM layers running left-to-right and right-to-left in bidirectional wrapper and shallow concatenating the outputs. Since there weren’t any multilingual models on ELMo for Indian languages we ended up training it from scratch and achieved an accuracy of 82.7% on validation set.
ELMo utilizes LSTM’s it shares same hyper-parameters as them namely : 1. Number of units : number of LSTM units or output dimension. 2. Dropout : dropout rate(0-1) for outputs. 3. Recurrent dropout : dropout rate(0-1) for recurrent output . ELMo architecture in contrast to the fairly small dataset could be trained from scratch for better accuracy.
BERT is a transformer[
BERT models are implemented in pytorch and utilises hugging face transformers API to create and train models, while the ELMo architecture was implemented using tensorflow and was trained from scratch all the notebooks associated with training of models are available in the link 1
The performance of the proposed approach using BERT model is given in Table1. From 1, it has been noted that fine-tuning multilingual model Indic BERT gave a score of 81% on validation set.
The performance of the proposed system using ELMO model is given in Table 2.
In Table 2 BiLSTMu refers to BiLSTM with side-by-side stacked uni-directional(left-to-right) LSTM. BiLSTMd refers to BiLSTM with separate left-to-right and right-to-left LSTM stacked, outputs shallow concatenated and fed to fully connected layers for classification.
1https://github.com/Abhishek-krg/Multilingual-codemixed-language-classification
From Table 2, it has been noted that BiLSTMd model with 32 units achieves highest accuracy of 82. 7%. Considering the performances of multilingual LM, we have experimented XLM-Roberta. The results are tabulated in Table 3.
The performance of the proposed system using the test data is given in Table 4.
In this paper, we proposed ofensive language identification using Dravidian code-mixed text using ELMO and BERT models. From the performance metrics above it is clear that BERT despite being a far better architecture couldn’t achieve expected results while the BiLSTMd architecture gave better results on HASOC dataset. This could be due to the reason that BERT is a very dense model and requires huge-datasets to train on while LSTM based RNN architectures on the other hand can achieve better results on simpler classification tasks. P. B, S. Chinnaudayar Navaneethakrishnan, J. P. McCrae, T. Mandl, Overview of the HASOC-DravidianCodeMix Shared Task on Ofensive Language Detection in Tamil and Malayalam, in: Working Notes of FIRE 2021 - Forum for Information Retrieval Evaluation, CEUR, 2021. [11] D. Kakwani, A. Kunchukuttan, S. Golla, G. N.C., A. Bhattacharyya, M. M. Khapra, P. Kumar, IndicNLPSuite: Monolingual Corpora, Evaluation Benchmarks and Pre-trained Multilingual Language Models for Indian Languages, in: Findings of EMNLP, 2020. [12] M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, L. Zettlemoyer, Deep contextualized word representations, CoRR abs/1802.05365 (2018). URL: http://arxiv.org/ abs/1802.05365. arXiv:1802.05365.