As communities become multilingual, code-mixing becomes a prevalent phenomenon. Traditional models trained on monolingual data have proved to be inadequate for code-mixed data. This work was submitted for the Dravidian-CodeMix-HASOC 2021 task on ofensive language detection.The paper depicts a deep learning and Naive Bayes based approach towards solving the problem of understanding text written in non-native script and classifying them into ofensive and not-ofensive text. A Bidirectional Long Short Term Memory (BiLSTM) model is employed to solve this classification task. The model is tested for F1 score to obtain the degree to which the model correctly classifies the text as ofensive or non-ofensive, where the BiLSTM is observed to have performed well than the Naive Bayes approach
Recently, social media has become the most sought after mode of practicing one’s right to speech. The inadequacy of the rules imposed by social media platforms has led to the rise of ofensive speech that tends to destabilize communities. Additionally, most of the social media comments are made in non-native script. This code mixed text makes the task even more challenging as models trained on monolingual data fail on code-mixed data due to the complexity of code-switching at diferent linguistic levels in the text. It is dificult to transliterate Tamil to English efectively using the libraries indic-nlp. Most of the Tamil text did not match with its English equivalent because of the letters that are unique to Tamil and do not exist in the English scripture. Additionally, Tamil has many words that have diferent meanings in diferent contexts and predicting the ofensiveness of the comment was a challenging task.
In the current scenario people are working on newer models like BERT and their variants are Tamil Malayalam
Training
Validation currently employed and researched upon for ofensive language classification. Here we have used BiLSTM for detecting ofensive tweets.
Anti social comments in YouTube have been classified into ofensive and non ofensive in
[
Tweets are preprocessed to remove punctuation, repeated characters, non Arabic characters
before classifying them into ofensive or not. Dataset belonged to OSACT Shared task having
10,000 Arabic tweets [
KanCMD dataset with 7,671 post has code mixed text in Kannada posted as YouTube comments
[
Four diferent models have been used to detect ofensive language [
The given datasets are already split into 2 files for training and validation separately. The training file for Tamil language comprises of 1980/3999 ofensive labels and validation set contains 474/939 ofensive labels as shown in Table 1. Similarly Malayalam dataset also contains 2047/3999 and 478/950 ofensive labels in corresponding training and validation sets [ 9]. As we can see the two classes are almost equally balanced and hence do not require any kind of further preprocessing to prevent over fitting of the dominant class.
The proposed architecture uses both a machine learning classifier and a deep learning architecture for ofensive tweet classification. BiLSTM [ 10] and Naive Bayes classifier [ 11] are the two chosen approaches for the given shared task.
Our method employs a bidirectional LSTM to classify comments as ofensive and not ofensive. A bidirectional LSTM runs two LSTMs to read the text from front and back simultaneously. This marks the basis of diference between LSTM and BiLSTM. LSTM stores only the past while BiLSTM stores both the past and future. This makes a Bi-LSTM more eficient in understanding the mood and context of the comment when compared to other models.
→− ℎ = →(− +→− →.− ℎ − 1 +→− ) − ← ℎ = − (← +− ← − .← ℎ +1 +− ← ) ˆ = ( ℎ + ) = (→[−ℎ− ; ←ℎ ] + ) (1) (2) (3)
The Equations 1, 2, 3 [12] render the base for computation at a layer in a bidirectional lstm model. The score results from the forward LSTM and backward LSTM are concatenated as given in the third equation. Mathematically, at time t, an intermediate neuron receives one set of parameters from the previous time-step of the same LSTM layer and two sets of parameters from the previous LSTM layer, one from the forward and the other from the backward LSTM. The aggregated sum of results of all the hidden states is passed to the softmax activation function to compute the probability of each class.
The first layer of the model constitutes the embedding layer that converts the tokenized words into fixed-sized vectors. The Embeddings layer in Keras module was used to obtain a dense vector representation for words which learns the relational order of words and this adjusted embedding is fed to the BiLSTM layer [13].
This BiLSTM layer learns the context of sentences and learns the weightage of every word or a group of words in deciding whether it is ofensive or not. BiLSTM also considers the order of occurrence of the words which makes the model more eficient in determining the context and mood of the sentence. The proposed architecture with BiLSTM is shown in Figure 1. The loss function used in Binary cross entropy which is optimized using the Adam optimizer with a learning rate of 0.01. The model performance during training is evaluated using the accuracy metric.
The Naive Bayes model uses an approximation of the well known Bayes theorem. It takes the probability of each feature given an output to predict the probability of classifying the given input to a particular output. This algorithm is widely used in sentimental analysis with large datasets. The model architecture is shown in Figure 2. Bayes Theorem and Naive Bayes Theorem are represented as in Equation 4, 5.
( |12 * ... * ) = ( ) (12...| )
(12...) ( |12...) = ( ) (1| ) (2| )... (| ) (12...| ) (4) (5)
As we can see from above equations, Naive Bayes theorem assumes conditional independence of each feature present in the input. Therefore it fails to account for cases where a set of words determine the ofensiveness of a tweet rather than individually.
The redundant rows containing no meaningful tweets were removed as the first step of preprocessing the training dataset for BiLSTM network. The tweets were converted to lower case to avoid the same words being counted as separate words. Then the tweets were tokenized using the Keras Tokenizer, which splits the sentences into words and assigns a unique number to each word. The text to sequences function fits the encoding on the training data converting the string of words to that of numbers. The tokenized tweets were pre-padded to a uniform length of 100 tokens. Since, the number of words is not constant in the dataset, a safe upper bound of 100 words was chosen to fit the longest sentences in the dataset as well. Finally, the column containing the output was converted into categorical data. Since the training dataset was limited to non-native script and script without emoticons, no special libraries were used for transliteration or demotifying the text.
The same pre-processing steps as the training dataset are applied to the testing dataset. Additionally, the testing dataset contained comments in native script and raw data with emoticons. To deal with such data, the following libraries were used: • langdetect : To detect the language of the script based on individual characters in the text • indic-transliteration : To transliterate native script to roman script • re : To identify and remove emoticons from the comments
A single Embedding layer is used, whose output is fed to a BiLSTM layer with an output dimension of 150. The output layer has an activation function of softmax. Adam optimizer with a learning rate of 0.01 and binary_crossentropy loss function were used to train the model.
The train set is initially cleaned of punctuation and other English stop words with the help of stop word and punctuation corpus in nltk.The cleaned data is sent to another layer for transforming emojis found in the tweets to their corresponding emotions.Additionally a English language based Wordnet lemmatizer was used which didn’t add up to any further improvement (add this to modifications) , The transformed tweets are used to generate a bag of words count vector. The mutual info classify function was used to obtain the k strongest word features from the dataset. k is iteratively set to the numbers in the set { 100, 500, 1000} This is used to train the multinomial naive Bayes model.The test data and validation data follow the same preprocessing procedure.
Multinomial Naive Bayes was used directly from the sklearn library with the default parameters with Additive Laplace smoothing parameter as 1. Model F1-score BiLSTM 0.6628 BiLSTM 0.6979 NaiveBayes 0.804 NaiveBayes
Tamil Malayalam
Tamil Malayalam
Computation is done in Python using Google Colab notebook and its GPU was used to train the model 1. A general purpose RAM size of 8GB was allotted with a 2.3GHz Intel Xenon CPU.
1https://github.com/ask-1710/FIRE2021-OfensiveLanguageDetection.git
The two proposed models are submitted for evaluation in HASOC-Dravidian CodeMix Shared Task on Ofensive Language Detection in Tamil and Malayalam. Out of which the deep learning BiLSTM model was evaluated to be placed in rank 16 and rank 11 in Tamil Task and Malayalam task respectively based on the recall, precision and F1-score as shown in Tables 3 and 4. The BiLSTM model which performed well on Tamil language during training did not reflect the same when it was tested on unseen data. It has achieved only 55% accuracy in validation for Malayalam language but performed better than its Tamil counterpart in testing.
The Naive Bayes model was not ranked may be due to the conditional independence assumed by that approach on each feature of the input, and not fare well. Therefore it would have failed to account for cases where a set of words determine the ofensiveness of a tweet rather than individually.
To predict the ofensiveness of the text, one machine learning based Naive Bayes approach and a deep learning based BiLSTM were proposed in this work. The deep learning based model has not performed well in training when compared with Naive Bayes approach but have shown a noticeable performance during testing and placed in 16 and 11 ranks for Tamil and Malayalam languages respectively. Other classic algorithm Naive Bayes were also tested and not considered on the basis of accuracy values. From the results obtained it can be observed that that our proposed BiLSTM model too gives lower accuracy on unseen data. In the future we plan to use an ensemble model consisting of BiLSTM and BERT models to get better results. The performance can also be improved by employing augmentation techniques to handle the class imbalance problem. mixed social media tex, in: Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages, 2021, pp. 36–45. [7] K. Yasaswini, K. Puranik, A. Hande, R. Priyadharshini, S. Thavareesan, B. R. Chakravarthi, IIITT@ DravidianLangTech-EACL2021: Transfer Learning for Ofensive Language Detection in Dravidian Languages, in: Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages, 2021, pp. 187–194. [8] D. Sivalingam, S. Thavareesan, Of Tamil@ DravideanLangTech-EASL2021: Ofensive Language Identification in Tamil Text, in: Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages, 2021, pp. 346–351. [9] B. R. Chakravarthi, P. K. Kumaresan, R. Sakuntharaj, A. K. Madasamy, S. Thavareesan, P. B, S. Chinnaudayar Navaneethakrishnan, J. P. McCrae, T. Mandl, Overview of the HASOC-DravidianCodeMix Shared Task on Ofensive Language Detection in Tamil and Malayalam, in: Working Notes of FIRE 2021 - Forum for Information Retrieval Evaluation, CEUR, 2021. [10] L. Deng, Y. Liu, Deep learning in natural language processing, Springer, 2018. [11] I. Rish, et al., An empirical study of the naive Bayes classifier, in: IJCAI 2001 workshop on empirical methods in artificial intelligence, volume 3, 2001, pp. 41–46. [12] R. Socher, R. S. Mundra, CS 224D: Deep Learning for NLP1 (2016). [13] J. Xie, B. Chen, X. Gu, F. Liang, X. Xu, Self-attention-based BiLSTM model for short text ifne-grained sentiment classification, IEEE Access 7 (2019) 180558–180570.