Social media such as facebook twitter is flooded with user generated content [1,2,3,4]. In particular, hate speech on social media is on the rise at a rapid pace 1 posing a significant threat to the sustainable society. Social media YouTube defines hate speech as "any speech that involves race, age, sexual orientation, disability, religion, and racism to promote hate or violence among groups" 2 . Through these social platforms, the hate speech is reaching to concerned person even in their bedroom and last forever [5]. The hate speech has a terrible impact on users' mental status, resulting in depression, sleeplessness, and even suicide. It is challenging for the authorities to prove someone guilty due to their anonymous identity and across border laws since some countries have the freedom of expression to write. In contrast, others adopt a very stringent policy for the same message [6].
FIRE 2020: Forum for Information Retrieval Evaluation, December 16-20, 2020, Hyderabad, India email: abhinavanand05@gmail.com (A. Kumar); sunil.saumya@iiitdwd.ac.in (S. Saumya); jps@nitp.ac.in (J.P. Singh) orcid:
The manual identification of hate speech is almost impossible and needs to be thoroughly investigated automatically. A considerable amount of research work on English hate speech has been published. Davidson et al. [7] extracted N-gram TF-IDF features from tweets and applied logistic regression to classify each tweet into three classes hate, offensive, and neither. Kumari et al. [8] presented a model to identify cyberbullying instances using features optimization with genetic algorithm. Similarly, Agarwal and Sureka [9] extracted linguistic, semantic, and sentimental features and learned an ensemble classifier to detect racist contents. Kapil et al. [6] proposed LSTM and CNN based model to identify the hate speech in social media posts whereas, Badjatiya et al. [10] learned semantic word embedding to classify each tweet as racist, sexist, or neither. Kumari and Singh [11] presented a deep learning model to detect hate speech for English text. The code-mixed and script-mixed sentences are among the major challenges for machine learning models due to the unavailability of a sufficient dataset. A code mixed dataset on Tamil with English code-mixed Tanglish [12] and Malayalam with English code-mixed Manglish [13] were recently proposed for sentiment analysis task.
The purpose of this study is to recognize the hate speech in Indian languages like code-mixed Tamil-English, code-mixed Malayalam-English, and script-mixed Malayalam-English languages. The dataset used in the study belongs to HASOC-Dravidian-CodeMix-FIRE2020 challenge [14]. The data was gathered from YouTube and Twitter with a target for two sets of tasks. Task 1 asks to develop a classification system to differentiate script-mixed Malayalam comments into offensive and non-offensive. Task 2 requires to build a classifier to differentiate Tanglish and Manglish (Tamil and Malayalam have written using Roman Characters) into offensive and not-offensive classes. The current paper explored several deep learning and machine learning models to identify offensive posts from the code-mixed and script-mixed posts. For deep learning, we utilized attention-based Bi-LSTM-CNN, BERT, and DNN models. In contrast, for conventional machine learning, Support vector machine, Logistic regression, Random forest, and Naive Bayes classifiers are used.
The rest of the article is organized as follows; The proposed methodology is explained in Section 2. The experiment setting and obtained results are discussed in Section 3. Finally, we conclude the paper in Section 4.
The detail description of the submitted model in the FIRE-2020 workshop is listed in section 2.1, whereas the details of extensive experiments with different character N-gram TF-IDF features with the different classifiers are listed in section 2.2. The detailed statistic of the datasets used in this study is listed in Table 1. While pre-processing of the texts, we kept & and @ in our dataset by translating it into 'and' and 'at' respectively and removed other special characters. We also removed single letter word, punctuation, and replaced all numeric letters into their corresponding English word (e.g., 1-one, 9-nine). Finally, all texts are converted into the lowercase.
A hybrid attention-based Bi-LSTM and CNN network is used in the case of Tamil and Malayalam codemixed text. The overall model diagram of the hybrid attention-based Bi-LSTM and CNN network can be seen from Figure 1. We used character embedding for the CNN network, whereas word embedding is used in the attention-based Bi-LSTM network. For character embedding, one-hot encoding vectors are used. For word embedding, we created our FastText3 word embedding by utilizing the language- To process word-embedding, two Bi-LSTM layers having 512 and 256 output dimension space are used, followed by an attention layer. Finally, the output of attention-based Bi-LSTM and CNN layer is concatenated and passes through a softmax layer to predict offensive and not-offensive text. The detailed working of the CNN and attention-based Bi-LSTM network can be seen in [15,16,17,18]. The performance of deep neural networks is susceptible to hyper-parameters. Therefore, we performed extensive experiments by varying the learning rate, batch size, optimizer, epochs, loss function, and activation function. The proposed system worked best with the learning rate of 0.001, batch size of 32, Adam as an optimizer, epochs = 100, binary cross-entropy as a loss function, and ReLU activation in the internal layers of the network whereas softmax activation function at the output layer.
In the case of Malayalam script-mixed text, a fine-tuned pre-trained BERT 4 model is used to classify the text into offensive and not-offensive classes. We fixed 30-words for the text to input in the model and used a batch size of 32 and a learning rate of 2𝑒 −5 to fine-tune the pre-trained bert-basemultilingual-uncased BERT model. The detailed description of the BERT model can be seen in [19].
Along with the submitted model in the FIRE-2020 workshop, we also explored the uses of different word and character N-gram TF-IDF features with the conventional machine learning classifiers and deep neural network. We experimented by extracting different combinations of 1-gram, 2-gram, 3gram, 4-gram, 5-gram, and 6-gram word features and character features from the text and applied Support Vector Machine (SVM), Logistic Regression (LR), Naive Bayes (NB), Random Forest (RF), and Dense Neural Network (DNN). For the DNN network, we used four layers of a fully connected network with 1024, 256, 128, and 2-neurons. We used a dropout rate of 0.3, ReLU, and softmax as the activation function, batch size of 32, binary cross-entropy as the loss function, and Adam as the optimizer. Various word-level N-gram TF-IDF features with the said machine learning models did not perform well compared to the submitted model. Therefore we are not reporting the results of these models. On the other hand, when various machine learning models learned with different character N-gram TF-IDF features like 1-gram, 2-gram, 3-gram, 4-gram, 5-gram, and 6-gram, the performance was remarkable. In the case of code-mixed Tamil and Malayalam text, the top 10,000 characters Ngram (1-gram to 6-gram) TF-IDF features performed best. In the case of Malayalam script-mixed text, the top 20,000 characters N-gram (1-gram to 6-gram) TF-IDF features performed best. The detailed results of each of the classifiers are listed in section 3.
The results of the attention-based Bi-LSTM-CNN and BERT models for Tamil code-mixed, Malayalam code-mixed, and Malayalam script-mixed text are listed in Table 2. In the case of Tamil code-mixed Next, experiments were performed for machine learning models using character N-gram (1 to 6gram) TF-IDF features. The results for the Support Vector Machine (SVM), Logistic Regression (LR), Naive Bayes (NB), Random Forest (RB), and Dense Neural Network (DNN) are listed in Table 3. In the case of Tamil code-mixed text, the NB classifier performed best and achieved a precision, recall,
Among all the submitted model in the FIRE-2020 workshop for this task, the best model achieved precision, recall, and 𝐹 1 -score of 0.90 for code-mixed text, precision, recall, and 𝐹 1 -score of 0.78 for Malayalam code-mixed text and precision, recall, and 𝐹 1 -score of 0.95 for Malayalam script-mixed text. Compared to the best-submitted model, our proposed system equally performed well in Tamil and Malayalam's code-mixed text. In the case of Malayalam script-mixed text, our proposed system performed slightly better with the precision of 0.96.
The identification of hate speech from the code-mixed and script-mixed Dravidian sentences have enormous challenges. This work explored the usability of several deep learning and machine learningbased models for classifying offensive and not-offensive sentences. The character N-gram TF-IDF based Naive Bayes classifier performed best with the weighted precision, recall, and 𝐹 1 -score of 0.90 for Tamil code-mixed text. The Logistic regression classifier with character N-gram TF-IDF features performed best with the weighted precision, recall, and 𝐹 1 -score of 0.78 for Malayalam code-mixed text. The Dense Neural Network with character N-gram TF-IDF features performed best with the weighted precision of 0.96, recall of 0.95, and 𝐹 1 -score of 0.95 for Malayalam script-mixed text.