The current surge in social media usage has resulted in the widespread availability of harmful and hateful content. Such inflammatory content identification in social media is a crucial NLP problem. Recent research has repeatedly demonstrated that context-level semantics matter more than word-level semantics for assessing the existence of hate content. This paper investigates many state-of-the-art transformer-based models for hate content detection in code-mixed datasets. We emphasize transformerbased models since they capture context-level semantics. In particular, we concentrate on Google-MuRIL, XLM-Roberta-base, and Indic-BERT. Additionally, we have experimented with an ensemble of the three mentioned models. Based on substantial empirical evidence, we observe that Google-MuRIL emerges as the top model with macro F1-scores of 0.708 and 0.445 for HASOC shared tasks 1 and 2, placing us 1 and 6ℎ on the overall leaderboard standings respectively.
The dificulty of constructing local languages in universal key inputs has led to a recent trend in code-mixed data as well. The several alternative interpretations of the Hinglish words in various contextual settings make automated categorization challenging in the case of code-mix Indic languages like Hinglish, where Hindi is expressed in romanized English. When we deal with conversational speaking in the Hinglish language, it also gets very challenging. When identifying hate speech, the conversation’s context is absolutely essential. Whether one agrees or disagrees with the previous comment or the prevailing philosophy in the discourse, it is possible to develop hatred for a certain target group, as shown in Figure 1. If the context of the parent tweet is taken into consideration, it is possible to determine whether a conversational thread contains hate or inflammatory material, even though it is not always obvious from a single comment or a reply to a comment that it does.
Earlier, in HASOC 2020[
Conversational hate speech detection in the codemix dataset is a challenging task. A majority
of work has been done using the various transformer-based pre-trained models for the first time
at HASOC 2021 [
Datasets[21] have been sampled from Twitter. To lessen the impact of bias, organisers have selected contentious stories on several subjects that are likely to feature hateful, ofensive, and profane posts. The controversial stories are as follows: • Temple-Mosque Controversy • Covid Controversy • Common Civil Code • Hinduphobia • Namaz on public place • Farmer Protest • Historical Hindu Muslim • Islamophobia • Russian-Ukrainian conflict etc.
Task 1’s major goal is the coarse-grained binary classification of conversational hate speech and ofensive language, primarily for Hinglish and German. HOF denotes a tweet, comment, or reply that promotes or involves hate speech or other rude or obscene languages, as opposed to a tweet, comment, or reply that is free of any harmful, vulgar, or hateful language. The dataset statistics for Hinglish and German for binary classification are shown in Table 1.
For task 2 (multi-class classification), the HOF class is classified into three sub-classes: Standalone Hate (SHOF), Contextual Hate (CHOF), and Non-Hate (NONE). It’s not always possible to tell from a single comment or reply to a comment whether or not a conversational thread contains hate-ofensive information. It is simple to identify by providing the parent tweet’s context. As seen in Figure 1, the response is positive, but only insofar as it expresses hatred for the person who posted the original tweet in the remark. It validates the venom that was written in the comment. It is therefore also hate speech. Table 2 contains the dataset statistics for the conversational code-mixed data for multi-class classification.
We flatten each interaction into a separate parent-comment-reply chain before feeding the data into our model. We concatenate the tweets and add a "[SEP]" token to each tweet, comment, and reply to help users distinguish one from the next. Every instance has a final label applied to it that corresponds to the last comment in the discussion chain. Except for the comma (,), full-stop (.), bar (|), and question mark (? ), we removed all punctuation from the tweets. By eliminating URLs, mentions, and new-line characters, we clean up the data. We replaced the emojis with their CLDR short names. For multi-class classification, the target class’s frequency is highly imbalanced. For our model to take into account this issue, we have assigned diferent weights to each class by dividing the frequency of each class by the size of the dataset and subtracting this result from one.
Three diferent transformer model types were used in our studies; they are explained below: • XLM-Roberta: A masked language model built on transformers and pre-trained on 100 languages. On numerous cross-lingual benchmarks, it produced cutting-edge performance.
Facebook AI published this model in 2019. • Indic-BERT: a pre-trained ALBERT model that has been trained on a sizable dataset of 12 Indic languages, including Assamese, Bengali, English, Gujarati, Hindi, Kannada, Malayalam, Marathi, Oriya, Punjabi, Tamil, and Telugu. There are roughly 9 billion tokens in the multilingual corpus. • Google MuRIL[22]: In 2020, Google released it. It is a multi-lingual BERT for Indic languages. MuRIL-BERT was pre-trained in 17 Indic languages, including English, Assamese, Bengali, Gujarati, Hindi, Kannada, Kashmiri, Malayalam, Marathi, Nepali, Oriya, Punjabi, Sanskrit, Sindhi, Tamil, Telugu, and Urdu.
Using binary cross-entropy loss, we improved BERT transformers and trained our linear classifier layer on top of them. Each BERT model had a dropout layer on top of which added a fully connected linear layer. The representation of the CLS token served as the input to this fully connected layer.
In this task, the target class’s frequency is highly imbalanced. To deal with this class-imbalance issue, we have used the weights for each class. We have assigned diferent weights to each class by dividing the frequency of each class by the size of the dataset and subtracting it from one.
The ensemble of the three models together has also been put to the test. For our purposes, we decided to combine the output of the three models using the majority voting process. There are two ways to perform a majority voting system: • Hard Voting Ensemble: In this instance, the model chooses the prediction class with the most votes out of all of the fine-tuned transformer models. • Soft Voting Ensemble: The model in this case sums the class probabilities from all of the ifne-tuned transformer models and selects the class with the highest sum of probabilities.
We utilised a batch size of 64 and a maximum sequence length of 512 for all models. We employed early halting on the validation loss with patience of 10 epochs to obtain the optimised learnable parameters. A 2e-5 initial learning rate Adam optimizer was employed.
We divided the training data into three groups for evaluation purposes: a train set, a validation set, and a test set, with a ratio of 80:10:10. We track the top model using validation loss at each epoch. Using the Google-MuRIL BERT, we are achieving the best results in binary code-mixed categorization. In the case of binary classification, we have found that the Google-MuRIL model’s macro F1-score is 5–6% higher than the XLM–Roberta–Base model. We utilised various random seeds to test our models and found that their performance was largely the same.
For the binary classification problem, Table 3 compares the results of Google-MuRIL, XLMRoberta-base, Indic-BERT, and an ensemble of these three models. Table 4 displays the classification outcomes for multi-class categorization.
In accordance with the findings from the validation data (in section 5.1), we adjusted the model’s hyperparameters for the entire training set of data. We turned in the five runs for tasks 1 and 2
that were shared. We may deduce from the leaderboard that Google-MuRIL is the best model for both shared tasks, as indicated in Table 5.
We have compared the outcomes for various transformer-based BERT architectures in this research. In both tasks, we found that Google-MuRIL outperforms all alternative transformerbased systems. It has also been seen that changing the random seed does not change the model performance. We have also observed that the performance of all three BERT models is better than the ensemble model. So, some of the actions will be to speculate on the reason behind them.
This research is funded by the Science and Engineering Research Board (SERB), Dept. of Science and Technology (DST), Govt. of India through Grant File No. SPR/2020/000495. (2020). [19] T. Caselli, V. Basile, J. Mitrović, M. Granitzer, Hatebert: Retraining bert for abusive language detection in english, arXiv preprint arXiv:2010.12472 (2020). [20] B. Chinagundi, M. Singh, T. Ghosal, P. S. Rana, G. S. Kohli, Classification of hate, ofensive and profane content from tweets using an ensemble of deep contextualized and domain specific representations (2021). [21] S. Modha, T. Mandl, P. Majumder, S. Satapara, T. Patel, H. Madhu, Overview of the HASOC Subtrack at FIRE 2022: Identification of Conversational Hate-Speech in HindiEnglish Code-Mixed and German Language , in: Working Notes of FIRE 2022 - Forum for Information Retrieval Evaluation, CEUR, 2022. [22] S. Khanuja, D. Bansal, S. Mehtani, S. Khosla, A. Dey, B. Gopalan, D. K. Margam, P. Aggarwal, R. T. Nagipogu, S. Dave, et al., Muril: Multilingual representations for indian languages, arXiv preprint arXiv:2103.10730 (2021).