Hate speech is considered to be one of the major issues currently plaguing online social media. Repeated and repetitive exposure to hate speech has been shown to create physiological efects on the target users. Thus, hate speech, in all its forms, should be addressed on these platforms in order to maintain good health. In this paper, we explored several Transformer based machine learning models for the detection of hate speech and ofensive content in English and Indo-Aryan languages at FIRE 2021. We explore several models such as mBERT, XLMR-large, XLMR-base by team name "Super Mario". Our models came 2 position in Code-Mixed Data set (Macro F1: 0.7107), 2 position in Hindi two-class classification (Macro F1: 0.7797), 4ℎ in English four-class category (Macro F1: 0.8006) and 12ℎ in English two-class category (Macro F1: 0.6447). We have made our code public 1.
Online Social media platforms such as Twitter, Facebook have connected billions of people and
allowed them to publish their ideas and opinions instantly. The problem arises when the bad
actors(users) share contents to spread propaganda, fake news, and hate speech etc[
To reduce the harmful content (such as ofensive/hate speech) from these platforms, they
employ moderators [
There is a lot of state-of-the-art hate speech detection research content present in the market,
mostly in English languages [
Despite being the third most spoken language, Hindi is always being considered a low resource language because of its mostly typological representation. Marathi is also kind of very low resource language because there is rarely some work present to identify Hate/ Ofensive content. Finally, Code-mixed data is also following a current trend because of complexity in writing local languages in universal key inputs.
Earlier, in HASOC 20194 three datasets have been launched to identify Hate and ofensive
content in English, German and Hindi languages, and in HASOC 20205 another dataset has been
launched, aiming to identify ofensive posts in the code-mixed dataset. Extending the previous
work, this time HASOC[
In this paper, we have investigated several Transformer based models for our classification task, which has already been seen to be outperforming the existing baselines and standing as a state-of-the-art model. We perform pre-processing, data sampling, hyper-parameter tuning etc., to build the model. Our models are standing in the 2 position in Code-Mixed Data set, 2 position in Hindi two-class classification, 4ℎ in English four-class classification and 12ℎ in English two-class classification. The rest of the paper is organised as follows: Related literature for Hate speech and ofensive language detection Section 2. We have discussed the Dataset Description in Section 3; In Section 4, we have presented the System Description. Finally, we have evaluated the experimental setup in Section 5 and the Conclusion in Section 6.
The problem of hate/ofensive speech has been studied for a long time in the research community.
People were continuously trying to improve the models in order to identify hateful/ofensive
content more precisely. One of the earliest works that tried to detect hate speech by using
lexicon-based features [
In 2018, Pitsilis et al. [15], tried deep learning models such as recurrent neural networks (RNNs) to identify the ofensive language in English and found that it was quite efective in this task. RNN’s remember the output of each step the model conducts. This approach can capture linguistic context within a text, which is critical to detection. In contrast, RNN’s have been projected to work well with language models, other neural network models, such as CNN. LSTM has had notable success in detecting hate/ofensive speech [16, 17].
Although the research on hate/ofensive speech detection has been growing rapidly, one of
the current issues is that most of the datasets are available in the English language only. Thus,
hate/ofensive speech in other languages are not detected properly and this could be harmful.
This is also a problem for companies like Facebook, which can only detect hate speech in certain
languages (English, Spanish, and Mandarin) [
Recently, Transformer based [24] language models such as BERT, m-BERT, XLM-RoBERTa [25]
are becoming quite popular in several downstream tasks, such as classification, spam detection
etc. Previously, it has been already seen these Transformer based models have been
outperformed several deep learning models [
The shared tasks present in this competition are divided into two parts. The datasets have been sampled from Twitter. Subtask-1 ofers in English, Hindi with two problems, and Marathi with one problem. The subtasks-2 dataset contains English, Hindi and code-mixed Hindi tweets. Details about the problem statements have been discussed below:
The primary focus of Subtask 1A on Hate speech and Ofensive language identification, mainly for English, Hindi and Marathi [26], is coarse-grained binary classification. In Table 1 we have presented the dataset statistics on English and Hindi for binary classification.
This subtask is a fine-grained classification of English and Hindi. Mostly Hate-speech and ofensive posts from Subtask 1A are further classified as (HATE) Hate speech, (OFFN) Ofensive, (PRFN) Profane, (NONE) Non-Hate. In Table 2 we have presented the dataset statistics on English for Four-class classification (We participated for English language only.).
A conversational thread can also contain hate and ofensive content, which not always can be identified from a single comment or reply to a comment. In this type of situation, context is important to identify the hate or ofensive content.
According to Figure 1 the parent tweet is expressing hate and profanity towards Muslim countries regarding the controversy happening in Israel at the time. The two comments on the tweet have written "Amine", which means "truthfully" in Persian, which supports the hate but with the context of the parent. This sub-task focused on the binary classification of such conversational tweets with tree-structured data into (NOT) Non-Hate-Ofensive and (HOF) Hate and Ofensive. In Table 1 we have presented the dataset statistics of the code-mixed data for binary classification as well.
While manually going through the data, we found the dataset contains lots of special characters, emoji’s, blank spaces, links etc. Mostly custom functions have been used in pre-processing the (HATE) Hate speech (OFFN) Ofensive (PRFN) Profane (NONE) Non-Hate
Total 683 622 1196 1342 3843 224 195 379 483 1281 ifles, but some libraries were helpful, like "emoji", "nltk" as a baseline. Performed pre-processing steps are: • We have replaced all the tagged user names to @user. • We have removed all non-alphanumeric characters except full stop and punctuation (| , ?) in Hindi and Marathi. We have kept all the stop words because by that way machine will be able to identify the sequence of characters properly. • We have removed emojis, flags and emotions. • We have removed all the URLs.
• We have kept the hashtags because the hashtags contains some contextual information
We have presented our proposed models for Ofensive language detection and Hate speech detection in English, Marathi, Hindi and Code-mixed posts. The overall pipeline of the methodology has been represented in Figure 2. The baseline we have used is Transformer based pre-trained architecture of BERT [25]. More intuitively, we have used a couple of various versions of BERT, more specifically mBERT [ 25] and XLM-Roberta [27]. The beauty of XLM-Roberta is it is trained in an unsupervised manner on the multilingual corpus. XLM-Roberta has achieved state-of-the-art results in most language modelling tasks.
Most of our task was binary classification problem based on respective embedding. We finetuned BERT Transformer and classifier layer on top and used binary target labels for individual classes. We have used this procedure with dehatebert-mono-english [28] and XLM-Roberta [27] for English dataset. We have used multilingual BERT (mBERT) and XLM-Roberta for Marathi, Hindi and Code-Mixed classification. Binary cross-entropy loss can be computed for previously mentioned classification task can be mathematically formulated as:
′=2 = − ∑︁ () = − 1(1) − (1 − 1)(1 − 1)
=1
Where it’s assumed that there are two classes: C1 and C2. t1 [
In this procedure, we have considered the problem as a Multi-Class classification task. We have ifne-tuned the BERT model to get the contextualized embedding by the attention mechanism. We have tried Weighted XLM-Roberta large and weighted dehatebert-mono-english for the Four-Class classification task.
The main challenge in any classification problem is the imbalance in data. This imbalance in data may create a bias towards the most present labels, which leads to a decrease in classification performance. According to table 1 it is clearly evident that except code-Mixed data set, there are class imbalances present in the English, Hindi and Marathi dataset. In the English dataset (HOF), Hate and Ofensive labels are 46% more than (NOT) Non-Hate-Ofensive class. Similarly, in Hindi (NOT), Non-Hate-Ofensive class labels are 54% more than (HOF) Hate and Ofensive class. In Marathi, also (NOT) Non-Hate-Ofensive class labels are 44% more than (HOF) Hate and Ofensive class.
There is a lot of research has been done in this domain to make the data balance. Oversampling and Undersampling are very much popular data balancing methods, but they have coherent disadvantages also. We tried to implement data balance by using the class weight procedure. Table 3 describes the class weight distribution we have used in order to manage the data imbalance.
In Multi-Class classification also clearly, there is data imbalance present, and we normalized it. It is evident from Table 3 that (HATE) Hate Speech and (OFFN) Ofensive counts are quite
English
Hindi Marathi (NOT) Non Hate-Ofensive
Class Weight (HOF) Hate and Ofensive
Class Weight similar, but they are almost 50% less than (PRFN) Profane and (NONE) Non-Hate individually. We computed class weight for Multi-Class classification, which is present in Table 4.
For all the models presented here, we have pre-train on the target dataset for 20 epochs in order to capture the semantics. Along with that, we fine-tuned weighted and unweighted using cross-entropy loss functions[29]. We have used HuggingFace[30] and PyTorch [31]. Initial phases, we have used Adam optimizer[32] with an initial learning rate as 2e-5. We have not used early stopping while training.
Our observation was among most of the individual Transformer based BERT models, and the best performance was coming using XLM-Roberta-large (XLMR-large) in English Two-Class and Four-Class dataset and Marathi dataset. In contrast, we are getting the best performance in Code-Mixed dataset by using Custom XLM-Roberta-large. In the case of the Hindi dataset, mBERT is giving the best performance. The beauty of XLM-Roberta is that it has been pretrained on the parallel corpus. We have noted that the performance of XLM-Roberta-large is very much consistent with most of the regional languages.
While achieving the performance scores. We have used multiple random seeds and have observed that performance was heavily getting impacted for diferent seeds. It has been observed that while using mBERT, the performance varied 6-7% across our experimented languages. In the case of XLM-Roberta models, the performance was mostly the same and, it varied a maximum 1-2%. Table 5 shows the performance of XLMR-base, XLMR-large and mBERT-base for Two Classifiers XLMR-base XLMR-large mBERT-base Indic-BERT dehate-BERT Submission Name
English Macro F1
Hindi Macro F1
Code-Mixed
Macro F1
Marathi
Macro F1 Class classification results. We have shown the classification results for Four-Class classification in Table 6.
This team have not actively participated in the competition for the Marathi dataset, but later post-competition implemented all the transformer based models mentioned in this paper and found Macro F1 as 0.8756, which matches with the 3rd rank holder team from the competition.
In this shared task, we have compared and evaluated multiple Transformer-based architectures and discovered that XLM-Roberta-large mainly performs better than other Transformer-based models. However, performance varies based on a random seed. It has been observed that by changing random seed, XLM-Roberta performance was impacted less than other Transformerbased models. So, some of the actions will be to identify this observation and speculate the reason behind it. We have also used a couple of Transformer based models like IndicBERT and dehateBERT but was not getting enough raising performance compared with XLMRobeta and mBERT. Our immediate next step will be to investigate the reasons behind the lower performance of IndicBERT and dehateBERT, as IndicBERT is specifically pretrained with Indian languages and dehateBERT is an already fine-tuned model on the hate speech dataset. Trust and 2012 International Confernece on Social Computing, 2012, pp. 71–80. doi:10. 1109/SocialCom-PASSAT.2012.55. [14] T. Davidson, D. Warmsley, M. Macy, I. Weber, Automated hate speech detection and the problem of ofensive language, in: ICWSM, 2017. [15] G. K. Pitsilis, H. Ramampiaro, H. Langseth, Detecting ofensive language in tweets using deep learning, ArXiv abs/1801.04433 (2018). [16] Y. Goldberg, A primer on neural network models for natural language processing, Journal of Artificial Intelligence Research 57 (2015). doi: 10.1613/jair.4992. [17] G. L. D. la Peña Sarracén, R. G. Pons, C. E. Muñiz-Cuza, P. Rosso, Hate speech detection using attention-based lstm, in: EVALITA@CLiC-it, 2018. [18] G. I. Sigurbergsson, L. Derczynski, Ofensive language and hate speech detection for Danish, in: Proceedings of the 12th Language Resources and Evaluation Conference, European Language Resources Association, Marseille, France, 2020, pp. 3498–3508. URL: https://aclanthology.org/2020.lrec-1.430. [19] Z. Pitenis, M. Zampieri, T. Ranasinghe, Ofensive language identification in greek, in:
LREC, 2020. [20] Çagri Çöltekin, A corpus of turkish ofensive language on social media, in: LREC, 2020. [21] T. Mandl, S. Modha, P. Majumder, D. Patel, M. Dave, C. Mandalia, A. Patel, Overview of the hasoc track at fire 2019: Hate speech and ofensive content identification in indo-european languages, Proceedings of the 11th Forum for Information Retrieval Evaluation (2019). [22] S. Mishra, 3idiots at hasoc 2019: Fine-tuning transformer neural networks for hate speech identification in indo-european languages, in: FIRE, 2019. [23] S. Renjit, S. M. Idicula, Cusatnlp@hasoc-dravidian-codemix-fire2020:identifying ofensive language from manglishtweets, ArXiv abs/2010.08756 (2020). [24] A. Vaswani, N. M. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser,
I. Polosukhin, Attention is all you need, ArXiv abs/1706.03762 (2017). [25] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, in: NAACL, 2019. [26] S. Gaikwad, T. Ranasinghe, M. Zampieri, C. M. Homan, Cross-lingual ofensive language identification for low resource languages: The case of marathi, in: Proceedings of RANLP, 2021. [27] A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, V. Stoyanov, Unsupervised cross-lingual representation learning at scale, 2020. arXiv:1911.02116. [28] S. S. Aluru, B. Mathew, P. Saha, A. Mukherjee, A deep dive into multilingual hate speech classification, in: Machine Learning and Knowledge Discovery in Databases. Applied Data Science and Demo Track: European Conference, ECML PKDD 2020, Ghent, Belgium, September 14–18, 2020, Proceedings, Part V, Springer International Publishing, 2021, pp. 423–439. [29] S. Mannor, D. Peleg, R. Rubinstein, The cross entropy method for classification, in: Proceedings of the 22nd International Conference on Machine Learning, ICML ’05, Association for Computing Machinery, New York, NY, USA, 2005, p. 561–568. URL: https://doi.org/10.1145/1102351.1102422. doi:10.1145/1102351.1102422. [30] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. L. Scao, S. Gugger, M. Drame, Q. Lhoest, A. M. Rush, Huggingface’s transformers: State-of-the-art natural language processing, 2020. arXiv:1910.03771. [31] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, S. Chintala, Pytorch: An imperative style, high-performance deep learning library, 2019. arXiv:1912.01703. [32] I. Loshchilov, F. Hutter, Decoupled weight decay regularization, 2019. arXiv:1711.05101.