Hate speech continues to be a pressing concern in online social media (OSM) platforms, necessitating efective automated detection systems. In this paper, we propose a unified approach, encompassing both Task 1 & 4, to tackle the challenge of hate speech recognition within the HASOC 2023 framework. It addresses the complexities of multilingual OSM by employing cutting-edge Natural Language Processing (NLP) techniques and leveraging powerful language models put forward by team CNLP-NITS-PP. The key objective is optimising precision-recall trade-ofs in hate speech detection, spanning English and Indo-Aryan languages. The empirical results demonstrate the efectiveness of our approach in isolating explicit signs of hate speech, emphasizing model eficiency, interpretability, and the importance of diverse linguistic nuances in creating safer online environments. This integrated work sets the stage for advancements in hate-span detection and underlines the significance of fostering responsible and inclusive online conversations across various language environments.
Social media platforms such as Twitter and Facebook have become integral to modern life,
providing a global platform for individuals to express themselves. However, the openness of
these platforms has also led to the proliferation of harmful content, including hate speech and
harassment [
Detecting ofensive content is challenging due to its diverse linguistic forms, necessitating
context-aware models to pinpoint hateful or abusive text snippets [
The spread of hate speech and inflammatory language on social media platforms is a major worldwide problem in today’s digital age, as communication plays a crucial role in determining public debate. Low-resource languages like Sinhala, Gujarati, Bengali, Bodo, and Assamese, which have received little attention in the field of Natural Language Processing (NLP), are severely afected by this problem.
Our research activities cover a range of tasks for identifying harmful and hateful content in
these underrepresented languages1. In Task 1, we tackle Sinhala (Task 1A), a language with
a unique alphabet and intricate grammatical structures, and further broaden our emphasis to
Gujarati (Task 1B), where a dearth of labelled data poses a significant obstacle. Task 4 extends
the study’s horizons by including Bengali, Bodo, and Assamese [
The importance of our work lies in its role in safeguarding cultural identities and developing secure online environments for these language speakers. To respond to the complexities of hate speech in diferent linguistic and cultural contexts, we use cutting-edge NLP approaches, language-specific feature engineering, and pre-processing. Additionally, we investigate how models developed for languages with abundant resources may be applied to languages with limited resources to improve hate speech identification.
Through this extensive study project, we hope to advance responsible digital communication, better understand how to identify hate speech in diferent linguistic contexts, and create a more welcoming online space for every language.
Hate speech detection in fairly low-resourced languages such as Sinhala and regional Indian
languages has recently attracted research attention [
For the Sinhala Language, a few research works have been conducted. (Munasinghe et al.,
2022)[
In case of Indian Languages, prior work on hate speech detection has concentrated primarily
on Hindi and Malayalam. For instance, (Bohra et al., 2018)[
For Gujarati, (Khurana et al., 2022)[
For the Bengali language, (Das et al., 2020)[
Assamese is a relatively low-resource language. (Ghosh et al., 2023)[
For the Bodo language, there is limited prior research. The HASOC 2023 shared task provides
the pioneering benchmark hate speech detection dataset in Bodo. This will encourage further
research in this low-resource language (Chakravarthi et al., 2021)[
(Vetagiri et al., 2023a)[
Much previous work has relied on machine learning and deep neural networks. But these necessitate substantial labelled datasets, which are scarce for low-resource languages. Recent emphasis has focused on multilingual models such as mBERT, which can leverage data from high-resource languages. Domain adaptation approaches have also proven efective in adapting models trained on English data.
The HASOC 2023 shared tasks furnishes standard labeled benchmark datasets for hate speech detection. This will catalyze research in these languages and progress the state-of-the-art. Multilingual models and cross-lingual transfer learning are promising avenues to explore for these languages.
Task 1 of the HASOC’23 aimed at identifying hate, ofensive, and profane content in social
media posts in two languages, namely Sinhala(Task 1a) and Gujarati(Task 1b) [
Task 4 was similar to Task 1 and required us to detect hate speech in three other Indian languages, Bengali, Assamese and Bodo. For all the given languages, the training and test datasets had already been provided.
Creating coarse-grained binary classification models to divide tweets into the following two categories was the primary goal for the tasks: • Hate and Ofensive(HOF): Posts that contain hate speech, vulgarity, or ofensive material. • Non-Hate and Ofensive (NOT): Posts devoid of ofensive language, hate speech, or any other negative elements.
The Sinhala dataset provided for train and test had been sourced from the recently released SOLD: Sinhala Ofensive Language Detection dataset, which served as a comprehensive resource for the particular task. The training dataset had been further divided into three columns. The ifrst one consisted of the post id, the second of the tweet text, while the third column consisted of the labels, HOF and NOT, for each of the corresponding tweets in the same row.
For Task 1b, the training dataset had 200 tweets, primarily categorized into two labels, HOF and NOT, besides three other columns, including the tweet id, the UserName and the date of creation. It is noteworthy that the exact source of the dataset has not been mentioned in the materials provided for the competition.
The training and test datasets for the task had already been provided in all three languages, Assamese Bengali and Bodo. However, it is worth noting that none of the sources for the data were explicitly mentioned.
Distribution of HASOC’23 training datasets for Task 1 and Task 4. For each language, the total no of text entries and the corresponding no of tweets per class are shown below.
Test Set size: For Task 1A, the Sinhala test dataset consisted of 2500 tweets. This had to be labeled as either HOF or NOT based on our model.
For Task 1B, the Gujarati test dataset consisted of a total of 1196 tweets to be labelled similarly.
Total text entries
Hate and Ofensive(HOT)
Not Hate(NOT) For Task 4, the Bengali, Assamese, and Bodo test datasets consisted of 320, 1009, and 420 text entries, respectively. These entries had to be labeled as either HOF or NOT based on our model.
We used a number of standard preprocessing methods prior to training our model using the given datasets. Given that the training datasets provided had their texts sourced from Twitter, it was anticipated to contain certain unwanted elements, such as emojis, URLs, mentions and special characters. In order to guarantee the accuracy and relevancy of the text data, we followed procedures to remove such unwanted noise.
In this section, we describe the methodology and the experimental setup used for the various tasks under HASOC 2023. We conducted a thorough investigation into various neural network architectures, pretrained Large Language models, and classical machine learning models to identify the most efective model for the task.
For the task of Identifying Hate, ofensive and profane content in Sinhala & Gujarati, the models that resulted in the best performance are as follows: • A CNN-based Binary Classification Model with FastText Embeddings.
• A CNN-BiLSTM based Hybrid Model with FastText/GloVE Embeddings
Inspired by the works of Kim et al., [
For the convolutional layer, we employed a one-dimensional convolution operation utilizing 100 filters with a kernel size of 3, leading to a systematic scanning of the text sequences and identifying pertinent patterns in the data. An activation function, the rectified linear unit (ReLU), was also applied to introduce non-linearity and enable the model to capture complex relationships in the data. Subsequently, a dense layer with 50 neurons and a ReLU activation function, coupled with an L2-norm constraint, was added to transform the extracted features further. Dropout with a rate of 0.5 was applied as a regularization technique to prevent overfitting. The resultant vector was then concatenated with the feature vector and the output was passed onto a dense output layer with sigmoid activation and cross entropy loss as shown in Figure 1, to produce the binary hate classification for the model.
Based on the contributions of Vetagiri et al., [
The output of these two layers is then passed through a dense layer with a sigmoid activation function for Binary Classification. To prepare the Input data, the model uses the exact pretrained FastText embeddings for the respective language mentioned above, representing the words as 300-dimensional dense vectors, with the sequence length capped at 70 words, which was held as non-trainable. A similar implementation of this model using pre-trained GLoVE embeddings Kumar et al., [23] showed identical results. For the CNN layers, we first employed a SpatialDropout1D layer, a dropout variant that selectively drops entire 1D feature maps during training, to combat overfitting. Subsequently, a one-dimensional convolution layer with 64 iflters and a kernel size of 3 was used to capture local textual patterns with fine granularity.
For the BiLSTM part, we used the initial layer with 128 units and a return sequence setting with a dropout of 0.1 and recurrent dropout of 0.1 to process the text inputs in both forward as well as reverse directions followed by a Global Average Pooling and a dense layer with 128 units and a rectified linear unit (ReLU) activation function to introduce non-linearity. Subsequently, a dropout layer is employed whose output is then concatenated with the feature vector and passed through a dense layer with sigmoid activation as shown in figure 2 to produce the overall model for Binary Hate Classification. Our models are trained using the RMSprop optimiser, and our loss function is a binary cross-entropy function. To fine-tune our hyper-parameters over a range of values, we conduct a grid search and select the best-performing model according to validity accuracy. However, no attempt at experimentation by reversing the order of the CNN & Bi-LSTM Layers was made for this particular task.
The two model architectures used in Task 1 were also experimented with in Task 4. These architectures were used to implement the CNN and the CNN-BiLSTM models which used pre-trained FastText embeddings in the respective languages (except for Bodo, for which Hindi embeddings were used) representing each word as a dense vector with 300 dimensions.
In addition to these two architectures, several others were also experimented with, the details for which are discussed below:
Considering the low-resource nature of Task 4 and the limited size of the datasets, we experimented with pre-trained models based on the Bidirectional Encoder Representations from Transformers (BERT) architecture Devlin et al., [24] [25]. We experimented with the Tensorflow Hub to access the pre-trained BERT models.
We used the "bert-multi-cased-preprocess/3" for text processing and the "bert-multi-cased-L12-H-768-A-12/4" encoder for contextualized word embeddings from the TensorFlow Hub, which is trained on multilingual Wikipedia Data. The model utilizes a BERT preprocessing layer for tokenization and embeddings of the input text, followed by a BERT encoder layer to generate contextual embeddings containing the complex contextual relationship within the language. This is followed by a dropout Neural Layer with a 10% dropout rate to enhance generalization and mitigate overfitting. The final trainable dense layer with 769 employs the sigmoid activation function, producing the binary classification outputs for the given languages 4.2.2. GPT-2 Model We also explored GPT-2 as a state-of-the-art pre-trained large language model for the task. GPT-2 is a transformer-based model that takes a sequence of words, represented as dense vectors, as input and uses many intermediate layers to extract contextual information for the input text. The output is then passed through a dense layer, producing the Binary Classifier. For the task, we used a pre-trained GPT-2 that contained 768 parameters, fine-tuned on the training dataset for each language in an 80-20% split and a further 20% split from the training set for validation. The input text is tokenised and passed through the model for fine-tuning. The model uses an Adam optimizer for optimization with a learning rate of 1e-5. The batch size for the model is 8
Due to the small size of the training datasets provided for all three languages, we tried implementing classical Machine-Learning approaches as well to accomplish our goals. We experimented with multiple Machine-Learning architectures such as Support Vector Machines (SVMs), Linear Regression, Logistic Regression and Random Forest Classifiers.
However, although the training set accuracies and the macro F1 scores for all the above architectures were quite close, we observed that the Logistic Regression Model achieved the best overall performance. Hence, we created a simple Logistic Regression Model using SciKitLearn Library, trained the Model with the training datasets for each language, and implemented simple tf-idf vectorisation for embedding the input sentences
5.1. Task 1 & 4 For evaluation the models, the test accuracy for this test data was used for the initial evaluation of the models. In addition to this, the Macro F1 scores, which were acquired by the models on submission in the HASOC 2023 Submission portal were also considered. For Task 4, although the performance of each of the aforementioned models was analyzed, only a few of the models for each language gave the best performance, summarized in Tables 1 & 2.
As it is evident from Tables 1 & 2, the CNN-BiLSTM+FastText/GLoVe Model gave the best Macro F1 Score for Task 1A(Sinhala), and the CNN+FastText Model gave the best Macro F1 Score for Task 1B(Gujarati). For Task 4, in both Bengali and Assamese, the CNN+FastText Model gave the highest accuracy with a Macro F1 score of 0.60108 in Bengali and 0.59485 in Assamese. For Bodo, evidently, the simple Logistic Regression Model gave a much better performance than the other model architectures with a Macro F1 score of 0.66925, possibly due to the small size of the training dataset.
Our research has been dedicated to the vital task of identifying hate speech, particularly in Indo-Aryan languages such as Bengali, Assamese, Bodo, Gujarati, and Sinhala. We’ve devised a comprehensive strategy that unites these linguistic intricacies under a single versatile model. Through domain-aware pre-training and meticulous alignment of our models with language-specific context, we’ve significantly enhanced hate speech detection. Furthermore, our exploration into model ensemble techniques has bolstered detection accuracy and resilience across diverse language settings, laying a foundational step towards comprehensive hate speech detection in Indo-Aryan languages. Our overarching goal is to foster a safer and more inclusive digital space for speakers of diverse linguistic backgrounds.
As we look to the future, our work paves the way for further advancements in hate-span detection, focusing on model eficiency, interpretability, and an expansive training data corpus encompassing evolving hate speech trends and linguistic variations. We also recognize the potential of real-time monitoring and context-aware integration in dynamically evolving online environments.
We wish to extend our appreciation to the Computer Science and Engineering Department of the National Institute of Technology Silchar for granting us the opportunity to carry out our research and experiments. We are grateful for the support, resources, and research environment ofered by the CNLP & AI Lab at NIT Silchar. layer for text classification, Neurocomputing 337 (2019) 325–338. URL: https:// www.sciencedirect.com/science/article/pii/S0925231219301067. doi:https://doi.org/ 10.1016/j.neucom.2019.01.078. [22] A. Sherstinsky, Fundamentals of recurrent neural network (rnn) and long short-term memory (lstm) network, Physica D: Nonlinear Phenomena 404 (2020) 132306. URL: https://www.sciencedirect.com/science/article/pii/S0167278919305974. doi:https://doi. org/10.1016/j.physd.2019.132306. [23] S. Kumar, S. Kumar, D. Kanojia, P. Bhattacharyya, “a passage to India”: Pre-trained word embeddings for Indian languages, in: Proceedings of the 1st Joint Workshop on Spoken Language Technologies for Under-resourced languages (SLTU) and Collaboration and Computing for Under-Resourced Languages (CCURL), European Language Resources association, Marseille, France, 2020, pp. 352–357. URL: https://aclanthology.org/2020.sltu-1. 49. [24] J. Devlin, M. Chang, K. Lee, K. Toutanova, BERT: pre-training of deep bidirectional transformers for language understanding, CoRR abs/1810.04805 (2018). URL: http://arxiv. org/abs/1810.04805. arXiv:1810.04805. [25] K. Ghosh, A. Senapati, U. Garain, Baseline bert models for conversational hate speech detection in code-mixed tweets utilizing data augmentation and ofensive language identification in marathi, in: Fire, 2022. URL: https://api.semanticscholar.org/CorpusID:259123570.