The current paper identifies the ofensive and hate content in three datasets of English, Hindi, and German. The dataset appeared in HASOC-2020 track. A fine-tuned Bidirectional Encoder Representations from Transformers (BERT) model is proposed to identify hate and ofensive contents from the social media posts. The experimental results show that the proposed BERT model achieved significant performance in identifying hate and ofensive content. For English subtasks A and B F1-score reported were 0.5031 and 0.1623, for Hindi subtasks A and B F1-score reported were 0.5300 and 0.0940 and for German subtasks A and B F1-score reported were 0.5109 and 0.1214 respectively.
Modern computers and technology have increased the social media user base exponentially. Consequently, the growth in the number of users posts per day has also increased exponentially. The internet has become an integral part of our everyday lives in recent years [1, 2, 3]. Most of us are increasingly becoming a social network addict in this digital era and have spent time surfing through the various media all day long. According to the latest survey, internet users now spend over 45 minutes every day on social media sites such as Facebook, Twitter, Instagram, and WhatsApp. These social media platforms allow freedom of speech and there is no limit to express our views through these media. This feature enables users to post friendly and non-friendly content. The friendly contents often help us in many critical situations such as disaster management, photo sharing, video streaming, citizen engagement, and so on [4, 5, 1, 2, 3]. On the other hand, unfriendly content may leave the victim in many harmful situations that may impact on user’s mental illness such as depression, sleep disorders, or even suicide in some cases [6, 7, 8]. Those unfriendly contents aim to harm a person or group based on their gender, caste, religion, wealth, and so on, and such contents are termed as hate speech or ofensive contents [6, 7].
Although ofensive contents on social media are available in many forms such as textual, audio, video, and image, the majority are in the form of text. Some of the works [8, 9, 10] reported by diferent researchers where they used textual social media contents to identify hate and offensive text. Kumari and Singh [8] proposed convolutional neural network-based to identify hate, ofensive, and profanity from English and Hindi tweets. Mishra and Pal [9] proposed an attention-based bidirectional long-short-term-memory network to identify hate, ofensive, and profanity from the English, Hindi, and German tweets. Mujadia et. al [10] proposed an ensemble-based model consisting of a support vector machine, random forest, and Adaboost classifiers to identify hate contents from the English, Hindi, and German language tweets.
The current paper aims to identify the ofensive or hate contents in Indo-European languages. Especially, the paper identifies the ofensive tweets which are written in either English, Hindi, or German. The datasets used in the paper are floated in HASOC-2020 track [11] 1. For each language, English, Hindi, and German two tasks were given. For the sub-task A, participants have to perform a course-grained binary classification where every tweet was either Hate and Ofensive (HOF) or Non- Hate and ofensive (NOT). For the sub-task B, participants have to further classify Hate and Ofensive (HOF) tweets into either hate speech, or ofensive, or profane classes. The subjectivity of the attack is diferent in hate speech and ofensive words. Further, the extremely ofensive content is termed as profane. For all the languages and tasks, we proposed diferent models which are based on the Bidirectional Encoder Representations from Transformers (BERT) technique.
The rest of the paper is organized as follows; Section 2 elaborates on the proposed system architecture, this is followed by Section 3 that explains model setting and experiment results, Finally, Section 4 summarizes the paper.
This section discusses the diferent steps for the proposed fine-tuned Bidirectional Encoder Representations from Transformers (BERT) model. For the given tasks, the organizer provided the training and development dataset with the label information. The submitted model is then tested by the organizer with the private testing datasets. The overall data statistic for English, Hindi, and German language tweets used in this work is listed in Table 1.
The text containing emoticons and emojis were converted into the corresponding text using the available dictionary2. After conversion of emoticons and emojis, we fixed a maximum word length of 30 for each text to give an input for the Bidirectional Encoder Representations from Transformers (BERT) model. For English dataset, we used pre-trained distilbert-baseuncased BERT model. For the Hindi and German dataset, we used pre-trained distilbert-basemultilingual-cased BERT model. The distilbert-base-uncased3 is a pre-trained model trained on 1HASOC provides a forum and a data challenge for multilingual research on the identification of problematic content.
2https://github.com/NeelShah18/emot/blob/master/emot/emo_unicode.py 3https://huggingface.co/transformers/pretrained_models.html the Toronto Book and full English Wikipedia corpus. The distilbert-base-uncased contains 6 layers, 768 dimension and 12 heads with 66 Million parameters. The distilbert-base-multilingualcased4 is a multi-lingual pre-trained model trained on the 104 diferent languages of Wikipedia texts. The distilbert-base-multilingual-cased contains 6 layers, 768 dimension and 12 heads with 134 Million parameters. To fine tune the respective distilbert-base-uncased and distilbert-basemultilingual-cased models, a batch size of 32 and learning rate of 2 ∗ 10−5 were used to train the model for the 20 epochs.
1.0 250 e0.8 t a
R 200 ive0.6 it s o
P 150 rTeu0.4
0.2 100 50 0.00.0
0.2 The results of the fine-tuned BERT models for the English, Hindi, and German dataset is listed in Table 2. We listed the results of the validation (Val) datasets and the results provided by the organizers for the private testing (Test) datasets. For English sub-task A, a macro averaged precision, recall, and 1-score of 0.87, 0.87, and 0.86, respectively were achieved for the validation dataset whereas a macro 1-score of 0.5031 was achieved for the test dataset (as can be seen in Table 2. The confusion matrix and ROC curve for English sub-task A validation dataset can be seen from Figures 1 and 2, respectively. For English sub-task B, a macro averaged precision, recall, and 1-score of 0.60, 0.56, and 0.57 were achieved respectively for the validation dataset. For the testing dataset, a macro 1-score of 0.1623 was achieved. The confusion matrix for English sub-task B can be seen from Figures 3 and 4, respectively for the validation dataset.
For Hindi sub-task A, a macro averaged precision, recall, and 1-score of 0.67, 0.64, and 0.65 were achieved for the validation dataset whereas a macro averaged 1 score of 0.5300 for the test dataset. The confusion matrix and ROC curve for the Hindi sub-task A validation dataset can be seen from Figures 5 and 6, respectively. For Hindi sub-task B, a macro averaged precision, recall, and 1-score of 0.39, 0.39, and 0.36 were achieved for the validation dataset. The confusion matrix for Hindi sub-task B validation dataset can be seen from Figures 7 and 8, respectively.
For German sub-task A, a macro averaged precision, recall, and 1-score of 0.81, 0.73, and 0.76 were achieved for the validation dataset whereas a macro averaged 1-score of 0.5109 was achieved for the test dataset. The confusion matrix and ROC curve for the German sub-task A validation dataset can be seen from Figures 9 and 10, respectively. Similarly, for the German sub-task B, a macro averaged 1-score of 0.65, 0.58, and 0.60 were achieved for the validation dataset. The confusion matrix and ROC curve for the German sub-task B validation dataset 4https://huggingface.co/transformers/pretrained_models.html 0.00.0
0.2
Hate and ofensive social media contents may leave the victim in many harmful situations that may impact on user’s mental illness such as depression, sleep disorders, or even suicide in some cases. Therefore identification of such hate and ofensive social media contents are essential natural language processing tasks. In this work, we have proposed a fine-tuned BERT model for the identification of hate and ofensive tweets from the English, Hindi, and German language text. The proposed fine Tuned BERT model achieved significant performance for all 0.8 e t a R iitve0.6 s o P eu0.4 r T 0.2 three English, Hindi, and German language tweets. For English subtasks A and B F1-score reported were 0.5031 and 0.1623, for Hindi subtasks A and B F1-score reported were 0.5300 and 0.0940 and for German subtasks A and B F1-score reported were 0.5109 and 0.1214 respectively. The future work may explore the role of embedding with fine tuned BERT model for better classification performance. [1] A. Kumar, N. C. Rathore, Relationship strength based access control in online social networks, in: Proceedings of First International Conference on Information and CommuHATE [11] T. Mandl, S. Modha, G. K. Shahi, A. K. Jaiswal, D. Nandini, D. Patel, P. Majumder, J. Schäfer, Overview of the HASOC track at FIRE 2020: Hate Speech and Ofensive Content Identification in Indo-European Languages), in: Working Notes of FIRE 2020 - Forum for Information Retrieval Evaluation, CEUR, 2020.