The identification of Hate Speech in Social Media has received much attention in research recently. There has been an ever-growing increase in demand particularly for research in languages other than English. The Hate Speech and Offensive Content (HASOC) track has created resources for Hate Speech Identification in three different languages namely Hindi, German, and English. We have participated in both Sub-tasks A and B of the 2020 shared task on hate speech and offensive content identification in Indo-European languages. Our approach relies on a combined model of multilingual RoBERTa (a Robustly Optimized BERT Pretraining Approach) model with pre-trained vectors and a Random Forest model using Word2Vec, TF-IDF, and other textual features as input. Our system has achieved a maximum Macro F1-score of 50.28% for English Sub-task A which is quite satisfactory relative to the performance of other systems and secured 8th position among participating teams.
Accuracy and efficiency of any supervised classification method are heavily dependent on the
corpora on which it is trained to make the dataset a very important entity for such classification
tasks. Hate Speech as such is a topic that has attracted the attention of researchers time and again
resulting in several previous initiatives of corpora creation [
The rest of the paper has been organized as follows. Section 2 describes the data on which the task was performed. The methodology followed is described in Section 3. This is followed by the results and concluding remarks in Section 4 and 5 respectively.
The corpus that has been used for this task contains texts in 3 different languages namely English, German, and Hindi, and consists of 3794, 2452, and 2963 tweets respectively. We have divided the dataset in the ratio 80:20 for training and validation purposes respectively. The distribution of data instances is given in Tables 1 and 2.
The three possible categories established in the dataset under this Sub-task are: • (NOT) Non-Hate-Offensive - This post does not contain any Hate speech, profane, offensive content. • (HOF) Hate and Offensive - This post contains Hate, offensive, and profane content.
English German Hindi
NOT HOF All NOT HOF All NOT HOF All • (OFFN) Offensive:- Posts under this class contain offensive content. • (PRFN) Profane:- These posts contain profane words.
English German Hindi
HATE OFFN PRFN
All HATE OFFN PRFN
All HATE OFFN PRFN All
For HASOC Task, our method uses a RoBERTa model for classification of reviews in each language into Hate and Offensive (HOF), Non- Hate, and offensive (NOT) sentiment labels. Then a method based on a Random Forest classifier with Word2Vec embeddings and TF-IDF (Term Frequency-Inverse Document Frequency) of commonly recurring words as input features were used to further classify the HOF sentiment into (HATE) Hate speech, (OFFN) Offensive and (PRFN) Profane. The workflow of our methodology is shown in Figure 1.
It consisted of the following steps:
1. Replacing emojis and emoticons by their corresponding meanings [
RoBERTa is equipped with the BERT’s language masking method, i.e. the system intentionally learns to predict sections of text which are hidden. Implementation of RoBERTa was done in PyTorch. This allows modification of the key hyperparameters in BERT, which include training with much larger mini-batches and learning rates and eliminating BERT’s objective of next-sentence pre-training. This leads the path in the improvement of the masked language modeling objective for RoBERTa compared with BERT and also leads to better downstream task performance.
For the sentiment analysis task from the Huggingface team transformers library, the pre-trained RoBERTa model has been accessed. XLM RoBERTa base model has been used for the task. Since it has been pre-trained on 100 different languages, the same model could be used for all three languages (Hindi, English, German) datasets. The RoBERTa base uses the BERT-base architecture hence it has 12-layer, 768-hidden, 12-heads, 125M parameters. The pre-trained RoBERTaTokenizer for the RoBERTa large model has been used to get the token representations. The learning rate of 1 − 5 has been selected and the model is trained for 10 epochs. The batch size has been set to be 32. For the training procedure, a Dropout Layer for some regularization and a fully-connected layer for the output is used in the model. The Dropout Layer reduces overfitting in the model by preventing complex co-adaptations on training data.
Different textual features are extracted from each of the text presents in the HOF and those
features are fed into a Random Forest classifier. The textual extracted features added to the model
for classification are:
1. The vector representations were obtained using Word2Vec.
2. The TF-IDF (Term Frequency-Inverse Document Frequency) [
The features extracted through the various above mentioned processes are selected by using the feature importance rankings for each feature. The features having a higher feature importance ranking were added to train the Random Forest model. The predictions given by the model for the test dataset were checked if they match with the HOF predictions given by the RoBERTa model. All the predictions which align with the HOF predictions are given by the RoBERTa model were kept as final outputs.
Our model secured 8ℎ and 23 positions in English Sub-task A and B respectively. For German Sub-tasks A and B, our model secured 20ℎ and 14ℎ positions respectively and for Hindi Sub-task A and B, our model secured 20ℎ and 16ℎ positions respectively. The performance of our model in terms of Macro F1-score is shown in Table 3.
A B
0.5028 0.1623
0.3231 0.0984
0.4599 0.1600
We have presented the system that we have used for participating in the 2020 shared task on hate speech and offensive content (HASOC) identification in Indo-European languages. Considering previous approaches, our approach is a comparatively different approach in terms of architecture as well as the methodology of feature extraction. It is a generalized and versatile framework and has shown satisfactory performance among all participating systems during the HASOC evaluations. In future works, we will further fine-tune the classification models to increase the performance and we will further experiment with the model in other languages. [13] T. Mandl, S. Modha, G. K. Shahi, A. K. Jaiswal, D. Nandini, D. Patel, P. Majumder, J. Schäfer, Overview of the HASOC track at FIRE 2020: Hate Speech and Offensive Content Identification in Indo-European Languages), in: Working Notes of FIRE 2020 Forum for Information Retrieval Evaluation, CEUR, 2020. [14] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Brew, Huggingface’s transformers: State-of-the-art natural language processing, CoRR abs/1910.03771 (2019). URL: http://arxiv.org/abs/1910.03771. arXiv:1910.03771. [15] J. Howard, et al., fastai, https://github.com/fastai/fastai, 2018. [16] A. Garain, A. Basu, R. Dawn, S. K. Naskar, Sentence simplification using syntactic parse trees, in: 2019 4th International Conference on Information Systems and Computer Networks (ISCON), 2019, pp. 672–676.