In this paper, we describe our models submitted to the HASOC shared task conducted as part of FIRE 2020. We have presented two directions to approach the problem of hate speech detection which are based on the advancements in the field of natural language processing. We present classical machine learning approaches using Support Vector Machine (SVM) and transfer learning approaches at sentence level using BERT. We have shown through experimental results that Transfer learning based approaches beat Machine learning based approaches in identifying the nuances of hate speech in a sentence. We have performed experiments on the English and Hindi datasets. On the public test dataset provided, we obtain a macro F1 score of 0.90 and 0.67 on English and Hindi languages respectively for subtask A and scores of 0.54 and 0.44 for subtask B. We also observe a diference of around 0.05 macro F1 score between BERT based models and SVM based models on English public test data, while we see only a diference of 0.01 on Hindi public test data in subtask A. We have highlighted the importance of a monolingual model over a multi lingual BERT based model for hate speech detection. We also highlight the importance of having a large, balanced training dataset on model performance for hate speech detection.
Hate speech detection has gained a lot of importance in the NLP community. The HASOC
2020 [
We have conducted our investigation on the English and Hindi datasets. Simple word based ifltering approaches to tackle this problem are not eficient enough as a sentence could only be tagged as hateful based on the context in which that word has been spoken. So, with this simple swear word-based filtering approach, we might end up falsely tagging a lot of sentences hateful.
The BERT model [
HASOC 2020 has ofered two subtasks:
Subtask-A: Classification of the post as hate speech (HOF) and non hate speech (NOT). All
categories of hate like profanity, ofensive language, hate speech are labelled as HOF as part of
this task. This is a binary classification task. The English dataset is almost perfectly balanced
while the Hindi dataset is imbalanced with only 25% of the data being HOF.
For English subtask-A, we have used an additional dataset named OLID [
This section is divided as follows: We first present a common pre-processing pipeline which we follow for both subtasks A and B. We then present our method description for subtask-A and then describe our methodology for subtask-B.
The dataset has been extracted from social media sites like twitter. So, it is expected that it
would have a lot of social media jargon and hence contains unconventional social media style
of writing. It is essential to pre-process the dataset before passing it on to the models. The
following is the pre-processing pipeline we followed for English:
1. URL Removal: All the urls have been replaced with a special token.
2. User Mention Removal: All the user mentions have been replaced with another special token.
3. Conversion of emojis and emoticons into corresponding text.
4. Expansion of hashtags, removal of stop words and unnecessary symbol, case folding.
5. Map all elongations to the same word. (For example "yeah", "yeahhhh", "yeeaahhh" into
"yeah")
For Hindi we additionally stem the words and we did not perform step-5. All other steps were
performed. We have used the NLTK [
We have experimented with two kinds of approaches: 1. Classical Machine Learning Approaches using SVM 2. Transfer Learning Approaches using Neural BERT
We perform all further processing on the pre-processed data. There are two steps in this:
1. Sentence Encoding: The resultant text after pre-processesing has to be converted into
vector form for the classification models to train upon. The output of this step is a
vector which tries to represent the sentence. For this, we had experimented with diferent
sentence encoding mechanisms:
a) Universal Sentence Encoder [
While trying the above machine learning approaches on the Hindi dataset, we experimented
The pre-processing pipeline followed before performing the experiments mentioned in this
section is same as that mentioned in Section 3.1. We describe our approach below:
1. English: For English, we have used the monolingual pre-trained cased BERT model [
7We are just using USE/Avg Fasttext as embedding schemes. We did not fine tune these architectures and hence we did not place these mechanisms in the Transfer Learning section.
8http://mylanguages.org/devanagari_romanization.php on two forms of data: 1. Original devanagari form
transliteration.
2. Transliterated form: We performed transliteration on the Devanagari form to convert it to the standard Roman form. We have used mylanguages website 8 to perform the We appended the OLID dataset to the 2020 English subtask-A dataset to obtain the final dataset. We took the pre-trained BERT model and fine tuned it with this final dataset. 2. Hindi: For Hindi, we have used the multilingual cased pre-trained BERT model 9 to get the sentence encoding. We will call this model in this paper. The classifier again is a dense layer with the output dimension as 1, denoting the label itself. We took this pre-trained multilingual model and fine tuned it on the 2020 Hindi dataset for subtask-A.
For subtask-B, the preprocessing pipeline is same as the one described in Section 3.1. The dataset for this subtask was pretty imbalanced. There were two strategies which we had tried, second one adding on to the first strategy. We follow the same approaches for English and Hindi. The sentence encoding schemes remain same as the ones mentioned in Section 3.2.1. 1. Approach 1: In this approach, we do not consider it as an extension to subtask-A. We simply take the encoded data and its corresponding label for subtask-B, for training, and ift an SVM on this data. This would translate to multiclass classification with 4 labels: NONE, HATE, PRFN and OFFN. We can see here that this classification is independent of the previous task’s output. 2. Approach 2: In this approach, we first filter out all the non-NONE labelled data from the training data for subtask-B and fit an SVM on it. Henceforth, we will call this model SVM_1 in the paper. Now we take the first subtask’s output and directly label the ’NOT’ predictions from the first task as ’NONE’. For the remaining posts, we predict using the SVM_1 model.
HASOC organizers have released public test data with actual ground truth labels this year. The ifnal leaderboard ranks were decided based on a private test dataset. All the results reported in Tables 2-5 are on the 2020 public test data. We included macro precision, macro recall and macro F1 score in our results. Macro values were calculated by averaging the corresponding value for all the classes. Organizers of HASOC have used macro-F1 as the primary metric in deciding the leaderboard positions. Table 6 shows the performance of our model on the private test data. The results of Hindi subtask-B were not announced at the time of submission of this paper.
For English subtask-A, results labelled with OLID+2020 denote models trained on 2020 HASOC dataset appended with OLID dataset as mentioned in the methodology section. The results without that label are trained on the 2020 HASOC dataset only. The label trans in Hindi results denotes that the model has been trained on the transliterated version of the dataset. All the results for subtask-B are based on the two approaches described in section 3.3. 0.86 0.85 sion sion BERT BERT BERT (OLID+2020) (OLID+2020) Method TfIdf Vect + SVM TfIdf Vect + SVM
An important thing to notice in the case of BERT models is that when there is monolingual model available, it is outperforming the multilingual model. We saw a diference of around 0.012 macro F1 score between both the models. The multilingual model though turns out to be better than the classical SVM model. We saw a diference of around 0.02 macro F1 score again
Method TTffIIddff VVeecctt ++ SSVVMM AAvvgg FFaasstttteexxtt ++ SSVVMM TfIdf Vect + SVM AAvvgg FFaasstttteexxtt ++ SSVVMM sion sion
We could see significant improvement in macro F1 score between model and the SVM model in cases where there was data imbalance. In subtask-B where the dataset is heavily imbalanced and is very small, we could see improvement upto 0.12 macro F1 score, which is quite significant. In subtask-A, there wasn’t any significant improvement for English as the dataset was already balanced. There was an improvement upto 0.15 macro F1 score for Hindi subtask-A which is expected due to the imbalance in the dataset.
From Table 5 we can see that there is an improvement of 0.12 macro F1 score between SVM models in attempt-1 and attempt-2. We see that the second approach has indeed reduced the bias towards the majority class [NONE] and has improved the overall Macro F1 score. We could see a significant number of samples now getting classified into classes other than ‘NONE’. From experimental analysis, we could see that usage of additional data for English Subtask-A did help in improving the macro F1 score. As we can see in Table 2 there was an improvement of 0.01 macro F1 score in the case of BERT and 0.03 macro F1 score in the case of TfIdf Vect + when we additionally use the OLID dataset.
We have submitted two runs for each subtask for English and Hindi. For English subtaskA, we submitted USE+ model and the (OLID+2020) model described in the previous sections. For Hindi subtask-A, we submitted based model and the Avg Fasttext model described above. For subtask-B, we submitted approaches 1 and 2 described in section 3.3 as part of the two runs for both the languages. For subtask-B, we submitted models with USE sentence encodings for English and for Hindi we submitted models with TfIdf sentence encoding scheme.
We have described two main directions for the hate speech detection problem: Classical machine learning approach and Transfer learning based approach. We have shown through experiments that BERT based models pre-trained on large English corpora and fine tuned on the dataset available yield good results. We have achieved a macro F1 of 0.90 and 0.67 for English and Hindi subtask A respectively which is 0.05 and 0.01 more than the best performing SVM counterpart. The proposed model performed reasonably well, it was of by very small margins compared to the best performing models in almost all cases. It was of by 0.01 and 0.02 F1 score for English and Hindi respectively in subtask A and 0.03 and 0.11 F1 score in subtask B. The poor performance in subtask-B could be attributed to the fact that the dataset was small and imbalanced. Furthermore, the classes individually being very fine grained makes it dificult for models to capture. We should try to build more robust models which capture these subtleties between classes and also possibly try to build a more balanced and larger dataset in order to improve the eficiency of the hate speech detection systems. Various approaches like LSTM, CNN, LSTM-Attention, etc can also be tried. Comparing the eficiency of these approaches with BERT and SVM based models is left to future work. We plan to experiment with BERT based architectures for subtask-B as part of future study. We hope our experimentation and analysis helps in tackling the problem of hate speech detection in some way.