This paper presents KMI-Panlingua's system description which was submitted at the FIRE Shared Task 2019 on Hate Speech and Offensive Content Identi cation in Indo-European Languages. Our team submitted systems for all the 3 sub-tasks in two languages - English and Hindi. We experimented with 2 kinds of systems - classic machine learning using SVM and BERT-based system. We discuss the systems and their results in this paper.
In the digital era, social media such as Facebook, Twitter, WhatsApp, etc, is one of the most important mediums to circulate information as well as communicate in the society. While it helps in quickly spreading the information in society, it has also become hotbeds for the spread of hate speech and o ensive contents. The hate speech and o ensive content could range from political and religious to caste and gender or any issue that could divide and polarise a community. So, it is required to build a robust automatic hate speech and o ensive content detection system which may help to lter out these types of content such that they do not spread in the community.
There have been a lot of e orts towards building such systems (notably [
The FIRE 2019 shared task on Hate Speech and O ensive Content Detection in Indo-European Languages (HASOC 2019) is another such e ort in this direction. In this paper, we discuss the development of automatic hate speech and o ensive content identi cation systems for all the 3 sub-tasks in two languages - English and Hindi - as part of this shared task. We experimented with 2 kinds of systems - classic machine learning using SVM and BERT-based system - to explore their relative applicability for the task.
The rest of the paper is divided into four section. Section 2 discusses of the dataset size and its types. Section 3 provides a detailed description of the conducted experiments. While section 4 reports the developed systems' results and their error analysis.Section 5 ends with the concluding remarks. 2
In order to conduct the experiments, we used the data for English and Hindi
languages which were shared in the FIRE Shared Task HASOC 2019[
Thus it is a binary classi cation task. 2. Sub-task 2: If the content is marked HOF in the rst sub-task then it is marked as Profanity (PRFN), O ensive (OFFN) or Hate Speech (HATE) in this stage. It is a 3-class classi cation problem. 3. Sub-task 3: The HOF contents are also marked for whether they are targeted towards an individual or a group (TIN) or not (UNT).
No additional dataset or resources (except the BERT pre-trained models) have been used for the task.
based on Hugging Face pytorch-transformers library 5) for the experiments. The implementation details of the Fast-Bert are given in the two blogs by the author 6, 7.
Support Vector Machines [
For SVM, we used 5-fold cross-validation for guring out the optimum model. We experimented with the following sets of features 1. Word n-grams (unigrams, bigrams and trigrams) 2. Character n-grams (trigrams to 5-grams) 3. A combination of di erent word n-grams and character n-grams features
A gird search was performed for C-values from 0.0001 upto 10 (with a 10x interval in between two C-values) for each of the feature combination and each of the sub-tasks. The classi ers that gave the best performance for each sub-task in each language are give in Table 2.
As we could see, a combination of word n-grams and character n-grams have given the best performance in all the cases. It is apparent from the table, as
6 https://medium:com/huggingface/introducing-fastbert-a-simple-deeplearning-library-for-bert-models-89ff763ad384 7 https://medium:com/huggingface/multi-label-text-classification-usingbert-the-mighty-transformer-69714fa3fb3d expected, word n-grams are not very helpful in case of Hindi while for English using upto trigrams gives the best performance in all the three sub-tasks. It is expected since Hindi is expected to have much more morphological information than English and those are generalised by the use of character n-grams. Moreover, character 5-grams are expected to be almost equivalent to word unigrams (or at least stemmed word n-grams) and so, in English, adding more than character trigrams in the rst task does not add any new information. For the other two sub-tasks, the performance of the SVM classi ers are not as good as that of the rst sub-task and the classi ers fail to generalise well enough for both the training as well as test dataset. As such the best performance of one particular classi er might be incidental. Moreover, it must also be noted that the di erence between classi ers trained using di erent feature sets was not huge and only a marginal improvement was noticed with di erent word-level and character-level feature combinations. 3.2
We could experiment with BERT in only sub-task 1 and 3 for English using BERT (because of the lack of su cient hardware resources required for netuning the BERT models for other sub-tasks as well as for Hindi). We netuned the pre-trained BERT-base-uncased model released by Google for this task. The ne-tuning was carried out on a standard Google Colab GPU system. For both the sub-tasks, the models were trained for 10 epochs and used the LAMB optimizer. 4
The overall results on the test set show that for sub-task 1 in English, BERT
substantially outperforms SVM by a huge margin. However, for sub-task 3, it fails
to perform at par with the SVM. In general, classi ers for sub-task 1 is able to
perform much better than those for sub-task 2 (which was a 3-class classi cation
problem) and sub-task 2 (which was also a binary classi cation problem, like
sub-task 1) [
Besides these, in other cases where we experimented with only SVM-based classi er, while our overall rank was relatively good, the di erence in macro F1 scores of our system and that of the top team hovered from 6 points (Hindi subtask 1) to 10 points (English sub-task 2). These comparisons are summarised in the Fig 2.
In addition to this big picture, if we take a closer look at the kind of errors our system has produced, it is quite apparent that the BERT system has better generalised and has led to an improvement in the precision score while maintaining a good recall, leading to an overall improvement in the performance (see Fig 3 and Fig 4 for a comparison of SVM and BERT systems for English sub-task 1).
In the sub-task 3 8 (see Fig 5 and Fig 6), the picture is little more complicated. There is an improvement in the precision of both the classes using the BERT system. However, since the total number of training instances are low, the features for classi cation are less. Thus the precision is extremely low for both the classes, more so for the minority class. Moreover, since the dataset is highly imbalanced, there are not su cient discriminating features. And thus the recall is also very low for both the classes. However, what is quite interesting to note is that for SVM, recall is extremely low for the minority class (UNT), while in BERT, it is exactly the opposite - the majority class gets a very low recall while the recall for the minority class is quite good. This shows a fundamentally di erent learning pattern for deep learning systems like BERT and thereby depicting a better tendency for generalisation.
Among the other tasks, the lack of su cient training instances as well as the availability of explicit, lexical features seem to have played a major role in the low performance in those tasks. In sub-task 2 of English (Fig 7), PRFN is de ned in a way that they have more explicit, lexical level features which could be generalised by a classi er like SVM, hence, a better performance than HATE and OFFN despite HATE being substantially higher in number. A similar trend is noticed in Hindi dataset where the performance is best in task 1 (Fig 8), which had a relatively large training sample. In sub-task 2 (Fig 9), PRFN is the best8 For sub-task 2 and 3, the category the classi ers were not trained for predicting NONE, as per the instructions given by the task organisers. However, only one test le was given for testing and it was compulsory to give a prediction for each sample. So we gave a NONE to all those test samples that were classi ed as NOT in sub-task 1. So the errors related to NONE are not made by the classi ers for sub-task 2 and 3; rather these errors have percolated because of misclassi cations in sub-task 1 and they should be interpreted as such in the confusion matrices for these two sub-tasks performing class, while in sub-task 3 (Fig 10), the majority class performs the best. Among the two languages, the classi ers for Hindi seems to be performing relatively better because of the greater number of training samples as well as relatively more balanced dataset (especially in sub-task 3). In this paper, we have given a description of the KMI-Panlingua system developed for HASOC at FIRE 2019. Our analysis of the results show that BERT is able to generalise better for the task than SVM and even in cases of unbalanced dataset, BERT is able to achieve high recall (but low precision) even for the minority class (with very little training samples). This depicts a fundamental di erence in the way a linear classi er like SVM and BERT learns. In addition to this, the low performance in sub-task 2 and 3 could be largely attributed to the unbalanced dataset and the absence of su cient training samples for di erent classes. A more balanced dataset with large learning samples for each class might produce better results in these instances.