Communication on the internet has become a lot faster than anything in the world through various social media platforms like Facebook, Twitter, Whatsapp, Viber, Telegram and many more. Now, the concern is to check what kind of information and speech is being spread by users so that the social media platforms do not work as hotbeds for hate speech and offensive content. Therefore, a robust automatic filter system is required to sweep away these malicious contents.Hate speech and offensive content ranges over several issues, such as politics, religion, colour, gender, caste, ethnicity, etc. which holds the potential to polarise the society [1]. The benefit of anonymity and fake accounts on social media are major contributing factors for ease in bullying and the spread of hate speech and offensive languages at light speed.
Prominent efforts have been put to develop systems to secure the platforms (distinctively [2], [3], [4], [5], [6], [7], [8] ). In addition to it, many shared tasks are being regularly organised for awareness and to come up with productive outcomes as automatic detection around the context of hate speech, aggression and offensive content [9], [10], [11], [1], [12], [13], [14].
One of its kinds in this horizon is FIRE 2021 shared task on Hate Speech and Offensive Content Detection in Indo-European Languages (HASOC 2021). In this paper, as part of the shared task, we elaborate on automatic hate speech and offensive content identification using SVM based system and its development for both segments of sub-task 1 in two languages -Hindi and English. The remaining portion of the paper is divided into four sections. Section 2 discusses the used corpus size and its types for training and testing. Section 3 gives a detailed sketch of the conducted experiments for this task. Moreover, section 4 delivers the developed system's results and their error analysis with classified types of errors. Eventually, section 5 wraps up with the concluding notes.
In order to direct the experiments for the identification/classification of hate speech and offensive language, we used the annotated twitter corpus for Hindi and English languages which were shared in the FIRE Shared Task HASOC 2021 [15]. An enumeration of the shared task corpus is given in Table 1. The corpus is labelled at two levels and they were presented as two segments in Subtask 1 for Hindi and English [16] given below -1. Subtask 1A: In sub-task 1A, the corpus is annotated as HOF and NOT. HOF stands for hate speech, offensive language, and profane words while NOT is non hate and non-offensive content. Hence it is a binary classification task. 2. Subtask 1B: In Subtask 1B, fine-grained classification is offered for the identification of hate speech and offensive language. If the content is marked HOF in the Subtask 1A then it is marked as Hate Speech (HATE), Offensive (OFFN), and Profanity (PRFN) in this stage. Hence it is a three class classification task.
We mainly experimented with SVMs classifier for Subtask 1A and 1B of Hindi and English corpus. We used the scikit-learn implementation of SVM ( [17], [18] as cited in [1]). Support Vector Machines (SVMs) [19] are one of the most efficient classic machine learning models used for different kinds of text classification tasks. We experimented with binary and three-class problems with our basic objective of exploring the efficiency and productivity of SVMs for the detection of hate speech and offensive content.
In the case of our system, we experimented with SVM for both segments of Subtask 1 with the consecutive sets of features (given below in list 1, 2, and 3) and different C-values (0.001, 0.01, 0.1, 1, 5, 10) for working out the best model. Our classifier's best performances in both languages are given in Table 2 with n-gram features. Selection of these combination of word n-grams and character n-grams is based on best performances of system for Subtask1A and Subtask 1B.
1. Character n-grams features (trigrams to five-grams).
2. Word n-grams features (unigrams, bigrams and trigrams).
3. A systematic combination of diverse character n-grams and word n-grams features.
From the above experiments, we get that given the particular dataset, for English Subtask 1A feature of character five-gram and word trigram with C-value 10 gives the best performance. For Hindi Subtask 1A feature of character four-gram and word bigram with C-value 5 gives the best performance. For English Subtask 1B feature of character four-gram and word trigram with C-value 5 gives the best performance. For Hindi Subtask 1B feature of character four-gram and word unigram with C-value 10 gives the best performance. In the overall judgment of both Subtasks, the combination of character n-grams and word n-grams performed well for Subtask 1A in Hindi and English than Subtask 1B. Though, the score-wise improvement in all sub-tasks was very low for different features. Word n-gram features are widely effective in the case of Subtask 1A, which is binary classification and on the other side of three-class classification these are not very helpful for Subtask 1B.
In the collective results, our system performed best on the test set in Subtask 1A for English in comparison to Hindi (also Subtask 1A) and Subtask 1B for both languages. The macro F1 scores of all segments of Subtask 1 are present in Table 3. Our SVM classifier was placed at the 34th position in English Subtask 1A (the macro F1 score is 8 points below that of the topmost team), while it is placed at the 25th place in English Subtask 1B (macro F1 score being almost 9 points below the best performance team), it is placed at the 29th position in Hindi Subtask 1A (with an overall difference of 7 points in the micro F1 score of the best team) and finally, it is placed at the 15th position in Hindi Subtask 1B (with a difference of almost 11 points in macro F1 score in comparison to the topper team). The performance comparison of our classifier and best classifier in all segments of Subtask 1 are summed up in the Figure 1. Apart from the results of our system for both languages, analysis of predicted errors on test data and its explanations are also most important. It is quite visible that the system we have developed has high precision and low recall in all Subtasks for Hindi and English. In the comparison of predicted labels in sub-task 1A for both languages. Our system predicted the values of HOF class in English Subtask 1A (see Figure 2) are much higher in numbers than the Hindi Subtask 1A (see Figure 3), which is quite opposite for NOT class in both segments of this Subtask. The performance of the system for different classes in different Subtasks is due to the sampling size of training sample data. Here, In Subtask 1A the proportion of both classes (HOF and NOT) were higher individually in English and Hindi.
Likewise, in three-class classification Subtask 1B the system performed well for some classes and predicted PRFN and NONE well in comparison of HATE and OFFN classes in English (see Figure 4). In this Subtask of Hindi (see Figure 5), NONE class is produced adequately good in numbers than the other classes (PRFN, OFFN, and HATE). The earlier trend of the proportion of training sample data is followed here in the case of three-class classification, where the 65% are PRFN and NONE classes of whole proportion in English, which is opposite in Hindi where PRFN is much lesser and OFFN, HATE are subsequent in numbers than NONE class. Another basis of the lower performance of the system in different Subtasks for both languages is the structure and morphological features of both languages, where the structure of Hindi is a little bit complex with a good number of morphological features. In error analysis, some different types of errors are classified on the basis of gold labels and system predicted labels in both languages. These are satire/sarcasm, slogan, coined and aggressive lexical items, idiomatic expressions, quotes, and code-mixed data, etc. Broadly, these error types are predicted in the form of lexical features for all segments of Subtask 1 represented in Table 4.
The paper deals with a detailed description of the S_Cube system which is developed for HASOC at FIRE 2021. Results of the experiment show that SVM extrapolates a cut above for the binary classifier task in Subtask 1A, effective in cases of uneven corpus too, which is far opposite in the case of the three-class classifier. SVM is able to achieve low recall (but high precision) for all Subtasks in both languages. We also observed that, the lower performance in Subtask 1B could be broadly ascribed to the uneven corpus and the lack of ample training sample size for This quote is used for supreme political leader of BJP. HATE -PRFN different classes. The lexical features like satire, slogans, idioms, quotes and code-mixed data are adding to the factor due to which system is producing error. Therefore, a more propped corpus with a substantial learning sample size for each class could give better results in these incidents.