This paper presents the system description of S_cube, which was submitted at the FIRE Shared Task 2021 on Hate Speech and Ofensive Content Identification in Indo-European Languages (HASOC). Our team submitted a system for Subtask 1 in two languages - English and Hindi, which has two diferent segments Subtask 1A and 1B for both languages. We experimented with the classic machine learning using Support Vector Machine (SVM). We discuss the system and its results with main findings for hate speech and ofensive content identification in this paper. Our model achieves an F1 Score of 0.7563 at English Subtask 1A while the performance is worse for Hindi Subtask 1B (0.7195 F1).
Communication on the internet has become a lot faster than anything in the world through
various social media platforms like Facebook, Twitter, Whatsapp, Viber, Telegram and many
more. Now, the concern is to check what kind of information and speech is being spread by
users so that the social media platforms do not work as hotbeds for hate speech and ofensive
content. Therefore, a robust automatic filter system is required to sweep away these malicious
contents.Hate speech and ofensive content ranges over several issues, such as politics, religion,
colour, gender, caste, ethnicity, etc. which holds the potential to polarise the society [
Prominent eforts have been put to develop systems to secure the platforms (distinctively
[
One of its kinds in this horizon is FIRE 2021 shared task on Hate Speech and Ofensive Content Detection in Indo-European Languages (HASOC 2021). In this paper, as part of the shared task, we elaborate on automatic hate speech and ofensive content identification using SVM based system and its development for both segments of sub-task 1 in two languages - Hindi and English.
Train Sub-task 1B HATE OFFN PRFN 683 622 1,196 566 654 213
Test Set TOTAL 1,281 1,532
The remaining portion of the paper is divided into four sections. Section 2 discusses the used corpus size and its types for training and testing. Section 3 gives a detailed sketch of the conducted experiments for this task. Moreover, section 4 delivers the developed system’s results and their error analysis with classified types of errors. Eventually, section 5 wraps up with the concluding notes.
In order to direct the experiments for the identification/classification of hate speech and ofensive language, we used the annotated twitter corpus for Hindi and English languages which were shared in the FIRE Shared Task HASOC 2021 [15]. An enumeration of the shared task corpus is given in Table 1. The corpus is labelled at two levels and they were presented as two segments in Subtask 1 for Hindi and English [16] given below 1. Subtask 1A: In sub-task 1A, the corpus is annotated as HOF and NOT. HOF stands for hate speech, ofensive language, and profane words while NOT is non hate and non-ofensive content. Hence it is a binary classification task. 2. Subtask 1B: In Subtask 1B, fine-grained classification is ofered for the identification of hate speech and ofensive language. If the content is marked HOF in the Subtask 1A then it is marked as Hate Speech (HATE), Ofensive (OFFN), and Profanity (PRFN) in this stage.
Hence it is a three class classification task.
We mainly experimented with SVMs classifier for Subtask 1A and 1B of Hindi and English
corpus. We used the scikit-learn implementation of SVM ([17], [18] as cited in [
In the case of our system, we experimented with SVM for both segments of Subtask 1 with the consecutive sets of features (given below in list 1, 2, and 3) and diferent C-values (0.001, 0.01, 0.1, 1, 5, 10) for working out the best model. Our classifier’s best performances in both languages are given in Table 2 with n-gram features. Selection of these combination of word n-grams and character n-grams is based on best performances of system for Subtask1A and Subtask 1B.
1. Character n-grams features (trigrams to five-grams). 2. Word n-grams features (unigrams, bigrams and trigrams). 3. A systematic combination of diverse character n-grams and word n-grams features.
From the above experiments, we get that given the particular dataset, for English Subtask 1A feature of character five-gram and word trigram with C-value 10 gives the best performance. For Hindi Subtask 1A feature of character four-gram and word bigram with C-value 5 gives the best performance. For English Subtask 1B feature of character four-gram and word trigram with C-value 5 gives the best performance. For Hindi Subtask 1B feature of character four-gram and word unigram with C-value 10 gives the best performance. In the overall judgment of both Subtasks, the combination of character n-grams and word n-grams performed well for Subtask 1A in Hindi and English than Subtask 1B. Though, the score-wise improvement in all sub-tasks was very low for diferent features. Word n-gram features are widely efective in the case of Subtask 1A, which is binary classification and on the other side of three-class classification these are not very helpful for Subtask 1B.
In the collective results, our system performed best on the test set in Subtask 1A for English in comparison to Hindi (also Subtask 1A) and Subtask 1B for both languages. The macro F1 scores of all segments of Subtask 1 are present in Table 3.
Our SVM classifier was placed at the 34th position in English Subtask 1A (the macro F1 score is 8 points below that of the topmost team), while it is placed at the 25th place in English Subtask 1B (macro F1 score being almost 9 points below the best performance team), it is placed at the 29th position in Hindi Subtask 1A (with an overall diference of 7 points in the micro F1 score of the best team) and finally, it is placed at the 15th position in Hindi Subtask 1B (with a diference of almost 11 points in macro F1 score in comparison to the topper team). The performance comparison of our classifier and best classifier in all segments of Subtask 1 are summed up in the Figure 1.
Apart from the results of our system for both languages, analysis of predicted errors on test data and its explanations are also most important. It is quite visible that the system we have developed has high precision and low recall in all Subtasks for Hindi and English.
In the comparison of predicted labels in sub-task 1A for both languages. Our system predicted the values of HOF class in English Subtask 1A (see Figure 2) are much higher in numbers than the Hindi Subtask 1A (see Figure 3), which is quite opposite for NOT class in both segments of this Subtask. The performance of the system for diferent classes in diferent Subtasks is due to the sampling size of training sample data. Here, In Subtask 1A the proportion of both classes (HOF and NOT) were higher individually in English and Hindi.
Likewise, in three-class classification Subtask 1B the system performed well for some classes and predicted PRFN and NONE well in comparison of HATE and OFFN classes in English (see Figure 4). In this Subtask of Hindi (see Figure 5), NONE class is produced adequately good in numbers than the other classes (PRFN, OFFN, and HATE). The earlier trend of the proportion of training sample data is followed here in the case of three-class classification, where the 65% are PRFN and NONE classes of whole proportion in English, which is opposite in Hindi where PRFN is much lesser and OFFN, HATE are subsequent in numbers than NONE class. Another basis of the lower performance of the system in diferent Subtasks for both languages is the structure and morphological features of both languages, where the structure of Hindi is a little bit complex with a good number of morphological features.
In error analysis, some diferent types of errors are classified on the basis of gold labels and system predicted labels in both languages. These are satire/sarcasm, slogan, coined and aggressive lexical items, idiomatic expressions, quotes, and code-mixed data, etc. Broadly, these error types are predicted in the form of lexical features for all segments of Subtask 1 represented in Table 4.
The paper deals with a detailed description of the S_Cube system which is developed for HASOC at FIRE 2021. Results of the experiment show that SVM extrapolates a cut above for the binary classifier task in Subtask 1A, efective in cases of uneven corpus too, which is far opposite in the case of the three-class classifier. SVM is able to achieve low recall (but high precision) for all Subtasks in both languages. We also observed that, the lower performance in Subtask 1B could be broadly ascribed to the uneven corpus and the lack of ample training sample size for Hindi & English Examples
Translation & Explaination 1. Vodafone had raised a dog, it became very famous then Mukesh Ambani was fond of it.
The government is silent, the public is upset 1. People of BJP (Bhartiya Janta Party)1 are Coward. 2. We want new government in this may not like this government who makes false promises
Gold Label Predicted Label NOT - HOF, OFFN - HATE NONE - HATE HOF - NOT, HATE - NONE diferent classes. The lexical features like satire, slogans, idioms, quotes and code-mixed data are adding to the factor due to which system is producing error. Therefore, a more propped corpus with a substantial learning sample size for each class could give better results in these incidents. english and german (2020). [15] S. Modha, T. Mandl, G. K. Shahi, H. Madhu, S. Satapara, T. Ranasinghe, M. Zampieri, Overview of the HASOC Subtrack at FIRE 2021: Hate Speech and Ofensive Content Identification in English and Indo-Aryan Languages and Conversational Hate Speech, in: FIRE 2021: Forum for Information Retrieval Evaluation, Virtual Event, 13th-17th December 2021, ACM, 2021. [16] T. Mandl, S. Modha, G. K. Shahi, H. Madhu, S. Satapara, P. Majumder, J. Schäfer, T. Ranasinghe, M. Zampieri, D. Nandini, A. K. Jaiswal, Overview of the HASOC subtrack at FIRE 2021: Hate Speech and Ofensive Content Identification in English and Indo-Aryan Languages, in: Working Notes of FIRE 2021 - Forum for Information Retrieval Evaluation, CEUR, 2021. URL: http://ceur-ws.org/. [17] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, E. Duchesnay, Scikit-learn: Machine learning in Python, Journal of Machine Learning Research 12 (2011) 2825–2830. [18] L. Buitinck, G. Louppe, M. Blondel, F. Pedregosa, A. Mueller, O. Grisel, V. Niculae, P. Prettenhofer, A. Gramfort, J. Grobler, R. Layton, J. VanderPlas, A. Joly, B. Holt, G. Varoquaux, API design for machine learning software: experiences from the scikit-learn project, in: ECML PKDD Workshop: Languages for Data Mining and Machine Learning, 2013, pp. 108–122. [19] M. A. Hearst, Support vector machines, IEEE Intelligent Systems 13 (1998) 18–28. URL: http://dx.doi.org/10.1109/5254.708428. doi:10.1109/5254.708428.