IRLab@IITBHU at HASOC 2019: Traditional Machine Learning for Hate Speech and O ensive Content Identi cation

Abstract

In this paper, the results obtained from the Support Vector Machine, XGBoost method by IRLab@IIT(BHU) on HASOC shared task-organized at FIRE-2019 are reported. The HASOC shared task has three subtasks, namely Hate speech identi cation, O ensive language identi cation and Fine-grained classi cation for the English, Hindi and German languages. The best result for English is obtained after applying Support Vector Machine, XGBoost with a frequency-based feature for hate speech and o ensive content identi cation.

Keywords

• O ensive
• Hate Speech
• Language
• Social Media

INTRODUCTION

Hate speech is a type of communication of verbal expression to attack a human or group based on characteristics such as caste, religion, ethnic origin, sexual orientation, disability or gender [ 7 ]. Hate speech and o ensive content in IndoEuropean languages have become a common phenomenon in the social media. Recent years have seen the spread of o ensive language on social media platforms such as Facebook and Twitter. With the freedom of privilege of expression granted to social media users, it became easy to spread disrespect or hatred against individuals or groups. Automated hate language and o ensive content detection systems may contain the spread inhibition of toxin textual material. Beyond psychological harm, such toxic online content can give rise to real hate crimes [ 10 ]; which justi es the need to automatically detect abusive language and o ensive content shared on social media platforms. 2

RELATED

WORK

Over the last few years, several studies on hate speech and o ensive content identi cation have been published. The literature has explored di erent o ensive and abusive language identi cation problems ranging from aggression to cyber bullying, hate speech, poisonous comments and o ensive language. We brie y discuss each of them in this section. 2.1

Aggressive content identi cation

The rst shared task on aggression identi cation is Trolling, Aggression and Cyberbullying (TRAC-1) at COLING 2018. In this task Aggressive Language Identi cation on Facebook and Twitter data targeted using word embeddings and sentiment features [ 13 ]. Moreover, the best result were obtained through sentiment features with Random Forest (RF) and Support Vector Machine (SVM) with 0.5830, 0.5074 accuracies, respectively. Later, e orts went to develop a classi er that could discriminate between overly aggressive, hidden aggressive, and non-aggressive text. Long short-term memory (LSTM), Convolutional Neural Network (CNN)-LSTM, Bidirectional LSTM with Glove embeddings, the combination of the Passive-Aggressive (PA) and SVM classi ers with characterbased n-gram where n is from 1 to 5, TF-IDF as feature representation were used for aggression identi cation. The best system explained above to achieve a weighted F-score of 0.64 on the Facebook test set entitled as English and Hindi, and the best scores for the surprise set were 0.60 and 0.50 for Hindi and English respectively [ 8, 16, 1, 18, 19 ]. 2.2

Bullying content identi cation

Bullying, also known as peer victimization, has been recognized as a serious national health issue by the White House (2011), the American Academy of Pediatrics (2009), and the American Psychological Association (2004) [ 5, 2, 6 ]. The growing research into cyberbullying in online social networks have catalyzed by the widespread and profound consequences of abuse. Earlier research works on automatic cyberbullying detection have mainly focused on using (sophisticated) text-based methods [ 4, 12, 15 ]. Expanded the text-based identi cation approach to model the use of hashtags, simultaneously with the emotions the spatio-temporal cyberbullying measures to understand and explore. [ 17 ]. 2.3

Hate speech identi cation

Hate speech is a statement of intent to o end another and use cruel or abusive language based on actual or perceived membership to another group [ 3 ]. Established a lexical baseline for discriminating between profane and hate speech on the standard dataset this is the main aim of the paper [ 9 ]. The authors adopted a linear support vector machine classi er with three groups of extracted features for these tests: word skip-grams, surface n-gram and Brown cluster. 2.4

O ensive language identi cation

User-generated content on social media platforms such as Twitter often includes a high level of rude, o ensive or sometimes hateful language [ 20 ]. Increasing vulgarity in online discussions and user comment sections have recently been discussed as relevant issues in society as well as in science [ 14 ], and identi ed o ensive tweets with an accuracy of 83.14 %, f1-score 0.7565 on the real test data for the classi cation of o ensive vs non-o ensive. 3

METHODOLOGY

In this paper, we focus on hate, o ensive, and profane exclusively, for English. We participated in the competition using the team name IRLAB@IITBHU. Figure 1 shows the methodology of the paper. 3.1

Data

The dataset was created from Twitter, Facebook and distributed in tab-separated format. We have participated for all three sub-tasks of English language [ 11 ]. The size of training and testing data is 5852 and 1153 posts for the English language, respectively. In Sub-task A the HOF containing Hate, o ensive, and profane posts are 288, and NOT not containing any Hate speech, o ensive content posts are 865. In Sub-task B, HATE Hate speech posts are 124, and NONE posts are 865. O ensive posts are 71, and Profane posts are 93. In Sub-task C, NONE posts are 865, and TIN (Targeted Insult) posts are 245, and UNT (Untargeted) posts are 43. 3.2

Pre-processing

First the data were cleaned using the tweet preprocessing library4. We got the cleaned data after removing the Retweets Symbols (RT), Hashtag, URL's, Twitter Mentions, Emoji's and Smileys. The preprocessed data also exclude the English stop words (available in NLTK5) while tokenizing the sentences for the extraction of frequency-based feature extraction. The Hate speech, O ensive and Profane have been predicted through TF-IDF feature. 3.3

Classi er

We use two machine learning classi ers Support Vector Machine (SVM) and XGBoost (XGB) classifying for classi cation of Hate speech, O ensive and Profane. The input for both the classi er is in the form of TF-IDF feature matrix and output is a label for the categorical result. Both the classi ers give a di erent score, as classi ers have di erent specialities. 4 https://pypi.org/project/tweet-preprocessor/ 5 https://www.nltk.org/ 4

RESULTS

We start by investigating the accuracy of our TF-IDF features based on machine learning method for this task. We rst train the classi er, with each of them using a type of TF-IDF feature. The results of these experiments are listed in Table 1 and Table 2. In Sub-task A, accuracy of XGBoost is 81% better as compared to SVM 73%. The Sub-task B and Sub-task C accuracy is 80% the same for the XGBoost.

CONCLUSION

In this paper we used text classi cation techniques to recognise among hate speech, profane and o ensive posts. As a baseline we use XGBoost and a SVM classi er. The best result was achieved by XGBoost achieving 81% accuracy. The results displayed in this paper showed that identi cation of profanity from abusive language is a very challenging task.

References

1. Aroyehun , S.T. , Gelbukh , A. : Aggression detection in social media: Using deep neural networks, data augmentation, and pseudo labeling . In: Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC-2018) . pp. 90 { 97 ( 2018 )

2. Association , A.P. , et al.: Apa resolution on bullying among children and youth . Washington, DC: American Psychological Association pp. 1 { 4 ( 2004 )

3. Britannica , E.: Britannica academic. Encyclop dia Britannica Inc ( 2015 )

4. Dinakar , K. , Jones , B. , Havasi , C. , Lieberman , H. , Picard , R. : Common sense reasoning for detection, prevention, and mitigation of cyberbullying . ACM Transactions on Interactive Intelligent Systems (TiiS) 2 ( 3 ), 18 ( 2012 )

5. House , W.: Background on white house conference on bullying prevention ( 2011 )

6. Committee on Injury, V. , Prevention , P. , et al.: Role of the pediatrician in youth violence prevention . Pediatrics 124 ( 1 ), 393 { 402 ( 2009 )

7. Johnson , N., Leahy , R. , Restrepo , N.J. , Velasquez , N. , Zheng , M. , Manrique , P. , Devkota , P. , Wuchty , S.: Hidden resilience and adaptive dynamics of the global online hate ecology . Nature pp. 1 { 5 ( 2019 )

8. Kumar , R. , Ojha , A.K. , Malmasi , S. , Zampieri , M. : Benchmarking aggression identi cation in social media . In: Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC-2018) . pp. 1 { 11 ( 2018 )

9. Malmasi , S. , Zampieri , M. : Detecting hate speech in social media . arXiv preprint arXiv:1712.06427 ( 2017 )

10. Matsuda , M.J.: Public response to racist speech: Considering the victim's story . In: Words That Wound , pp. 17 { 51 . Routledge ( 2018 )

11. Modha , S. , Mandl , T. , Majumder , P. , Patel , D. : Overview of the HASOC track at FIRE 2019: Hate Speech and O ensive Content Identi cation in Indo-European Languages . In: Proceedings of the 11th annual meeting of the Forum for Information Retrieval Evaluation ( 2019 )

12. Nahar , V. , Li , X. , Pang , C. , Zhang , Y.: Cyberbullying detection based on textstream classi cation . In: The 11th Australasian Data Mining Conference (AusDM 2013 ) ( 2013 )

13. Orasan , C. : Aggressive language identi cation using word embeddings and sentiment features . In: Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC-2018) . pp. 113 { 119 ( 2018 )

14. Ramakrishnan , M. , Zadrozny , W. , Tabari , N.: UVA wahoos at SemEval-2019 task 6: Hate speech identi cation using ensemble machine learning . In: Proceedings of the 13th International Workshop on Semantic Evaluation . pp. 806 { 811 . Association for Computational Linguistics, Minneapolis, Minnesota, USA (Jun 2019 ). https://doi.org/10.18653/v1/ S19 -2141, https://www.aclweb.org/anthology/S19- 2141

15. Reynolds , K. , Kontostathis , A. , Edwards , L.: Using machine learning to detect cyberbullying . In: 2011 10th International Conference on Machine learning and applications and workshops . vol. 2 , pp. 241 { 244 . IEEE ( 2011 )

16. Risch , J. , Krestel , R.: Aggression identi cation using deep learning and data augmentation . In: Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC-2018) . pp. 150 { 158 ( 2018 )

17. Sui , J.: Understanding and ghting bullying with machine learning . Ph.D. thesis, Ph. D. dissertation, The Univ. of Wisconsin-Madison , WI , USA ( 2015 )

18. Wiegand , M. , Siegel , M. , Ruppenhofer , J.: Overview of the germeval 2018 shared task on the identi cation of o ensive language ( 2018 )

19. Zampieri , M. , Malmasi , S. , Nakov , P. , Rosenthal , S. , Farra , N. , Kumar , R.: Predicting the Type and Target of O ensive Posts in Social Media . In: Proceedings of NAACL ( 2019 )

20. Zampieri , M. , Malmasi , S. , Nakov , P. , Rosenthal , S. , Farra , N. , Kumar , R.: Semeval-2019 task 6: Identifying and categorizing o ensive language in social media (o enseval) . In: Proceedings of the 13th International Workshop on Semantic Evaluation . pp. 75 { 86 ( 2019 )