DA Master at HASOC 2019: Identi cation of Hate Speech using Machine Learning and Deep Learning approaches for social media post

Abstract

This paper describes the research that our team, DA Master, did on the shared task HASOC, conducted by FIRE-2019, which involves identi cation of hate and o ensive language in Twitter and Facebook posts. The task is divided into three sub-tasks. Our team conducted our experiments on an English dataset. We employed Machine Learning techniques like Logistic Regression and Navies Bayes classi er and a Deep Learning based approach which utilizes Convolutional Neural Networks. Our best model obtained Macro F1 score of 0.6472 for SubTask-A, 0.4068 for SubTask-B and 0.4303 for SubTask-C.

Introduction

In the past few years, there has been an exponential rise in the number of people who have uninhibited access to the internet. The arrival of the "Netizen" populace, and their a nity for interacting via social media platforms like Twitter, Facebook, Instagram etc. has led to the manifestation of diverse online user behaviour. Sometimes the online behaviour exhibited by individuals on social media platforms is hostile, targeting an individual(s) or a community by posting insults or threats. Studies have shown that hate/o ensive messages are becoming extremely common on social media platforms, accounting for almost 500 million posts daily on Twitter1. Thus, the problem of detecting and sti ing the propagation of hateful or o ensive posts on social media is a pertinent issue for research.

To promote research in this eld several competitions have been organized, such as O enseEval[ 1 ], GermanEval[ 2 ]. Recently, HASOC [ 3 ] was organized as a shared task for FIRE 2019. The task is aimed at identifying hateful and o ensive language in social media posts. The task was organized for three languages namely English, Hindi and German, but our team conducted our investigation only on the English dataset.

Organizers proposed a hierarchical model which consists of three sub-tasks: { TASK-A: Identi cation of hate and o ensive language, posts are classi ed as follows:

Hate and O ensive(HOF)

Not Hate and O ensive(NOT) { TASK-B: Categorization of Hate and O ensive, posts identi ed as HOF in task-A are further categorized as:

Hate Speech(HATE) O ensive(OFFN)

Profane(PRFN) { TASK-C: Identi cation of target,in which posts identi ed as HOF in task-A are further classi ed on the basis of whether the post targets a particular individual/group or not.

Targeted Insult(TIN)

Untargeted Insult(UNT)

The rest of paper is organized as follows. In Section 2 we discuss related works and the dataset is discussed in Section 3. Proposed methods are presented in Section 4 and the results in Section 5. Finally, we conclude our work in Section 6. 2

Related Work

The past few years have seen an increase in aggressive user behaviour on social media platforms, usually exhibited via the use of targeted or untargeted hateful and o ensive language. This has encouraged NLP researchers to work in this eld and develop automatic detection systems to preemptively discard such posts. The term hate speech was coined by [ 4 ]. Hate speech detection proposed by [ 5 ] used a Convolutional Neural Network with Word2Vec for classifcation. Schmidt and Wiegand [ 6 ] gave a detailed survey on hate speech detection using Natural Language Processing. They identi ed various features for hate speech like words, sentiment, linguistic, etc. After performing various experiments they observed that, bag of words or word embedding based representation gave better classi cation results, a character-level approach works better than token-level, sometimes the text may not contain hate speech but it may be accompanied by images or other media which might contain hateful messages.

Recent work on the topic by [ 7 ] has focused on the type and target of the o ensive language and was the rst to categorically de ne o ensive language. They created a dataset based on these distinctions which was used to train models utilizing SVM, BiLSTM and CNN. 3

Dataset

The datasets provided by the organizers [ 3 ] were annotated in a hierarchical fashion. Table 1 shows detailed analysis of the provided datasets. It can be observed from Table 1 that the train data for TASK-A consists of 61% Non Hate posts and 39% were classi ed as hateful, o ensive or profane, whereas TASK-B has a distribution of approximately 50% Hate , 30% O ensive and 20% Profane posts, and more than 90% of the TASK-C dataset are posts which are targeted insults and the remaining are untargeted insults.

Therefore, only 39% of the given data can be used for training further tasks. There is a huge imbalance in TASK-C data.

The authors have provided 1153 posts for testing, which were to be lled based on a prediction of model. 4

Proposed Method

For our experiment we used two approaches. For the rst approach we represented the input features using CountVectorizer and T df, which were then used to train logistic regression and Naive-Bayes classi ers. The models used were imported from the Scikit-learn Python package.

Our second implementation took inspiration from [ 8 ]. Our model used a hierarchical CNN i.e 3 layers of 1D convolution. The words were represented using pre-trained GloVe [ 9 ] 200d vectors. No data pre-processing was performed by in any of the experiments. 5

Results 6

Conclusion

The organizers had allowed a maximum of 3 submission for each sub-tasks. Macro F1 Score was the primary metric for evaluation. Table 2 shows results obtained by our methods and that of top performed of each task. Figure 1, 2, 3 show confusion matrix of our best performing system for all three sub-tasks. In this paper we have presented two approaches to solve the hierarchical task of detecting and classifying hate speech and o ensive posts. The accuracy of our methods did not cross 70% for any of the sub-tasks. The methods used did not perform well when for the sub-task B, a possible reason for it may have been that, the model could not di erentiate between profane, hateful and o ensive posts properly. Our future experiments will try to solve the problem of class imbalance by using data augmentation and explore the e ciency of pre-trained language models like ULMFit, XLNet, RoBERTa for tackling this hierarchical task.

References

[1] Marcos Zampieri , Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar . Semeval -2019 task 6: Identifying and categorizing o ensive language in social media (o enseval) . In Proceedings of the 13th International Workshop on Semantic Evaluation , pages 75 { 86 , 2019 .

[2] Michael Wiegand , Melanie Siegel, and Josef Ruppenhofer . Overview of the germeval 2018 shared task on the identi cation of o ensive language . 2018 .

[3] Sandip Modha , Thomas Mandl, Prasenjit Majumder, and Daksh Patel . Overview of the HASOC track at FIRE 2019: Hate Speech and O ensive Content Identi cation in Indo-European Languages . In Proceedings of the 11th annual meeting of the Forum for Information Retrieval Evaluation , December 2019 .

[4] William Warner and Julia Hirschberg . Detecting hate speech on the world wide web . In Proceedings of the Second Workshop on Language in Social Media , pages 19 { 26 . Association for Computational Linguistics, 2012 . URL https://www.aclweb.org/anthology/W12-2103.

[5] Bj orn Gamback and Utpal Kumar Sikdar . Using convolutional neural networks to classify hate-speech . In Proceedings of the First Workshop on Abusive Language Online , pages 85 { 90 . Association for Computational Linguistics, 2017 . https://doi.org/10.18653/v1/ W17 -3013. URL https: //www.aclweb.org/anthology/W17-3013.

[6] Anna Schmidt and Michael Wiegand . A survey on hate speech detection using natural language processing . In Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media , pages 1 { 10 , Valencia , Spain, April 2017 . Association for Computational Linguistics . https://doi.org/10.18653/v1/ W17 -1101. URL https://www.aclweb. org/anthology/W17-1101.

[7] Marcos Zampieri , Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar . Predicting the type and target of o ensive posts in social media . pages 1415{1420 , 01 2019 . https://doi.org/10.18653/v1/ N19 - 1144.

[8] Kim Yoon . Convolutional neural networks for sentence classi cation . In Empirical Methods in Natural Language Processing (EMNLP) , pages 1746 { 1751 , 2014 . URL http://aclweb.org/anthology/D/D14/D14-1181.pdf.

[9] Je rey Pennington , Richard Socher, and Christopher D. Manning . Glove: Global vectors for word representation . In Empirical Methods in Natural Language Processing (EMNLP) , pages 1532 { 1543 , 2014 . URL http://www. aclweb.org/anthology/D14-1162.