IIIT-Hyderabad at HASOC 2019: Hate Speech Detection

Abstract

Automatic identi cation of o ensive language in various social media platforms especially Twitter poses a great challenge to the AI community. The repercussions of such writings are hazardous to individuals, communities, organizations and nations. The HASOC shared task attempts for automatic detection of abusive language on Twitter in English, German and Hindi languages. As a part of this task, we (team A3-108) submitted di erent machine learning and neural network based models for all the languages. Our best performing model was an ensemble model of SVM, Random Forest and Adaboost classi ers with majority voting.

Keywords

• Machine Learning
• Neural Networks
• Adaboost
• LSTM
• Linear SVM
• Random Forest
• TF-IDF

Introduction

Social media is a great platform to communicate with people from di erent demographic groups. With the exponential rise of use of hand-held devices across the world, people are spending considerable amount of time on social media like Facebook, Twitter, Instagram. Recent studies [ 2 ] suggest that most of the online content generated on these platforms contains di erent forms of abusive language. Cyberbullying and cyberterrorism have become a big menace for the human society. A lot of disparaging tweets [ 9 ] target people based on their color, race, ethnicity, nationality, religion, caste. The administrators of social media have started employing methods to tackle the adversarial e ects of the contents being generated at their ends. HASOC [ 8 ] tries to automatically identify hate speech and hurtful language in 3 di erent languages namely English, German and Hindi. The distribution of di erent labels across tasks for each language in the provided training data is shown table 1. The rst task is a binary classi cation task to identify whether a tweet is o ensive or not. The other two tasks deal with ner categories of hate speech and o ensive posts. Two kinds of approaches were employed for the identi cation tasks. { Machine Learning Techniques { Neural Network Approaches 2.1

Prepossessing

Preprocessing is essential when we are dealing with textual data. For machine learning approaches, we used the spacy 1 tokenizer for English and German, the nltk 2 Twitter tokenizer for tokenizing the input. We also normalized the Twitter handles and hashtags as \USRTOK", and urls as \URLTOK". 2.2

Feature Engineering

We used generic features for the representation of each tweet as we did not want to design any language speci c features. Each word appearing in the tweet was reduced to its lemma for English and German. We did not lemmatize the Hindi words as there was no publicly available spacy model. The features were TF-IDF vectors at character and word levels for all the tasks. We also used length of a tweet as a feature. We experimented with di erent kinds of classi ers. Each one was trained either individually or was a part of an ensemble classi er. Di erent voting procedures were also tried out. In hard voting, majority voting is carried out among the participating classi ers to decide each label. A voting classi er with soft voting selects the maximum from the computed sums of the predicted probabilities for the constituent classi ers. The following were implemented using scikit-learn [ 10 ] machine learning library.

{ Linear SVM { Adaboost or Adaptive Boosting (AB) { Random Forest (RF) { Voting Classi er (VC) We tried various combinations of word and character level n-grams for the classication. By performing grid-search, we observed that combining word unigrams and character n-grams where n f2; 3; 4; 5g TF-IDF vectors as well as the combination of character and word level n-grams.

1 https://spacy.io 2 http://www.nltk.org

2.3

Neural Network Models

We also used LSTM based neural network classi ers for all subtasks. We used word features, Embedding layer (300 dimensions) as the inputs to the LSTM [ 4 ] (128 units) layer, softmax layer (for prediction) in the keras 3 toolkit. In this pipeline, we used categorical crossentropy as the loss function, the Adam optimizer to optimize the parameters [ 5 ] and trained on 50 epochs. In section 3, we show and discuss results in detail. 2.4

Our Submissions

We submitted 3 runs for each task in each language. The 1st run was the LSTM based approach. The 2nd run was an ensemble of SVM, Random Forest and Adaboost classi ers with hard voting. This classi er used TF-IDF features of word unigrams and character 2, 3, 4, 5 grams. The 3rd submission was similar to the 2nd one with an additional feature of length of every tweet. 3

Results

Di erent classi ers were trained to predict the class of each question. We include the top performing system outputs in table 2. Each model and feature set is shown in the table. Two metrics were used to evaluate the systems. Macro F1 score was the primary metric whereas weighted F1 was the secondary one. Macro F1 is an unweighted mean of the metrics calculated for each label. Weighted F1 is obtained by assigning weights based on the number of samples for each true label. 4

Observations

Some tweets are ambiguously labeled.

{ \All the best to #TeamIndia for another swimming competition on Sunday against #Pakistan. #INDvPAK #ShameOnICC #CWC19 #CWC19Rains https://t.co/MG2cIE0zib" { \#ShameOnICC 1. ICC on Dhoni`s gloves Vs 2.ICC planning the World

Cup https://t.co/4kO3zKt7ln" The rst tweet is not o ensive while the second one is marked as o ensive in the training set. Although both of them are related to a similar topic, a classi er trained on these kinds of examples will predict them as non-o ensive as the number of non-o ensive tweets was more. Adaboost [ 3 ] was the best performing classi er among the three classi ers used in our submissions. This was due to its ability to combine multiple weak classi ers to create a strong prediction model. But an ensemble of SVM, Random Forest and Adaboost performed even

3 https://keras.io

English German Hindi 1 2 3 1 2 1 2 3 Language Task# Run# Model better than each classi er. Soft voting performed worse than the technique of hard voting while the nal predictions were made. In Twitter, the number of spelling variations is high due to character constraints. So character n-gram based TF-IDF was superior to its word counterparts individually. When we combined both word and character n-gram, the increase in performance was marginal. Adding length of the tweet as a feature did not improve the model. It is wrong to assume that the tweets containing abusive language are usually short. Machine learning approaches outperformed the neural networks in almost all the tasks. This conforms to the hypothesis that the machine learning techniques are superior their neural network counterparts in a low resource setting. This could be due to the higher number of parameters that deep learning approaches try to learn from a very limited amount of data. We also gured out that the classi ers performed well when the classes were balanced. Predicting profane tweets was di cult as the frequency of such tweets was the least across the data for each language. All our classi ers performed very poorly for all the tasks in German. The systems were unable to capture any form of hate speech. A lexicon containing German slur, profane and abusive words can prove to be useful. Usually an o ensive tweet is full of words portraying negative sentiments. German sentiment lexicons can be looked up to identify such tweets.

We also observed that we missed a lot of important cues when we replaced all Twitter handles and hashtags by a generic token. \#BorisJohnsonShouldNotBePM", \#bloodonhishands", \#TrumpIsATraitor" are made up of multiple words. These words in isolation can be a useful feature for the identi cation task.

From the above results, one can do case by case analysis to nd out the improvement possibilities that can help di erent classi ers on di erent tasks.

We also performed an analysis to understand the complexity of the task by automatically applying Latent Dirichlet Allocation (LDA) on the provided training data. Before applying LDA [ 1 ], by using Gensim toolkit [ 11 ], we performed basic tokenizaion [ 6 ], text normalization and stop-word removal. For this analysis, we focused only on sub-task-1 ( Non Hate-O ensive vs Hate and O ensive) to understand the task di culty on a coarse level. Figures 1, 2 and 3 show these derived text clusters from LDA for English (EN), Hindi (HI), German (DE) respectively. From this clusters, we can argue that they do not give any inherent separation on given task labels.

As a next step to this, we also used learned LDA model of subtask-A or task-1 to visualise training data by plotting each instances using T-SNE [ 7 ] in 2D. Figures 4, 4 and 4, represent the training text and corresponding labels that we got from LDA (left) with blue and orange colors for respective languages . The graph on the right represents the actual labeled data in two dimensions for respective languages. We can observe from gures that the given task of classi cation is quite di cult as simple topic modeling do not provide any major incites for the classi cation. This also suggests the need of feature engineering and use of external resources for further improvement. We presented our supervised approaches for the FIRE task of Hate Speech and O ensive Content Identi cation in Indo-European Languages (HASOC). Our experiments provide strong cues to go for traditional machine learning algorithms with feature engineering instead of recent neural network based approaches when the number of samples is very few and the class distribution is heavily skewed. An ensemble classi er with word and character TF-IDF features performed the best among all the classi ers. Detecting o ensive language in tweets is hard when explicit keywords indicative of such forms are missing e.g \I don`t know how much more I can take! 45 is a compulsive liar! #Trump30Hours #TrumpIsATraitor". We can explore unsupervised techniques on raw tweets for learning a better representation of implicit form of hate speech. Convolutional neural networks (CNN) could be used to model the interactions between character n-grams in the tweets.

References

1. Blei , D.M. , Ng , A.Y. , Jordan , M.I. : Latent dirichlet allocation . Journal of machine Learning research 3(Jan) , 993 { 1022 ( 2003 )

2. Chen , Y. , Zhou , Y. , Zhu , S. , Xu , H. : Detecting o ensive language in social media to protect adolescent online safety . In: 2012 International Conference on Privacy, Security, Risk and Trust and 2012 International Confernece on Social Computing . pp. 71 { 80 . IEEE ( 2012 )

3. Freund , Y. , Schapire , R.E.: A decision-theoretic generalization of on-line learning and an application to boosting . Journal of computer and system sciences 55(1) , 119 { 139 ( 1997 )

4. Hochreiter , S. , Schmidhuber , J.: Long short-term memory . Neural computation 9(8) , 1735 { 1780 ( 1997 )

5. Kingma , D.P. , Ba , J.: Adam: A method for stochastic optimization . arXiv preprint arXiv:1412.6980 ( 2014 )

6. Loper , E. , Bird , S. : Nltk: the natural language toolkit . arXiv preprint cs/0205028 ( 2002 )

7. Maaten , L.v.d., Hinton , G.: Visualizing data using t-sne . Journal of machine learning research 9(Nov) , 2579 { 2605 ( 2008 )

8. Modha , S. , Mandl , T. , Majumder , P. , Patel , D. : Overview of the HASOC track at FIRE 2019: Hate Speech and O ensive Content Identi cation in Indo-European Languages . In: Proceedings of the 11th annual meeting of the Forum for Information Retrieval Evaluation (December 2019 )

9. Nockleby , J.T.: Hate speech . Encyclopedia of the American constitution 3 , 1277{ 1279 ( 2000 )

10. Pedregosa , F. , Varoquaux , G. , Gramfort , A. , Michel , V. , Thirion , B. , Grisel , O. , Blondel , M. , Prettenhofer , P. , Weiss , R. , Dubourg , V. , Vanderplas , J. , Passos , A. , Cournapeau , D. , Brucher , M. , Perrot , M. , Duchesnay , E.: Scikit-learn: Machine learning in Python . Journal of Machine Learning Research 12 , 2825 { 2830 ( 2011 )

11. Rehurek , R. , Sojka , P. : Software Framework for Topic Modelling with Large Corpora . In: Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks . pp. 45 { 50 . ELRA, Valletta, Malta (May 2010 ), http://is.muni.cz/ publication/884893/en