We describe our top-team solution to Task 1 for Hindi in the HASOC contest organised by FIRE 2019. The task is to identify hate speech and o ensive language in Hindi. More speci cally, it is a binary classi cation problem where a system is required to classify tweets into two classes: (a) Hate and O ensive (HOF) and (b) Not Hate or O ensive (NOT). In contrast to the popular idea of pretraining word vectors (a.k.a. word embedding) with a large corpus from a general domain such as Wikipedia, we used a relatively small collection of relevant tweets (i.e. random and sarcasm tweets in Hindi and Hinglish) for pretraining. We trained a Convolutional Neural Network (CNN) on top of the pretrained word vectors. This approach allowed us to be ranked rst for this task out of all teams. Our approach could easily be adapted to other applications where the goal is to predict class of a text when the provided context is limited.
CNN
Deep
The \Hate Speech and O ensive Content Identi cation in Indo-European
Languages" track1 (HASOC) is one of the tracks in FIRE 2019 conference2 [
The training dataset is comprised of 4665 labelled tweets in Hindi. The training dataset is created from Twitter and participants are allowed to use external datasets for this task. In the competition setup, the testing dataset is comprised of 1319 unlabelled tweets that were also created from Twitter. The testing dataset and leaderboard were kept unknown to participants until the results were announced. Competitors had to split the training set to get validation set and use the validation set through the competition to compare models. The testing set was only used at the end of the competition for the nal leaderboard.
Th proposed approach relies on very little feature-engineering and
preprocessing as compared to many existing approaches. Section 2 discusses our
topranked model building approach. It consists of two steps: (a) pretraining word
vectors using a relevant collection of unlabelled tweets and (b) training a
Convolutional Neural Network (CNN) model using the labelled training set on top of
the pretrained word vectors. Section 3 describes other sophisticated alternative
models that we tried. Though these models did not perform as good as
compared to our winning model in this track, their performance provides further
insight into how to use machine learning models for identifying hate speech and
o ensive language in Hindi. Section 4 provides experimental results comparing
and analysing our various models both on testing set and validation set. The
source code of our model can be found online at [
Data Collection
Labelled Contest Dataset The goal of Task 1 for Hindi is to predict the
class (HOF or NOT) of a given tweet written in Hindi. Out of 4665 labelled
tweets in the training set, 2469 (52.92%) are HOF and 2196 (47.07%) are NOT.
We randomly kept 20% of training data for validation set. We used ten cross
validation in the remaining training set for hyper parameter setting.
Unlabelled External Dataset It is a di cult task to separate abusive tweets
from tweets that are sarcastic, joking, or contained abusive keywords in a
nonabusive context [
Preprocessing We de-identi ed person occurrence (e.g. @someone) with xxatp,
url occurence with xxurl, source of modi ed retweet with xxrtm and source of
unmodi ed retweet with xxrtu. We xed the repeating characters (e.g. goooood)
in word and removed common invalid characters (e.g. < br= >, < unk >, @ @,
etc). We used html unescape to replace hexadecimal escape sequences with the
character that it represents. We used multi-language spaCy module4 to
lemmatize words and a lightweight stemmer for Hindi language [
Word Embedding
Embedding models quantify semantic similarities between words based on their
distributional property that a word is characterised by the company it keeps.
These models quantify semantic properties of words by mapping co-occurring
words close to each other in an Euclidean space. Given a sizeable corpus, these
models can e ectively learn a high-quality word embedding from the co-occurrence
of words in the corpus. Word embedding maps each word from the vocabulary
to a vector of real numbers. Mikolov et al. [
In embedding models, a sliding window of a xed size moves along the text of a corpus. For a given position of the sliding window, let the word in the middle is current word wi and the words on its left and right within the sliding window are context words C. The continuous bag-of-words model predicts the current word wi from the surrounding context words C, i.e. p(wijC). In contrast, the skip-gram model uses the current word wi to predict the surrounding context words C, i.e. p(Cjwi). In Figure 1, for example in this corpus, if the current position of a running sliding window contains the phrase tum sirf chutiya kat ti ho. In continuous bag-of-words, the context words ftum, sirf, kat, ti, hog can be used to predict the current word fchutiyag, whereas, in skip-gram, the current word fchutiyag can be used to predict the context words ftum, sirf, kat, ti, hog.
The objective of model training is to nd a word embedding that maximises
p(wijC) or p(Cjwi) over a corpus. In each step of training, each word is either
(a) pulled closer to the words that co-occur with it or (b) pushed away from
all the words that do not co-occur with it. A softmax or approximate softmax
function can be used to achieve this objective [
∑ wi wi ∑ wi-2 wi-1 wi+1 wi+2 S iil d n g W i n d o w w o d n i W g n iil d S
wi-2 wi-1 wi+1 wi+2
Continuous bag-of-words Skip-gram
Fig. 1: Continuous Bag-of-Words and Skip-gram Word Embedding Models [
We use the continuous bag-of-words model in this contest as this model is
faster and has a slightly better accuracy for the words that appear frequently
based on our experimental results. We implemented this model using the module
Word2Vec in Gensim Python library. We set the word vector dimension to 200,
minimum word count to 2, number of iteration in pretraining to 10, sliding
window size to 5 and maximum vocabulary count to 0. We run this model on
the unlabelled external dataset described in Section 2.1 to get the pretrain word
vectors. Our pretrained word vectors and corresponding python code to use them
in classi er are available online at [
Model Architecture
The proposed architecture of our top-ranked model CNN to identify hate speech
and o ensive language in Hindi is given in Figure 2. This is an empirically
customised and regulated version of the architecture that we have used in our prior
work of misogynistic tweets identi cation on Tweeter [
Then, we pass these features to a fully connected hidden layer with 256 perceptrons that use the ReLU activation function. This fully connected hidden layer learns the complex non-linear interactions between the features from the convolution layer and generates 256 higher level new features. Finally, we pass these 256 higher level features to the output layer with single perceptron that uses the sigmoid activation function. The perceptron in output layer generates the probability of the tweet being HOF or NOT.
In this architecture (Figure 2), a proportion of units are randomly
droppedout from each layer except the output. This is done to prevent co-adaptation of
units in a layer and to reduce over tting. We set 50% units droppedout from the
input layer, the lters of size 3 and the fully connected hidden layer based on
best empirical results. Only 20% units are droppedout from the lters of size 4
and 5. Python code for this model is available online at [
could
be
par
tum
sirf
chutiya
kat
ti
ho
{ Long Short-Term Memory Network (LSTM) [
A total of nine machine learning models, including the winning customised CNN model, were trained to identify hate speech and o ensive language in Hindi. We used transfer learning of word vectors for both CNN and LSTM. The word vectors were pre-trained on a collection of relevant tweets and tuned with the training dataset during the model training. 4.1
Results The experimental results comparing models in custom validation set are given in Table 1. The detailed results of the winning CNN model in test dataset are given in Table 2.5 5 In the absence of any other information except the email message about the top-team performance, we are not able to provide the comparative results with other submitted team results. We will update this table with the rest of the team performance, once we receive information from the track organisers. Experimental results in both validation and test set show that CNN outperforms all other models. CNN is able to outperform LSTM and other baseline models because of the speci c nature of tweets. For example, tweets can be super condensed and indirect texts (e.g. satire), may not follow the standard sequence of the language and be full of noise.
Traditional models (e.g. SVM, XGBoost, RF, kNN, etc.) are based on bag-ofwords assumption. The bag-of-words (or bag-of-phrases) representation cannot capture sequences and patterns that are very important to identify hate speech and o ensive contents in tweets. For example, if a tweet ends saying if you know what I mean, there is a high chance that it is an o ensive tweet, even though individual words are innocent.
A LSTM model is popularly used in natural language processing research
because of its e ectiveness of handling sequences in text datasets. Empirical
results in Table 1 show that it performed as a second best model. However,
the sequence in a tweet can be highly impacted by the noise [
We introduce an e ective method for the task of hate speech and o ensive content identi cation in Hindi. We propose a custom CNN architecture built on word vectors pre-trained on a relevant corpus from the task-speci c domain. The proposed model was the top-ranked model in this task under the track. We conducted a series of experiments conducted using state-of-the-art models. Experimental results show that the contexts of hate speech and o ensive content can be captured through transfer learning of word embeddings (a.k.a. word vectors) and those contexts can signi cantly improve the performance of hate speech and o ensive content identi cation. We also observed that when transfer learning through word vectors is utilised, CNN performs better than LSTM because of the noisy nature of tweets. CNN can identify many small and large patterns in a tweet, if some of them gets altered by noise it can still use other patterns to identify the class of the tweet. On the other hand, LSTM uses the sequence of a tweet to identify its class, but noise in the tweet can alter the sequence and make it hard for LSTM to identify the class.