With the development of the Internet, the Web has become an information dissemination platform, an information ampli er, and a new social media. The information load and participation of the Internet far exceeds the existing traditional media, and various problems have emerged. There has been signi cant work in several languages in particular for English. However, there is a lack of research in this recent and relevant topic for most other languages. This track intends to develop data and evaluation resources for several languages. The objectives are to stimulate research for these languages and to nd out the quality of hate speech detection technology in other languages. The paper mainly describes the organization of the HASOC 2019 Task, a Shared Task on Hate Speech and O ensive Content Identi cation in Indo-European Languages. The task is organized in three related classi cation subtasks: subtask A is a coarse-grained binary classi cation to identify hate speech and o ensive language, a ne-grained classi cation subtask B is to further classify the data from the subtask A into three categories, and subtask C will check the type of o ense. This paper mainly focuses on English offensive language detection and shows the experimental result in subtask A and subtask B.
With the popularity of Internet applications and the convenience of free speech,
a lot of hate speech and other o ensive content on the Internet pose a huge
threat to the stability of society. The online communication platform has no
strict scrutiny of speech and post, making a variety of o ensive language, such
as insulting, harmful, derogatory or obscene, freely and quickly transmitted from
person to person, and can have an in uence on people's views and social trends
[
The main structure of the rest of this paper has been organized as follows. Section 2 will mainly describe recent researches about hate speech detection. Then the datasets released by HASOC to the participants for training and testing the systems will be introduced in section 3. Section 4 presents the two subtasks and the measures we exploited in the evaluation. Section 5 reports on approaches in the experiment and results of the system. Finally a conclusion will be given in section 6. 2
Existing works on hate speech has been very limited, largely due to a lack of
a general de nition of hate speech, a lack of analysis of its demographic
impact, and a lack of surveys of the most e ective characteristics [
Related researches on hate speech detection has been developing only in
recent a few years. The existing technology used in hate speech detection in
social media is mainly about Dictionaries and lexicons [
The most common method found in the work of [
In this paper, we choose to use English dataset to mainly address Subtask A and Subtask B, where di erent feature extraction methods (N-gram and word embedding) and classi cation algorithms (Logistic Regression and LSTM) will be implemented in this experiment. 3 3.1
The training dataset provided by HASOC is created mainly from the Twitter
and Facebook in English. It is raw data with text ID number, post content and
di erent class labels for three subtasks. The external dataset used the public
Twitter search API to collect the entire corpus, ltering for tweets not written
in English [
The size of English training data corpus is 5852 posts and the external dataset has 39292 texts. The size of test dataset has 1153 posts. The following Table 1 shows the details of three datasets we used. hasoc en 2, @politico No. We should remember very clearly that Individual1 just admitted to treason. TrumpIsATraitor McCainsAHero JohnMcCainDay, HOF, HATE, TIN
where the text has been classi ed by the annotators as hateful-o ensive, hateful, and insulting to an individual, group, or others. 4.1
Subtask A is a coarse-grained binary classi cation task to make Hate speech and
O ensive language identi cation [
For the class of this subtask, there is two labels: HOF and NOT. The label HOF means it contains any form of non-acceptable language such as hate speech, aggression, profanity otherwise NOT. 4.2
Subtask B is a ne-grained multi-level classi cation task to further identify three
classes: HATE, OFFN and PRFN. There are four annotations, where most of
posts is classi ed to OTHER, some to be HATE and the other two categories to
be relatively less. Dubious cases, which are di cult to decide even for humans,
will be left out [
In the result of binary classi cation, there are four di erent situations, namely
true positives (TP), false positives (FP), true negatives (TN) and false negatives
(FN). Based on the results of manual annotations, there are four commonly used
indicators to measure the performance of the classi er, namely accuracy, recall
and F1 scores [
{ Precision positive predictive value: it is a consistent result between manual and automatic classi cation. { Recall sensitivity: it shows the proportion of all positive cases, which is a measure of the ability of the classi er to identify positive samples.
P recision =
T P
T P + F P Recall =
T P T P + F N (1) (2)
In this report, the evaluation measures are the same for both subtask A and
subtask B. To provide a metric that is independent of class size, the classi cation
result will be mainly computed by macro-averaged F1 score and weighted F1
score [
It is rstly calculated for each category of indicator values, and then for the arithmetic mean of all categories.
M acroP =
n M acroR =
n M acroF = 2
M acroP M acroR
M acroP + M acroR { Weighted F1 score
It is rstly calculated for each label and then averaged by support weighting - the actual number of instances per label. (3) (4) (5) (6) W eightedP =
P T P P T P + P F P
W eightedR =
P T P
P T P + P F N W eightedF = 2
W eightedP W eightedR
W eightedP + W eightedR 5
The hate speech detection system is implemented in the process of four parts, namely text preprocessing, fearture extraction, classi er building, and classi cation. Then the classi cation results from di erent models will be analyzed. 5.1
Text preprocessing The text usually contains a lot of meaningless or unaffected information that may a ect research results at di erent stages, such as punctuation, common words, links, and numbers. In this step, regular expressions have been used to eliminate noise, including non-alphanumeric characters and numbers. And we remove text information noise like stopwords as well, which are probably of little value in hate speech detection later. The stop words list in NLTK corpus has been chosen to delete meaningless words in texts. Besides, the post in social media can commonly include many non-point content, such as the mention to user, speci c topic and URL links. So these contents is replaced by the corresponding words, namely USER, TOPIC and URL.
Feature extraction Before training the model, it is necessary to convert the text to various feature vectors because the preprocessed text cannot be directly recognized by the model. In this step, I mainly consider trying to use two common features: n-gram and word embedding features. They can be compared according to the nal results generated by the classi er. { N-gram feature
I mainly focus on unigram features, and then select bag-of-words (BOW) model for n-gram feature notation. It is fairly straighforward and each element demonstrates how often the term appears in a text sentence. Since the information of the low frequency words is more abundant, I use the Term Frequency-Inverse Document Frequency (TF-IDF) algorithm to convert the frequency into the weight of the word, which is a quite robust and accurate weight calculation method. Feature processing can be done through CountVectorizer() and T dfTransformer() in the NLTK package. { Word embeddings feature
The vector dimension is low and dense, and the information density is high in word embedding. We will use Word2Vec model, where the similarity between words can be directly re ected by the calculation of word vectors. In order to create the Word2Vec feature, it is decided to use the pre-trained Google Word2Vec model, which has more convincing information about word similarity. Then the similarity value of each word will be the average of its word embedded value in the Word2Vec feature vector.
Classi er building The whole experimental process is able to be achieved by NLTK, scikit-learn and the Keras system. The same two feature will be the input feature data for di erent classi ers for subtask A and subtask B. { Subtask A
For both n-gram feature and word embedding feature, traditional machine learning algorithm Logistic Regression (LR) and deep learning sequential algorithm Long Short-Term Memory(LSTM) are respectively implemented as a binary classi er. { Subtask B
The One-vs-all classi er will be built by using LR and LSTM for both two extracted features.
Classifcation There will be four di erent classi cation experiments. Three of them will implement experiments on HASOC test data initially given by HASOC organizers, where Logistic Regression classi er will input TF-IDF and Word2Vec features respectively, and LSTM model will utilize Word2Vec feature. The other experiment is implemented by providing HASOC organizers with our LSTM model with Word2Vec feature, which the nal result is based on a new test dataset used privately by HASOC organizers. HASOC test will show the nal F1 score result from HASOC organizers. 5.2
The feature dimensionality of one-hot representation is rather high, which is easy to lead to a poor training model. So the LR classi er with TF-IDF feature is considered as the baseline for comparison. { Subtask A
Our result is ranked in 9th position, seemingly a good score. There is not a big di erence between two F1 scores. The LSTM classi er with word embedding features has the best performance.
It can be seen that the LR classi er using pre-trained word embedding model do not work much better than the classi er using TF-IDF feature, which may be because the Google pre-trained word embedding model is based on the eld of news instead of Twitter. And because of this, the performance of the LSTM model is not particularly good in the nal result. 6
The spread of hate speech on social media has increased signi cantly in recent years, which could have a serious e ect on the society. Therefore, our work makes several contributions according to this problem. First, we try several methods classifying hate speech using both traditional machine learning model like LR and deep learning model like LSTM, to empirically improve classi cation accuracy. Second, we create a new hate speech dataset by combining an external dataset together with the original released one from HASOC organizers. Third, the pre-trained model for word embedding feature extraction is used to improve the accuracy of hate speech classi cation. Our results show a good performance in both two F1 scores in Subtask A and weighted F1 score in Subtask B, while subtask B needs a further ne-grained experiment based on speci c classes.