The paper describes the system submitted to HASOC2019: Hate Speech and O ensive Content Identi cation in Indo-European Languages. The task aims to categorize o ensive language in social media, we only participated in Sub-task A for English, which aims to identify o ensive language and hate speech. In order to address this task, we proposed a system based on an ordered neurons LSTM with an attention model, and used a K-folding approach to ensemble. Our model achieved the Macro F1-score of 0.7882 and the Weighted F1-score of 0.8395 in the Subtask A for English language, and achieved the highest result.
With the popularization of the Internet, more and more people are
communicating on online social platforms. Therefore, the hate speech and o ensive language
in the social network have been paid increasing attention by people[
HASOC2019 is proposed for identifying hate speech and o ensive content
in Indo-European Languages [
In this competition, we only participated in Subtask A for English language.
For Subtask A, we used a deep learning method to build an ordered neurons
LSTM(ON-LSTM) with an attention model. ON-LSTM di ers from the original
LSTM in the way that it encodes the hierarchical structure of sentences into
features to enhance the expressive power of LSTM[
The structure of this paper is as follows: In section 2 we introduce some related work on identifying hate speech and o ensive language. In section 3, we describe the datasets and how to build the model. In section 4 we describe the experimental results and analysis. 2
In recent years, the topic of identifying hate speech and o ensive language has
attracted the attention of a large number of researchers in industry and academia.
Fortuna and Nunes believed that the eld of automatic detection of hate speech
in the text is very important for online social platforms and has unquestionable
social impact potential[
Gemeval2018[
As can be seen from the above, most of the methods for obtaining optimal results are machine learning models, and the deep learning models have not achieved good results. However, in this task, the deep learning model we used obtained the best results.
In this task, we used an extra dataset OLID[
Our network architecture is shown in Figure 1. Our model is built on
ONLSTM[
e e e eit = cs(sof tmax(Wixt + Uiht 1 + bi))
e e e
!t = fet eit
(1)
(2)
(3)
(5)
(6)
(7)
ct = !t (ft ct 1 + it bct) + (fet !t) ct 1 + (fet !t) ct (4)
where cs and !cs represents the explain brie y to the left and right
respectively, the fet and eit are called master forget gate and master input gate
respectively. !t gives the vector where the intersection is 1 and the rest is
0. In this way, the high-level information may be stored for a long distance,
and the low-level information may be updated at each step of input, so the
hierarchical structure is embedded by information hierarchy.
{ Attention layer: The main function of this layer is to assign a weight to
each word in the sentence, making the words that are biased towards hate
speech and o ensive language more prominent, in order to classify the
sentence. Because all words in a sentence can be di erent in their expression of
emotions[
First, we feed the word annotation hi, and through a nonlinear layer to get a deeper representation ui. Then, we calculate the similarity between ui and the word-level context vector us and use the softmax function to get the normalized weight i. Finally, we calculate the sentence vector s by the weighted sum of the word annotation hi. The speci c formula is as follows: ui = tanh(Ws
hi + bs) i =
exp(uiT Pi exp(uiT us)
us) s = X
i hi i { Output layer: This layer classi es and predicts the nal aggregated information. This layer consists of only one Dense layer with sigmoid activation function. 3.3
In this paper, we use a K-fold ensemble approach to enhance the performance of the model. The idea of this method is to cross-validate the source and K-fold. We randomly divide the training set into K parts and use the K 1 subsets to do the training. The remaining subset is used as the veri cation set, and K times are repeated. Finally, the K results are subjected to an accumulation averaging operation to obtain the nal output. The k-fold ensemble approach is shown in Figure 2.
The purpose of this operation is that di erent data sets will be trained during each training process, and di erent features will be extracted during the process of extracting features from the model, which can further improve the generalization ability of the model.
Whether in o cial dataset or the OLID dataset, from Twitter or Facebook, the data is very noisy because it is not processed. Tweets are rst processed using the Tweetokenize tool 1. In order to extract the model into better features, we further process the data, the speci c steps are as follows: { The word is retained for hashtags. Because it is very unique in itself, and it may help with text categorization. { Username mentions, e.g.: words starting with "@", they are replaced with 'username'. Because we think that usernames don't contain emotional expressions, and various usernames can bring a lot of unknown words, which can lead to reduced model performance. { All contractions are split into two tokens(e.g.: "you're" is changed to "you" and "are"). { The emojis are replaced with the corresponding words by emotion lexicons2 to express the corresponding emotion. { Lemmatization is restored to general form by WordNetLemmatizer. { Tokens are converted to lower case. 1 https://www.github.com/jaredks/tweetokenize 2 https://emojipedia.org/
Since the o cial veri cation set is not provided, we randomly extracted 500 tweets from the OLID data set and the o cially released training set as the veri cation set, and the ratio of NOT and HOF in the veri cation set is 1:1. 4.2
In our model, for this encoding layer, we set the hidden units to 128 and num levels to 16; We added a layer of dropout between the encoding layer and the attention layer. The purpose of this layer is to improve the generalization of the model and prevent the model from over tting. Connected behind the Attention layer is a layer of Dense with Relu activation function, and the number of hidden units is 128. Finally, we added the Dropout layer and the BatchNormalization layer, and the rate of all Dropout layers is 0.25. The activation function of the nal output layer is sigmoid for binary classi cation. The loss function of this model is binary crossentropy, and the optimizer is adam.
By using 5-fold crossvalidation on the training data, we set the batch size to
512 and the epoch to 20 for training. And the pre-training word vector we used
is fastText, which is provided by Mikolov et al. [
This Subtask A is to evaluate the classi cation system by calculating the Marco F1 score and Weighted F1 score. According to the o cial results in Subtask A for English language, the best Marco F1 score and Weighted F1 score of our model are 0.7882 and 0.8395 respectively, ranked 1st place, and our result is 0.0188 higher than the second place for Marco F1 score. The three results we submitted are shown in Table 1, and the top 5 teams from the o cial leaderboard is shown in Table 2. Run 1 is the result without using the k-fold ensemble method, while run 2 used the k-fold ensemble method. It can be seen that the model's performance is increased by 0.02 for run 2, which is very e ective and provides more powerful performance for the model. The parameters of run 3 are only slightly di erent from those of run 2.
From the detailed results provided by the o cial, our model's F1-score for HOF is only 0.69, which is much lower than the 0.89 of NOT, indicating that our model excels at distinguishing between NOT, which may be due to training data. The number of HOFs is lower than NOT, and we have not dealt with the imbalance of data. The detailed results of the run 2 are shown in Table 3. In this paper, we presented a model to identify hate speech and o ensive language for English language, and also used the k-fold ensemble method to improve the generalization ability of the model, and achieved the best results in Subtask A for English language. In future research, we will consider handling data imbalances and introducing location features to try to further enhance the performance of the model.
This work was supported by the Natural Science Foundations of China under Grants 61463050, the NSF of Yunnan Province under Grant 2015FB113.