Neural network models have become increasingly popular for text classi cation in recent years. In particular, the emergence of word embeddings within deep learning architectures has recently attracted a high level of attention amongst researchers. In this paper, we focus on how neural network models have been applied in text classi cation. Secondly, we extend our previous work [4, 3] using a neural network strategy for the task of abusive text detection. We compare word embedding features to the traditional feature representations such as n-grams and handcrafted features. In addition, we use an o -the-shelf neural network classi er, FastText[16]. Based on our results, the conclusions are: (1) Extracting selected manual features can increase abusive content detection over using basic ngrams; (2) Although averaging pre-trained word embeddings is a naive method, the distributed feature representation has better performance to ngrams in most of our datasets; (3) While the FastText classi er works e ciently with fast performance, the results are not remarkable as it is a shallow neural network with only one hidden layer; (4) Using pre-trained word embeddings does not guarantee better performance in the FastText classi er.
Text classi cation is an essential component in many applications, such as
sentiment analysis[27, 29], news categorization[16], and in our research domain of
interest, abusive text detection[
Originating from neural network structures, deep neural networks aim to automatically abstract feature representations for data based on hierarchical layers. It produces the state-of-the-art results in many text classi cation tasks[27, 25, 14]. In this paper, we rst present a review of the recent deep neural networks that are widely used in the text classi cation. Afterwards, we carry out some preliminary experiments on abusive text detection using fundamental neural network techniques. We compare word embeddings to the more traditional feature representations using SVM classi ers. In addition, we investigate an o -the-shelf neural network based classi er.
The structure of the rest of paper is as follows: Section 2 discusses two modes of using neural networks for text classi cation and how these have been used in abusive text detection. In Section 3, we present experimental results. Finally, conclusions and future work are presented in Section 4. 2
In this section, we investigate the current deep neural networks that have been used in general text classi cation tasks. We consider the use of deep neural networks in two modes: unsupervised for feature representation, and supervised for text classi cation. Furthermore, we review some deep neural networks that have been used in the abusive text detection domain. 2.1
The cornerstone of using neural networks in unsupervised mode is word2vec[22] which is an approach to generate a distributed representation for words in a vector space with lower-dimension. These word vectors, also called word embeddings, are learned based on the concept that the words with similar meaning should have the similar surrounding words. Mikolov et al.[22] introduced two models, skip-gram and continuous bag of words (CBOW). The framework of skip-gram is shown in Figure 1. The architecture is very straightforward. In the training process, the model generates a representation of current word Wt based on predicting the nearby words (Wt 2,Wt 1,Wt+1,Wt+2) in a window. After training, the vector of weights from the hidden layer is the representation of the word, the word embedding. CBOW is a similar model to skip-gram except it swaps the input and output, using nearby words to predict the current word.
There are also approaches that use neural networks to generate the
distributed representation for blocks of text (sentence, paragraph or document).
Here, we introduce three typical unsupervised models, paragraph2vec[20],
SkipThought[18] and autoencoder[
Neural network architectures have also been used directly for text classi cation tasks. For example, the simple model named FastText, proposed by Joulin et al.[16], is an e cient classi er. As shown in Figure 3, a sentence with a set of N ngrams features (X1; X2; :::; XN 1; XN ) are embedded and then averaged to the middle layer, the output layer is the softmax function that computes the probability distribution over the pre-de ned classes. The advantage of the FastText model is its fast execution time. However, the performance of using a shallow neural network in supervised classi cation is rudimentary (see the results of experiments in Section 3).
A range of more complex neural network architecutures have been used in recent years for text classi cation. Two particular architectures that are widely used are convolutional neural networks and recurrent neural networks. The following subsections expand on details of these two models respectively. Convolutional Neural Network The CNN model uses multiple layers with convolving lters that aim to capture `local' features. It was originally used in computer vision[19]. Subsequently, CNN was adopted in natural language processing and produced impressive results for many text classi cation tasks. The basic framework of CNN is shown in Figure 4. The sentence is represented by a set of word embeddings which are then mapped through a variety of convolutional lters of di erent sizes. Afterwards, the structure applies max-pooling to reduce the dimensionality of the features in order to reduce the complexity of the model and prevent over tting. The nal layer is the probability distribution over classes.
Several researchers have adapted the CNN architecture to perform text
classi cation. Ren et al. [27] proposed a context-based CNN for sentiment analysis
in a Twitter dataset, incorporating context information from relevant tweets
into the model in the form of word embedding vectors; Yang Wang et al.[30]
designed a hybrid CNN to integrate metadata with text content for fake news
classi cation. For sarcasm detection in social media, Amir et al.[
The superiority of the CNN model is its ability to mine the relations in the contextual windows and capture the `local' information such as semantic clues through the convolutional lters. However, given multiple lters with a large number of trainable parameters, the CNN is a heavy data model which usually requires a huge amount of training data. In addition, a critical issue of CNNs is the inability to handle sentences of variable length as input due to the restriction of a xed input size. To address this particular issue, research has focused on the recurrent neural network.
Recurrent Neural Network The RNN is an extension of a deep neural model that has ability to handle variable-length sequence input. Instead of learning features by the traditional feedforward structure, the RNN involves recurrent units which can use the information of the previous states. In addition, this architecture can e ectively address the issue that the input of text content is in xed size. Figure 5 illustrates the basic RNN framework and unfolded into a graph of the timesteps at time t. The input xt is fed to the model at timestamp t, St is the hidden layer that captures input information xt and previous state St 1 at timestamp t 1, ot illustrates the output of the model.
Although the RNN model can capture the information from previous states, the key drawback is the vanishing gradients problem which make is di cult to learn and tune parameters from earlier states in the network. The limitation is addressed by two advanced models, gated recurrent units (GRU) and long short-term memory (LSTM). As shown in Fig 6, the normal recurrent unit is replaced by the these two variant units with multiple useful gates. For GRU unit, the gates r and z are designed to control long-term and short-term dependencies which can mitigate against the vanishing gradients problem; For the LSTM unit, in addition to updates and reset gates, it adds one more cell c as memory for the previous state, and the gate o stands for how much information should the cell output.
To date, RNN based models have been widely applied in text classi cation.
Tang et al.[29] employed a gated RNN architecture in sentiment analysis, which
showed a superior performance over a standard RNN model. Wang et al.[31]
applied LSTM to predict polarities of tweets and gained 1% better accuracy
comparing to the standard RNN model. Typically, the standard LSTM is a
single direction structure that can only capture textual information from one
directional sequence. A bidirectional LSTM, consisting of two LSTMs that are
run in parallel, has proved to be useful in text classi cation[34]. In addition,
Tai et al.[28] developed a variant LSTM model that is based on a tree topology.
Rather than the traditional LSTM unit composed from the current timestamp
input and previous state, the tree-LSTM unit composes from the current
timestamp and previous tree-based state. This model shows superiority for sentiment
classi cation than the standard LSTM.
Early research in text classi cation on addressing abusive social media
comments focused on exploring useful information such as lexical features[
Furthermore, there are an increasing number of researchers who started to
work on complex deep neural networks for tackling the problem of abusive text
detection. Badjatiya et al.[
From our previous paper[
All datasets used for abusive detection in this paper were extracted from social
media websites. Although these websites include a variety of user content sources
including forums, micro-blog, media-sharing, news article discussion, chat and
Q&A, they share the common characteristic that they allow online users to
freely post their comments. We identi ed 8 published datasets[
For each dataset, we carried out pre-processing operations as follows: all letters were changed to the lowercase; the mentioned names started with `@' symbol were replaced by the anonymous text "@username"; links following with "http://" or "https://" were replaced by the generic term. Considering the comments are typically short, we did not remove stop-words nor implement word stemming. 3.2
We employed Support Vector Machines (SVMs) with a linear kernel, one of
the most e cient classi ers[15], as our classi cation algorithm. We established
baseline results by using ngrams feature representation, implemented in two
ways: 1-4 word level and 2-4 character level. Based on our previous work[
To validate our results, we applied strati ed 10-fold cross validation on each dataset. In addition, Table 1 shows that most of the datasets are imbalanced, with the positive instances (abusive) far less in number than the negative instances (non-abusive). We used resampling to randomly oversample the minority class of our training data and averaged results over three iterations.
The results of our experiments are reported using a standard classi cation measure recall, which measures ability to nd all instances of a speci c class. Our working assumption is that the consequence of failing to detect abusive content is more serious than the non abusive content being predicted as abusive. Therefore, we focus on abusive recall rather than non-abusive recall. Equation 1 shows the calculation where TruePositive is the proportion of abusive comments correctly classi ed as abusive, and FalseNegative is the proportion of abusive comments were wrongly classi ed as non-abusive. Average recall is also reported.
Recall =
T rueP ositives T rueP ositives + F alseN egatives (1) 3.3
We present the results in Table 2. User generated comments are usually
freeformat in style and ngrams at word level capture word misspellings and
abbreviations. Therefore, the performance of using ngrams in word level is normally worse
than the ngrams at character level. In addition, based on the results of
characterlevel ngrams, extracting additional features that capture syntactic and semantic
information[
Next, we analyzed the embedding representation which is averaging the word embeddings that are appear in the comment. The existing word embeddings we used in our paper are collected from Glove across 4 di erent dimensions (50,100,200,300) 1 which were pre-trained on the Wikipedia data corpus. Although averaging is one of the most naive approaches in using word vectors, it shows better performance to ngrams for most of the datasets. In general, using higher dimension word embedding achieves better performance since the higher dimension vectors contain in theory more information than the lower dimension ones. However, in our experiments, some datasets show the opposite results. For example, in D5, the performance of using 300-dimension vectors slumped 10% when compared to the 50-dimension vectors; In D6 also, abusive recall rate decreased as higher dimension vectors were used. The selection of appropriate word embeddings appears to play an important role when dealing with the speci c classi cation task. One reason for this counter intuitive result is the di erence between the training corpus language and the language of our posts. In this paper, the Glove word embeddings used are trained on the Wikipedia text corpus which generally uses a formal language style. However, social media datasets typically contain casual expressions, meaning that the word context information captured in Wikipedia-trained word embeddings may be di erent from that used in the abusive datasets. In the future, we will attempt to use pre-trained word embedding on a source corpus similar to the experimental dataset to boost the classi er accuracy.
FastText[16] is a one-hidden layer neural network text classi er. We analyzed this model in two ways: with and without using pre-trained word embeddings. As 1 https://nlp.stanford.edu/projects/glove/ shown in Table 2, we note that the performance of FastText in abusive content detection is in fact worse than the other feature representations with an SVM classi er. However, FastText is an e cient classi er with a much faster execution time than the SVMs. We attribute the classi cation results to the straightforward structure of FastText. In addition, using pre-trained word embeddings in FastText does not guarantee better results, which also may be due to the data source used for the pre-trained word embeddings. At this stage, we have not found a general model that performs best across all 9 datasets. The ngrams with syntactic & semantic information achieves signi cantly better results to the standard ngrams in 7 of our 9 datasets. Averaging word embedding is considered as the most straightforward approach to generate sentence vectors and achieves good results in our experiments. In addition, the FastText simple neural network shows poor performance in the abusive detection task. We suggest that an advanced structure needs to be designed for abusive detection in future work. 4
The purposes of this paper were two fold: (1) to investigate how o -the-shelf deep neural networks have been used across the two tasks within text classication - feature representation and the classi cation tasks itself and (2) to run preliminary experiments in abusive detection across 9 di erent social media datasets. We highlight the following aspects from our work: Firstly, we systematically summarized current neural models and categorized them into two modes, unsupervised approaches for generating distributed feature representations and supervised approaches for classi cation. Secondly, we attempted to compare the classi cation performance of using traditional feature representation to word embedding feature. Simply averaging pre-trained word embeddings has better results to ngrams feature representation in most of datasets. In addition, we employed a recent neural network, FastText. Due to its shallow architecture, the performance is unexceptional. Using pre-trained word embeddings can not guarantee better results possibly probably due to the characteristics of corpus used for pre-training the embeddings being di erent from our datasets. We will validate this assumption in our subsequent experiments.
The ultimate goal of our research is to develop a powerful classi cation model that can assist social media moderators to detect abusive comments e ciently and e ectively. The path to the goal can be divided into two directions, designing appropriate features for abusive comments and designing an outstanding model for detection. In future work, we will focus our research on both directions by exploring deep neural networks in unsupervised mode and supervised mode. 14. Hill, F., Cho, K., Korhonen, A.: Learning distributed representations of sentences from unlabelled data. arXiv preprint arXiv:1602.03483 (2016) 15. Joachims, T.: Text categorization with support vector machines: Learning with many relevant features. Machine learning: ECML-98 pp. 137{142 (1998) 16. Joulin, A., Grave, E., Mikolov, P.B.T.: Bag of tricks for e cient text classi cation.
arXiv preprint arXiv:1607.01759 (2016) 17. Kim, Y.: Convolutional neural networks for sentence classi cation. arXiv preprint arXiv:1408.5882 (2014) 18. Kiros, R., Zhu, Y., Salakhutdinov, R.R., Zemel, R., Urtasun, R., Torralba, A., Fidler, S.: Skip-thought vectors. In: Advances in neural information processing systems. pp. 3294{3302 (2015) 19. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classi cation with deep convolutional neural networks. In: Advances in neural information processing systems. pp. 1097{1105 (2012) 20. Le, Q., Mikolov, T.: Distributed representations of sentences and documents. In:
Proceedings of the 31st ICML-14. pp. 1188{1196 (2014) 21. Mehdad, Y., Tetreault, J.R.: Do characters abuse more than words? In: SIGDIAL
Conference. pp. 299{303 (2016) 22. Mikolov, T., Chen, K., Corrado, G., Dean, J.: E cient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013) 23. Nobata, C., Tetreault, J., Thomas, A., Mehdad, Y., Chang, Y.: Abusive language detection in online user content. In: Proceedings of the 25th International Conference on World Wide Web. pp. 145{153 (2016) 24. Park, J.H., Fung, P.: One-step and two-step classi cation for abusive language detection on twitter. arXiv preprint arXiv:1706.01206 (2017) 25. Pavlopoulos, J., Malakasiotis, P., Androutsopoulos, I.: Deep learning for user comment moderation. arXiv preprint arXiv:1705.09993 (2017) 26. Pennington, J., Socher, R., Manning, C.: Glove: Global vectors for word representation. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). pp. 1532{1543 (2014) 27. Ren, Y., Zhang, Y., Zhang, M., Ji, D.: Context-sensitive twitter sentiment classication using neural network. In: AAAI. pp. 215{221 (2016) 28. Tai, K.S., Socher, R., Manning, C.D.: Improved semantic representations from tree-structured long short-term memory networks. arXiv preprint arXiv:1503.00075 (2015) 29. Tang, D., Qin, B., Liu, T.: Document modeling with gated recurrent neural network for sentiment classi cation. In: EMNLP. pp. 1422{1432 (2015) 30. Wang, W.Y.: " liar, liar pants on re": A new benchmark dataset for fake news detection. arXiv preprint arXiv:1705.00648 (2017) 31. Wang, X., Liu, Y., Sun, C., Wang, B., Wang, X.: Predicting polarities of tweets by composing word embeddings with long short-term memory. In: ACL (1). pp. 1343{1353 (2015) 32. Xu, W., Sun, H., Deng, C., Tan, Y.: Variational autoencoder for semi-supervised text classi cation. In: AAAI. pp. 3358{3364 (2017) 33. Young, T., Hazarika, D., Poria, S., Cambria, E.: Recent trends in deep learning based natural language processing. arXiv preprint arXiv:1708.02709 (2017) 34. Zhou, P., Qi, Z., Zheng, S., Xu, J., Bao, H., Xu, B.: Text classi cation improved by integrating bidirectional lstm with two-dimensional max pooling. arXiv preprint arXiv:1611.06639 (2016)