Introduction

In the last couple of years we have started to see deep learning making significant inroads into areas where computers have previously seen limited success. Rather than requiring a set of fixed rules that are defined by the programmer, deep learning uses neural networks that learn rich non-linear relationships directly from data. Deep learning has also seen some success in NLP, for example in text classification. Text classification is an essential component in many applications, such as web searching, information filtering, and sentiment analysis [3].

A key problem in text classification is feature representation, which is commonly based on the bag-of-words (BoW) model, where unigrams, bigrams, ngrams or some specific patterns are extracted as features. Moreover, several feature selection methods, such as pLSA [4] or LDA [5] are applied to select more discriminative features. Nevertheless, traditional feature representation methods often have problems when they try to capture the semantics of the words because they ignore contextual information. This is a problem in text classification because contextual information is the key in order to correctly classify a text. Although high-order n-grams and more complex features are designed to capture more contextual information and word orders, the data sparsity problem remains, which heavily affects the classification accuracy.

In a recurrent neural network approach, the models analyze a text word by word and store the semantics of all the previous text in a fixed-sized hidden layer [6]. They receive as input a sequence of vectors and return another sequence that represents some information about the sequence at every step in the input. Although RNNs can learn long dependencies, they often fail to do so and tend to be biased towards their most recent inputs in the sequence [8]. Likewise, Long Short-term Memory Networks (LSTMs) incorporate a memory-cell and have been shown effective capturing long-range dependencies.

Classic LSTMs create the representation of each word of the sentence using only the left context. Is interesting to use also the right context if we want to create a more complete representation of the words, though. This can be done with a second LSTM that reads the same sequence in reverse. This type of LSTMs are named bidirectional LSTMs (BI-LSTMs) [9] and they create the representation of the words concatenating the left representation and the right representation. These representations effectively include a representation of a word in context, which is useful for numerous tasks.

On the other hand, Conditional Random Fields (CRF) are a probabilistic framework for labeling and segmenting structured data, such as sequences and trees. The underlying idea is that of defining a conditional probability distribution over label sequences given a particular observation sequence, rather than a joint distribution over both label and observation sequences. The primary advantage of CRFs is the relaxation of the independence assumption. Independence assumption states that the variables do not depend on each other and they do not affect each other in any way and this is not always the case and, consequently, it can lead to serious inaccuracies. Likewise, CRFs have been shown really effective in different tasks such as POS tagging [7], text processing [10] or computer vision [11].

Taking that into account, in this work we have employed a BI-LSTM with Conditional Random Fields (CRF) [7] in order to prove its effectiveness in misogynous tweet identification. In this area, one of the last works is [1] where the authors address the problem of automatic detection and categorization of misogynous language in online social media, and they set the bases to organize AMI-IberEval 2018 shared task [2].

In the rest of this paper we will first present the experimental setup we have used to carry out our experiments in section 2, followed by the results obtained in the shared task test set in section 3 and the main conclusions of the work in section 4.

System Description

We have divided this section into three subsections. In subsection 2.1 we explain the preprocessors we have used to tokenize and normalize the tweets, in subsection 2.2 the data resources we have employed in addition to the data shared by the organization, while in subsection 2.3 we focus on the used system.

Preprocessors

We have made use of one of python's packages designed for preprocessing tweets [12]. The tool performs tokenization, word normalization, word segmentation (for splitting hash-tags) and spell correction, using word statistics from 2 big corpora (English Wikipedia, twitter -330mil English tweets). In addition, for Spanish we have used a set of simple rules proposed in [13] for spell correction.

External Data

Our AMI System needs word-embeddings in order to create a better word representation for each word we find in the corpus. Thus, we have used wordembeddings extracted from the Spanish Billion Word Corpus [14] and from Wikipedia 2014 and Gigaword 5 [15].

AMI System

In order to classify the tweets we have employed a neural network based architecture, more precisely a specific Bi-LSTM (an RNN subclass) with a CRF on top of it as proposed in [7]. This kind of neural network is widely used to pursue sequence to sequence tagging. One of the advantages of using Bi-LSTM in contrast to other machine learning techniques such as SVM, Perceptron or CRFs is that the size of the context is automatically learned by the LSTM and there is no need to perform any complicated text preprocessing to obtain features to feed the tool. As we mentioned previously, our system is a tagger and marks the beginning and the next words of the sequences (IOB) we want to label. In this case we want to predict whether a tweet contains misogynous content or not. Thus, we introduce the tweets and the word-embeddings at the beginning of the process as in [7]. When a word is missing in the word-embeddings, the system replaces the word with unknown (UNK ) label.

In all cases the system returns every word of each tweet tagged with Yes label when the tweet contains misogyny and with No label otherwise. If the opposite happened, we would consider a tweet as misogynous if at least has one Yes label. The examples below are the output of the system for two tweets written in English and represent the aforementioned:

Results and Discussion

In AMI-IberEval 2018 shared task the participants can try their misogynous content identification systems in two languages: English and Spanish. We have participated in both languages and we have included the results for English track in table 1 and the results for Spanish track in table 2.

English R Team Accu R Team Accu R Team Accu 1 14-exlab-r1 91. If we analyze the results for English track, we observe that our position within all participants of the shared task is tenth with 78.9 of accuracy. Although we are far from winning the shared task (-12.4), we are in the first third of the classification and the two previous systems in the classification are not far (+ 0.4) from us which demonstrates a good performance of our system identifying misogynous tweets in English. Spanish R Team Accu R Team Accu R Team Accu 1 14-exlab-r3 81. On the other hand, our system's accuracy identifying Spanish written tweets is 76.8. This time our position within the all participants is seventeenth just above the shared task's baseline. However, almost all the participants have obtained accuracies between 81.4 and 76.6 which indicates that the vast majority of the systems are close to each other. Likewise, identifying misogynous content in Spanish written tweets is more difficult mostly because the lack of top quality resources (corpus, word-embeddings, preprocessors ...) we can find relatively easy for English.

Once we analyzed our system's results for both languages, taking into account our system was designed for sequential tagging or sequence labeling we consider the experimental setup has performed well in a task it was not thought for. We realize the best option to do text classification would have been a convolutional neural network (CNN) specially because the best systems of the state of the art employ this type of neural networks. Nevertheless, our main purpose has been to test a BI-LSTM with CRF on text classification task and bearing in mind its constraints the system has achieved reasonable results.

Conclusions

This paper presents our approach to automatically identify misogynous content in Twitter tweets. In order to carry out the task, we have chosen a neural network approach due to their ability to achieving remarkable performance in sentence and document modeling. In this work we focus on RNN approach using a Bidirectional Long Short Term Memory (Bi-LSTM) with Conditional Random Fields (CRF) and experimental results show that the system can achieve good performance identifying misogynous tweets obtaining 78.9 accuracy on English tweets and 76.8 accuracy on tweets.