This article presents the participation of the Intelligent Systems Group (GSI) at Universidad Politecnica de Madrid (UPM) in the Sentiment Analysis workshop focused in Spanish tweets, TASS2017. We have worked on Task 1, aiming to classify sentiment polarity of Spanish tweets. For this task we propose a Recurrent Neural Network (RNN) architecture composed of Long Short-Term Memory (LSTM) cells followed by a feedforward network. The architecture makes use of two di erent types of features: word embeddings and sentiment lexicon values. The recurrent architecture allows us to process text sequences of di erent lengths, while the lexicon inserts directly into the system sentiment information. The results indicate that this feature combination leads to enhanced sentiment analysis performances.
Recent developments in the area of deep
learning are strongly impacting sentiment
analysis techniques. While traditional
methods based on feature engineering are still
prevalent, new deep learning approaches are
succeeding and reduce the need of labeled
corpus and feature de nition. Moreover,
traditional and deep learning approaches
can be combined obtaining improved
results
This paper describes our participation in
TASS 2017
Taller de Analisis de Sentimientos en la SEPLN (TASS) is a workshop that fosters the research of sentiment analysis in Spanish for short text such as tweets. The rst task of this challenge, Task 1, consists in determining the global polarity at a message level.
The dataset for the evaluation of this task
Copyright © 2017 by the paper's authors. Copying permitted for private and academic purposes. considers annotated tweets with 4 polarity labels (P, N, NEU, NONE). P stands for positive, while N means negative and NEU is neutral. It is considered that NONE means absence of sentiment polarity. This task provides a corpus, which contains a total of 1514 tweets written in Spanish, describing a diversity of subjects.
We have faced this challenge as an opportunity to evaluate how these techniques could be applied in the TASS domain, and their results compared with the traditional techniques we used in a previous participation in this challenge (Araque et al., 2015).
The reminder of this paper is organized as follows. Sect. 2 introduces related work.
Then Sect.3 describes the proposed polarity classi cation model and its implementation, which is evaluated in Sect. 4. Finally, conclusions are drawn in Sect. 6. 2
Many works in the last years involve the use
of neural architectures to learn text
classication problems and, more speci cally, to
perform Sentiment Analysis. A relevant
example of this are Recursive Neural Tensor
Networks
In the context of the TASS challenge, it
has not been the rst time that neural
architectures have been proposed for solving the
di erent tasks. In
The approach followed for the Sentiment Analysis at Tweet level Task consists in a RNN composed of LSTM cells that parse the input into a xed-size vector representation. The representation of the text is used to perform the sentiment classi cation. Two variations of this architecture are used: (i) a LSTM that iterates over the input word vectors or (ii) over a combination of the input word vectors and polarity values from a sentiment lexicon.
The general architecture of the model takes as inputs the words vectors and the lexicon values for each word from an input tweet. Then, the inputs are passed through a one-layer LSTM with a tunnable number of hidden units. The generated representation is then used to determine the polarity of the input text using a feedforward layer with softmax activation as output function. The output of this last layer encodes the probability that the input text belongs to each class. Fig. 2 shows this architecture, which is further described as follows: 1. The input vector is the word embedding of each word in a given tweet. It contains word-level information or sentiment word-level information. Each speci c case will be described in more detail afterwards. 2. The RNN number of units is chosen during training for optimization purposes. In this work we use a one-layer LSTM to avoid over tting of the network to the training data. 3. The weight matrix has as input dimension the RNN size, and the number of classes as output dimension. This means that, taking as inputs the last LSTM output, we obtain a vector whose length is the number of classes. This matrix is also optimized during the training process. 4. The nal probability vector is obtained by passing the result of the previous matrix multiplication through a softmax function, which converts the values of the components of this result vector into probabilities representation. Finally, the predicted label for the tweet is the component of the output vector with the highest probability.
Following, the two types of inputs used are described thoroughly. 3.2
For this input, the tweet text is tokenized into word tokens, which are then expressed in a one-hot representation. That is, each token is represented as a IRjVj 1 vector with all 0s and and one 1 at the index of that token in the sorted token vocabulary. For example, the representations for the tokens a, antes and zebra would appear as: wa = 1 0 0 0 ; wantes = 0 1 0 0 wzebra = 0 0 0 1
We limit the number of words to a certain vocabulary size in order to limit the computational cost of this preprocessing step. Before feeding this data to the network, each tweet is presented by the one-hot representation of all the tokens in the tweet. 3.3
Additionally, we include di erent sentiment information into the word representations by means of a sentiment lexicon. In this case, a similar approach as the word-level RNN is followed, but instead of using information about the di erent words contained on each tweet, information about the sentiment of each word is used. In this case, the preprocessing process is modi ed: 1. First, each tweet is split into tokens. 2. Secondly, a sentiment dictionary is used to map words with sentiment polarity values. In this way, each word is mapped into a positive, neutral or negative value. 3. Finally, the representation of a word consists in its word vector concatenated with its sentiment polarity label. 3.4
Given the reduced number of training examples that is available for this task (Sec. 4) a number of regularization techniques has been used in the experiments. Regularization is used in machine learning to control the complexity of a learning model so it does not over t to the training data and generalized better to the test data.
It is known that Recurrent Neural
Network tend to heavily over t to the training
set
TensorFlow is an interface for expressing machine learning algorithms, and an implementation for executing such algorithms (Abadi, Agarwal, and et al., 2015). For implementing the model previously described, rst we had to de ne a computation graph composed by the RNN architecture, matrices and operations needed. Once the graph is de ned, the training process consists in iteratively adjusting numerical values in order to reach the best results. This task was done following those ideas:
The values to optimize are the internal parameters and matrices that form the network. Those are: the word embedding representations, the LSTM internal weights and the feedforward weight matrix used as last layer. At the beginning of training, those values are initialized in a random way using a normal distribution N ( ; ), with = 0, and are considered variables to be optimized at each training step by TensorFlow.
Having those variables de ned, the
training process iteratively modi es
them in order to reach the better
results. In order to obtain a error signal
that can be used to modify the learning
parameters we use a cost function which
has to be minimized. That
minimization problem is solved by applying the
gradient descent method via
backpropagation
In each iteration of the training process, which are known as epochs, data from the training set ows through all the computation graph yielding to a prediction result. The cost metric is computed by comparing the obtained result with the true training labels. When the backpropagation is nished, the variable values are updated and the following iteration proceeds.
In order to enhance performance, we use early stopping on the accuracy on the development set. That is, for each epoch we monitor the performance of the network in the development set. If it has not improved for a number of epochs (in this work, 3 epochs) the training process is stopped and the model weights are freezed.
The number of iterations can be chosen as well as other parameters such as the RNN size. For testing new examples, we use as input the test data, passing all the tweets through our model having as a result the vector of probabilities of the class each tweet belongs to, choosing the class with a higher probability value for each tweet. 4
For the development of Task 1 a training
and development dataset is made public,
containing 1,514 labeled tweets which belong
to the InterTASS corpus. Additionally, we
use the TASS2015 edition training dataset
that was extracted from the general
corpus
There are three test datasets, one belonging to the InterTASS corpus and two belonging to the General Corpus of TASS: the full version, with all the 60,798 tweets; and the 1k version, that contains a subset of 1,000 tweets.
In order to enhance the classi cation performance several hyper-parameters have been explored, and the values that yield better performance are selected to be used in the testing phase. The vocabulary size is set to 20,000, with a batch size of 256 and the num
LSTM + MLP LSTM + MLP
LSTM + MLP LSTM + MLP + Lexicon LSTM + MLP + Lexicon LSTM + MLP + Lexicon
InterTASS TASS (1k) TASS (Full) InterTASS TASS (1k) TASS (Full) ber of epochs being 20. With this value, the early stopping mechanism stopped the training before its completion. Regarding the size of the layers, the number of dimensions of the word embeddings is set to 16, as well as it is done with the number of units in the LSTM layer. The dimensionality of the feedforward layer is given by the output of the LSTM, which is 16, and the number of classes of the classi cation task (in this case, 4). Note that these values are smaller than in the usual neural architectures in order to further prevent over tting. Also, we select the parameter to 0.05, and the dropout rate to 0.7. 5
Table 1 shows the results of the two variations of the proposed model: LSTM+MLP stack with or without lexicon values. In light of this results it is possible to a rm that the used architecture shows promising performances in the task of sentiment analysis of tweets. Although, the achieved performances are below the best in this year challenge. This indicates that further work should be done in order to improve the results.
The experimental results con rm the idea that the introduction of a sentiment lexicon into the word presentations results, in general, bene cial for the nal performance. We see this improvement in the InterTASS and 1k corpora. Nevertheless, when attending to the Full corpus, a performance decrease in the Marco-F1 is observed. 6
In this paper we have described the participation of the GSI in the TASS 2017 challenge. Our proposal relies on a Recurrent Neural Network architecture for Sentiment Analysis with Long Short-Term Memory cells. This network can be fed with both word vectors and sentiment lexicon values. This approach is able to represent a arbitrarily long sequence of text due to the dynamic recurrent structure of the architecture. Also, several techniques have been used for avoiding overtting. From the experiments, it is seen that adding a sentiment lexicon can enhance the classi cation performance.
However, the proposed model does not compare with the best results in the TASS competition. This can be due to a number of reasons, but the training process suggests that over tting is a relevant issue. Although bene t comes from the use of regularization techniques, the network is not able to largely generalize. To address this, we think that future work in this direction should include the expansion of the training set.
Other possible improvement for future work is doing a better preprocessing of input texts at word level. In addition, Convolutional Neural Networks could be used for feature extraction in combination with the Recurrent Neural Network architecture. This could lead to the computation of most complex features, which could also yield better results.
The authors gratefully acknowledge the
support of NVIDIA Corporation with
the donation of the Titan X Pascal GPU
used in this research. This research
work is partially supported through the
projects Semola
Abadi, M., A. Agarwal, and P. B. et al. 2015.
TensorFlow: Large-scale machine learning on heterogeneous systems. Software available from tensor ow.org.
Araque, O., I. Corcuera, C. Roman, C. A. Iglesias, and J. F. Sanchez-Rada. 2015. Aspect based sentiment analysis of spanish tweets. In TASS@ SEPLN, pages 29{ 34.