This paper presents three approaches for classifying the sentiment of tweets for di erent Spanish variants in the TASS 2018 challenge. The classi ers are based on Support Vector Machines (SVM), Convolutional Neural Netowrks (CNN) and Long Short Term Memory networks (LSTM). Although di erent classi ers worked better for di erent language variants, the use of word embeddings was key for obtaining performance improvements. Also, using a mixed-balanced training method for the LSTM resulted in a signi cant improvement in the detection of neutral tweets.
Sentiment analysis is one of the most important tasks related to subjectivity analysis within Natural Language Processing. The sentiment analysis of tweets is especially interesting due to the large volume of information generated every day, the subjective nature of most messages, and the easy access to this material for analysis and processing.
The existence of speci c tasks related to this
eld, for several years now, shows the interest of the NLP community in working on this subject. The International Workshop on Semantic Evaluation (SemEval) includes a task on Tweets Sentiment Analysis since 2013 1.
For Spanish, the TASS workshop, organized by the SEPLN (Sociedad Espan~ola para el Procesamiento del Lenguaje Natural), focuses on this task since 20122.
1https://www.cs.york.ac.uk/semeval-2013/ task2.html 2http://www.sepln.org/workshops/tass/2012/
In the TASS editions prior to 2017, most of
the participants presented machine learning
systems based on hand crafted features. For
example, in TASS 2016
In TASS 2017
Copyright © 2018 by the paper's authors. Copying permitted for private and academic purposes.
put domain-speci c and general-domain sets
of embeddings; Ceron-Guzman (2017), who
presented an ensemble of SVM and Logistic
Regression classi ers;
On the other hand, for the rst time,
SemEval 2018
In this paper we describe di erent
approaches for Spanish tweet classi cation
presented by the RETUYT-InCo team for the TASS
2018 sentiment analysis challenge
For this year's edition of the challenge, the organizers provided three sets of corpora for Spanish variants spoken in di erent countries: Spain (ES), Costa Rica (CR) and Peru (PE). For each of the variants, training, development and test data was provided. The training and development sets were annotated with four possible polarity categories per tweet: P, N, NEU or NONE. The test corpora had no annotations.
For some of our experiments, we also used the general TASS training data from a previous edition of the competition. This corpus was divided in training (85 %) and development (15 %) subsets. Table 1 shows the sizes of the di erent corpora and the number of tweets for each class.
Each corpus was pre-processed as follows: Redundant space characters and ellipsis were removed.
N NEU NONE
P Total
N NEU NONE
P Total
N NEU NONE
P Total
N NEU NONE
P Total
Sequences of three or more occurrences of the same character were replaced by a unique occurrence of that character. For instance, \holaaaa" was replaced by \hola".
Interjections denoting laughter (\jajajaja", \jejeje", \jajaj") were replaced by the token \jaja".
The text was converted to lowercase.
We did not include any grammatical information, like lemma, POS-tag, morphological or syntactic information. 3 3.1
We built a subjective lexicon consisting of
the union of three subjective lexicons
available for Spanish
In a previous work
We used a 300 dimension word embeddings
set, trained by
We built a regression algorithm based on SVM using the subjective lexicon as training set. This model should be able to predict a real number representing the polarity of each word. The model takes as input the 300 real values of the vector representing the word and returns a real value for the word polarity. For training, we assigned the value 1 to positive words and the value -1 to negative words. In table 2 we show the result of applying this classi er to some words.
Word
Prediction apoyamos 1.09973945
amigo 0.89985318 excelente 1.04574863 cansancio -0.98582263 abat an -1.02370082 horrible -0.91882273 apartamento -0.30991363
telefono -0.48884958
As these examples show, words expected to be positive have values close to 1 and words expected to be negative have values close to -1. On the other hand, neutral words have values closer to 0 than to 1 or -1. 3.4
We obtained the list of all the words in the training corpus and for each one we calculated the distribution of the four categories in all the tweets where this word occurs. We consider that a word is a category marker if it occurs at least 75 % times in this category. Using this information we built markers lists for the four categories: 429 positive words, 438 negative words, 12 neutral words, and 33 no opinion markers. 4 4.1
The SVM classi er con gurations are almost
the same as the ones described in
Centroid of tweet word embeddings.
Previous works showed that, while using
the centroid (or mean vector) is a
simple technique, it reaches good results for
several NLP problems, particularly for
sentiment analysis
As in the previous editions, the SVM
experiments were done using the scikit-learn
toolkit
Our CNN approach uses a simpler network
than the one used in
Our LSTM neural network architecture uses the embedding for each word as input, up to a maximum of 32 words. This input is sent through a LSTM layer and then a dense layer with a dropout of 0.2, before getting the output through a softmax layer, as shown in gure 2.
The initial experiments using this network yielded good accuracy results, but the macroF measure was very low because the network did not predict any output for the class NEU. This class has proven to be the most di cult to learn throughout our experiments. However, we started to get better results using a di erent training strategy: we created two versions of the training corpus, one of them with all the tweets, and the other one taking the same number of tweets for each category (exactly the same number of tweets as the NEU category, which was the one with the fewest tweets). We call this set the balanced corpus.
The training strategy involves training one epoch with the whole corpus and one epoch with the balanced corpus, then iterate this training process until the performance over the development set stopped improving. Training the network in this fashion yields a little less accuracy but it compensates in macro-F measure, as it captures a lot more tweets of the NEU category.
Both neural network approaches (CNN
and LSTM) were implemented using the
Keras library (Chollet, 2015) and trained using
the adam optimization algorithm
Three di erent corpora considering three Spanish variants were used for this task: from Spain (ES), Costa Rica (CR) and Peru (PE). Furthermore, the systems could be trained with training data for the corresponding Spanish variant (monolingual case), or they could be trained using data from other variants (cross-lingual case). We decided to submit the two best results for each classi er family on each of the variants and training combinations. Our results are shown in tables 3 and 4.
Taking in consideration the macro-F measure, our systems achieved good performance in all the test variants, ranking top 1 for monolingual CR and PE and cross-lingual ES and CR; and ranking top 2 for monolingual ES and cross-lingual PE. The best results for our systems in the monolingual training case were achieved by the neural networks approaches: in two cases, the best systems were LSTMs and in the other case it was a CNN. In the cross-lingual training cases, on the other hand, the three best systems were SVMs.
We submitted another system that
combined the output probabilities of the best
LSTM and SVM, in order to leverage the
information of both classi ers. This approach
had yielded good results in the past
As can be seen in table 5, one of the reasons the LSTM could have gotten better results over the test set was because it could We presented three approaches for TASS 2018 Task 1 about classifying the sentiment of tweets in di erent Spanish variants. The approaches we used are: SVM using word embedding centroids and manually crafted features, CNN using word embeddings as input, and LSTM using word embeddings, trained with focus on improving the recognition of neutral tweets. None of the classi ers was a clear winner in our experiments, as some of them worked better than others for di erent Spanish variants. However, we found that the training method used for the LSTMs signi cantly improved their macro-F measure by improving the detection of neutral tweets. In all cases, the use of word embeddings was key to improve the performance of the methods.
Dev
F1
P F1 46.4 47.1 45.0 44.8 43.8 47.0 Ceron-Guzman, J. A. 2017. Classi er ensembles that push the state-of-the-art in sentiment analysis of spanish tweets. En Proceedings of TASS. 47.6 47.4 42.1 46.2 Chollet, F. 2015. Keras. https://github. 47.3 com/fchollet/keras. 44.4
Cruz, F. L., J. A. Troyano, B. Pontes, y F. J.
Ortega. 2014. Building layered, multilingual sentiment lexicons at synset and lemma levels. Expert Systems with Applications, 41(13):5984{5994. Guillena A. Piad Mor s, y J. VillenaRoman, editores, Proceedings of TASS 2018: Workshop on Semantic Analysis at SEPLN (TASS 2018), volumen 2172 de CEUR Workshop Proceedings, Sevilla, Spain, September. CEUR-WS.