LyS at TASS 2015: Deep Learning Experiments for Sentiment Analysis on Spanish Tweets∗

Abstract

This paper describes the participation of the LyS group at tass 2015. In this year's edition, we used a long short-term memory neural network to address the two proposed challenges: (1) sentiment analysis at a global level and (2) aspect-based sentiment analysis on football and political tweets. The performance of this deep learning approach is compared to our last-year model, based on a square-regularized logistic regression. Experimental results show that strategies such as unsupervised pre-training, sentiment-specific word embedding or modifying the current architecture might be needed to achieve state-of-the-art results.

-

The 4th edition of the tass workshop addresses two of the most popular tasks on sentiment analysis (sa), focusing on Spanish tweets: (1) polarity classification at a global level and (2) a simplified version of aspectbased sentiment analysis, where the goal is to predict the polarity of a set of predefined and identified aspects (Villena-Rom´an et al., b).

The challenge of polarity classification has been typically tackled from two different angles: lexicon-based and machine learning (ml) approaches. The first group relies on sentiment dictionaries to detect the subjective words or phrases of the text, and defines lexical- (Brooke, Tofiloski, and Taboada, 2009; Thelwall et al., 2010) or syntacticbased rules (Vilares, Alonso, and Go´mezRodr´ıguez, 2015c) to deal with phenomena such as negation, intensification or irrealis.

The second group focuses on training classifiers through supervised learning algorithms that are fed a number of features (Pang, Lee, and Vaithyanathan, 2002; Mohammad, Kiritchenko, and Zhu, 2013; Hurtado and Pla, 2014) . Although competitive when labelled data is provided, they have shown weakness when interpreting the compositionality of complex phrases (e.g. adversative subordinate clauses). In this respect, some studies have evaluated the impact of syntacticbased features on these supervised learning techniques (Vilares, Alonso, and Go´mezRodr´ıguez, 2015b; Joshi and Penstein-Ros´e, 2009) or other related tasks, such as multitopic detection on tweets (Vilares, Alonso, and Go´mez-Rodr´ıguez, 2015a) .

More recently, deep learning (Bengio, 2009) has shown its competitiveness on polarity classification. Bespalov et al. (2011) introduce a word-embedding approach for higher-order n-grams, using a multi-layer perceptron and a linear function as the output layer. Socher et al. (2013) introduce a new deep learning architecture, a Recursive Neural Tensor Network, which improved the state of the art on the Pang and Lee (2005) movie reviews corpus, when trained together with the Stanford Sentiment Treebank. Tang et al. (2014) suggest that currently existing word embedding methods are not adequate for sa, because words with completely different sentiment might appear in similar contexts (e.g. ‘good’ and ‘bad’ ). They pose an sentiment-specific words embedding (sswe) model, using a deep learning architecture trained from massive distant-supervised tweets. For Spanish, Montejo-Ra´ez, Garc´ıaCumbreras, and D´ıaz-Galiano (2014) apply word embedding using Word2Vec (Mikolov et al., 2013) , to then use those vectors as features for traditional machine learning techniques.

In this paper we also rely on a deep learning architecture, a long short-term memory (lstm) recurrent neural network, to solve the challenges of this tass edition. The results are compared with respect to our model for last year’s edition, a logistic regression approach fed with hand-crafted features. 2

Task1: Sentiment Analysis at a global level

Let L={l0, l1, ..., ln} be the set of polarity labels and T ={t0, t1, ..., tm} the set of labelled texts, the aim of the task consists of defining an hypothesis function, h : T → L.

To train and evaluate the task, the collection from tass-2014 (Villena-Roma´n et al., 2015) was used. It contains a training set of 7 128 tweets, intended to build and tune the models, and two test sets: (1) a poolinglabelled collection of 60 798 tweets and (2) a manually-labelled test set of 1 000 tweets. The collection is annotated using two different criteria. The first one considers a set of 6 polarities (L6): no opinion (none), positive (p), strongly positive (p+), negative (n), strongly negative (n+) and mixed (neu), that are tweets that mix both negative and positive ideas. A simplified version with 4 classes (L4) is also proposed, where the polarities p+ and n+ are included into p and n, respectively.

In the rest of the paper, we will use h4 and h6 to refer our prediction models for 4 and 6 classes, respectively. 3

Task2: Sentiment Analysis at the aspect level

Let L={l0, l1, ..., ln} be the set of polarity labels, A={a0, a1, ...ao} the set of aspects and a T ={t0, t1, ..., tm} the set of texts, the aim of the task consists of defining an hypothesis function, h : A × T → L. Two different corpora are provided to evaluate this task: a social-tv corpus with football tweets (1 773 training and 1 000 test tweets) and a political corpus (784 training and 500 test tweets), called stompol. Each aspect can be assigned the p, n or neu polarities (L3).

The tass organisation provided both A and the identification of the aspects that appear in each tweet, so the task can be seen as identifying the scope s(a, t) of an aspect a in the tweet t ∈ T , with s a substring of t and a ∈ A, to then predict the polarity using the hypothesis function, h3(s) → L3.

To identify the scope we followed a na¨ıve approach: given an aspect a that appears at position i in a text, t=[w0, ..., wi−x, ..., ai, ..., wi+x, ..., wp], we created a snippet of length x that is considered to be the scope of the aspect. Preliminary experiments on the social-tv and the stompol corpus showed that x = 4 and taking the entire tweet were the best options for each collection, respectively. 4

Supervised sentiment analysis models

Our aim this year was to compare our lastyear model to a deep learning architecture that was initially available for binary polarity classification. 4.1

Long Short-Term Memory

Long Short-Term Memory (lstm) is a recurrent neural network (rnn) proposed by Hochreiter and Schmidhuber (1997). Traditional rnn were born with the objective of being able to store representations of inputs in form of activations, showing temporal capacities and helping to learn short-term dependencies. However, they might suffer from the problem of exploding gradients1. The lstm tries to solve these problems using a different type of units, called memory cells, which can remember a value for an arbitrary period of time.

In this work, we use a model composed of a single lstm and a logistic function as the output layer, which has an available implementation2 in Theano (Bastien et al., 2012) .

To train the model, the tweets were tokenised (Gimpel et al., 2011), lemmatised (Taul´e, Mart´ı, and Recasens, 2008) , converted to lowercase to reduce sparsity and finally indexed. To train the lstm-rnn, we relied on adadelta (Zeiler, 2012) , an adaptive learning rate method, using stochastic training (batch size = 16) to speed up the learning process. Experiments with non-stochastic training runs did not show an improvement in terms of accuracy. We empirically explored the size of the word embedding3 and the number of words to keep in the vocabulary4, obtaining the best performance using a choice of 128 and 10 000 respectively. 4.2

L2 logistic regression

Our last-year edition model relied on the simple and well-known squared-regularised logistic regression (l2-lg), that performed very competitively for all polarity classification tasks. A detailed description of this model can be found in Vilares et al. (2014a), but here we just list the features that were used: lemmas (Taul´e, Mart´ı, and Recasens, 2008) , psychometric properties (Pennebaker, Francis, and Booth, 2001) and subjective lexicons (Saralegi and San Vicente, 2013) . This architecture also obtained robust and competitive performance for English tweets, on SemEval 2014 (Vilares et al., 2014b) .

Penalising neutral tweets

Previous editions of tass have shown that the performance on neu tweets is much lower than for the rest of the classes (VillenaRom´an et al., a). This year we proposed a small variation on our l2-lg model: a penal1The gradient signal becomes either too small or large causing a very slow learning or a diverging situation, respectively.

2http://deeplearning.net/tutorial/ 3The size of the vector obtained for each word and the number of hidden units on the lstm layer.

4Number of words to be indexed. The rest of the words are set to unknown tokens, giving to all of them the same index. ising system for neu tweets to determine the polarities under the L6 configuration, where: given an L4 and an L6 lg-classifier and a tweet t, if h6(t) = neu and h4(t) 6= neu then h6(t) := h4(t). The results obtained on the test set shown that we obtained an improvement of 1 percentage point with this strategy (from 55.2% to 56.8% that is reported in the Experiments section). 5

Experimental results

Table 1 compares our models with the best performing run of the rest of the participants (out of date runs are not included). The performance of our current deep learning model is still far from the top ranking systems, and from our last-year model too, although it worked acceptably under the L6 manuallylabelled test.

Table 2 and 3 show the f1 score for each polarity, for the lstm-rnn and l2-lg models, respectively. The results reflect the lack of capacity of the current lstm model to learn the minority classes in the training data (p, n+ and neu). In this respect, we plan to explore how balanced corpora and bigger corpora can help diminish this problem. football and political tweets. The trend remains in this case and the machine learning approaches outperformed again our deep learning proposal. 6

Conclusions and future research

In the 4th edition of tass 2015, we have tried a long short-term memory neural network to determine the polarity of tweets at the global and aspect levels. The performance of this model has been compared with the performance of our last-year system, based on an l2 logistic regression. Experimental results suggest that we need to explore new architectures and specific word embedding representations to obtain stateof-the-art results on sentiment analysis tasks. In this respect, we believe sentiment-specific word embeddings and other deep learning approaches (Tang et al., 2014) can help enrich our current model. Unsupervised pretraining has also been shown to improve performance of deep learning architectures (Severyn and Moschitti, 2015) .

System elirf lys-lg•

gsi tid-spark lys-lstm•

Bespalov, D., B. Bai, Y. Qi, and A. Shokoufandeh. 2011. Sentiment classification based on supervised latent n-gram analysis. In Proceedings of the 20th ACM international conference on Information and knowledge management, pages 375—-382.

ACM.

Brooke, J, M Tofiloski, and M Taboada. 2009. Cross-Linguistic Sentiment Analysis: From English to Spanish. In Proceedings of the International Conference RANLP-2009, pages 50–54, Borovets, Bulgaria. ACL.

Gimpel, K, N Schneider, B O’connor, D Das, D Mills, J Eisenstein, M Heilman, D Yogatama, J Flanigan, and N A Smith. 2011. Part-of-speech tagging for Twitter: annotation, features, and experiments. HLT ’11 Proc. of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers, 2:42–47.

Hochreiter, S and J. Schmidhuber. 1997.

Long short-term memory. Neural computation, 9(8):1735–1780.

Hurtado, L. and F. Pla. 2014. ELiRF-UPV en TASS 2014: An´alisis de sentimientos, deteccio´n de t´opicos y ana´lisis de sentimientos de aspectos en twitter. In Proceedings of the TASS workshop at SEPLN.

References

Bastien , F. , P. Lamblin , R. Pascanu , J. Bergstra , I. Goodfellow , A. Bergeron , N. Bouchard , D. Warde-Farley , and Y. Bengio . 2012 . Theano: new features and speed improvements . arXiv preprint arXiv:1211 . 5590 .

Bengio , Y. 2009 . Learning deep architecJoshi, M and C Penstein- Ros´e. 2009 . Generalizing dependency features for opinion mining . In Proceedings of the ACLIJCNLP 2009 Conference Short Papers, ACLShort '09 , pages 313 - 316 , Stroudsburg, PA, USA. Association for Computational Linguistics.

Mikolov , T. , K. Chen , G. Corrado, and J. Dean . 2013 . Efficient estimation of word representations in vector space . arXiv preprint arXiv:1301 . 3781 .

Mohammad , S. M, S . Kiritchenko, and X. Zhu . 2013 . NRC-Canada: Building the State-of-the-Art in Sentiment Analysis of Tweets . In Proceedings of the seventh international workshop on Semantic Evaluation Exercises (SemEval-2013) , Atlanta, Georgia, USA, June.

Montejo-Ra´ez, A ., M. A. Garc ´ıa- Cumbreras , and M. C. D´ ıaz-Galiano. 2014 . Participacio´n de SINAI word2vec en TASS 2014 . In Proceedings of the TASS workshop at SEPLN.

Pang , B. and L. Lee . 2005 . Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales . In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics , pages 115 - 124 . Association for Computational Linguistics.

Pang , B. , L. Lee , and S Vaithyanathan . 2002 . Thumbs up? Sentiment classification using machine learning techniques . In Proceedings of EMNLP , pages 79 - 86 .

Pennebaker , J. W. , M. E. Francis , and R. J. Booth . 2001 . Linguistic inquiry and word count: LIWC 2001 . Mahway: Lawrence Erlbaum Associates, page 71 .

Saralegi , X. and I. San Vicente . 2013 . Elhuyar at TASS 2013 . In Alberto D´ıaz Esteban, In˜aki Alegr´ıa Loinaz, and Julio Villena Rom´an, editors, XXIX Congreso de la Sociedad Espan˜ola de Procesamiento de Lenguaje Natural (SEPLN 2013 ). TASS 2013 - Workshop on Sentiment Analysis at SEPLN 2013 , pages 143 - 150 , Madrid, Spain, September.

Severyn , A. and A. Moschitti . 2015 . UNITN: Training Deep Convolutional Neural Network for Twitter Sentiment Classification . In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015 ), pages 464 - 469 , Denver, Colorado. Association for Computational Linguistics.

Socher , R. , A. Perelygin , J. Wu , J. Chuang , C. D. Manning , A. Ng , and C. Potts . 2013 . Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank . In EMNLP 2013. 2013 Conference on Empirical Methods in Natural Language Processing. Proceedings of the Conference , pages 1631 - 1642 , Seattle, Washington, USA. ACL.

Tang , D. , F. Wei , N. Yang , M. Zhou , T. Liu, and B. Qin . 2014 . Learning sentimentspecific word embedding for twitter sentiment classification . In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics , pages 1555 - 1565 .

Taul ´e, M., M. A. Mart ´ı, and M. Recasens . 2008 . AnCora: Multilevel Annotated Corpora for Catalan and Spanish . In Nicoletta Calzolari, Khalid Choukri, Bente Maegaard, Joseph Mariani, Jan Odjik, Stelios Piperidis, and Daniel Tapias, editors, Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08) , pages 96 - 101 , Marrakech, Morocco.

Thelwall , M. , K. Buckley , G. Paltoglou , D. Cai , and A. Kappas . 2010 . Sentiment Strength Detection in Short Informal Text . Journal of the American Society for Information Science and Technology , 61 ( 12 ): 2544 - 2558 , December.

Vilares , D. , M. A. Alonso , and C. Go´mezRodr´ıguez. 2015a. A linguistic approach for determining the topics of Spanish Twitter messages . Journal of Information Science , 41 ( 2 ): 127 - 145 .

Vilares , D. , M. A. Alonso , and C. Go´mezRodr´ıguez . 2015b. On the usefulness of lexical and syntactic processing in polarity classification of Twitter messages . Journal of the Association for Information Science Science and Technology , to appear.

Vilares , D. , M. A Alonso, and C. Go´mezRodr´ıguez. 2015c. A syntactic approach for opinion mining on spanish reviews . Natural Language Engineering , 21 ( 01 ): 139 - 163 .

Vilares , D. , Y. Doval , M. A. Alonso , and C. Go´mez-Rodr´ıguez . 2014a. LyS at TASS 2014 : A prototype for extracting and analysing aspects from spanish tweets . In Proceedings of the TASS workshop at SEPLN.

2014b. LyS : Porting a Twitter Sentiment Analysis Approach from Spanish to English na . In Proceedings og The 8th InternationalWorkshop on Semantic Evaluation (SemEval 2014 ), number SemEval, pages 411 - 415 .

Villena-Roma´n, J., J. Garc´ıa- Morera , C. Moreno-Garc´ıa, S. Lana-Serrano, and J. C. Gonza´lez-Crist´oba . TASS 2013 - a second step in reputation analysis in Spanish . Procesamiento del Lenguaje Natural , pages 37 - 44 .

Villena-Roma´n, Julio, Janine Garc´ ıa -Morera, Miguel A. Garc´ıa- Cumbreras , Eugenio Mart´ınez-Ca´mara, M. Teresa Mart´ınValdivia, and L. Alfonso Uren˜a Lo´pez . Overview of TASS 2015 .

Villena-Roma´n, L., E. Mart´ınez-Ca´ mara, Janine Morera-Garc´ıa, and S. M. Jim ´enezZafra. 2015 . TASS 2014-the challenge of aspect-based sentiment analysis . Procesamiento del Lenguaje Natural , 54 : 61 - 68 .

Zeiler , M.D. 2012 . ADADELTA: An adaptive learning rate method . arXiv preprint arXiv:1212 . 5701 .