In this paper we describe our participation in TASS 2017 shared task on polarity classi cation of Spanish tweets. For this task we built a classi cation model based on the Lingmotif Spanish lexicon, and combined this with a number of formal text features, both general and CMC-speci c, as well as single-word keywords and n-gram keywords, achieving above-average results across all three datasets. We report the results of our experiments with di erent combinations of said feature sets and machine learning algorithms (logistic regression and SVM).
The use of microblogging sites in general, and
Twitter in particular, has become so well
established that it is now a common source
to poll user opinion and even social happiness
It is not surprising, then, that Twitter
datasets are increasingly being used for
sentiment analysis shared tasks. The SemEval
series of shared tasks included Sentiment
Analysis of English Twitter content in 2013
The General Corpus of TASS was
published for TASS 2013
TASS 2017 (Mart nez-Camara et al., 2017) keeps the Spain-only General Corpus of TASS, and introduces a new international corpus of Spanish tweets, named InterTASS.
The InterTASS corpus adds considerable dif
culty to the tasks not only because of its multi-varietal nature, but also because, unlike the General Corpus of TASS, content has
Copyright © 2017 by the paper's authors. Copying permitted for private and academic purposes. not been ltered or their users selected, which introduces many and varied decoding issues. 1.1
TASS 2017 proposes two classi cation tasks. Task 1 focuses on sentiment analysis at the tweet level, while Task 2 deals with aspectbased sentiment classi cation. We took part in Task 1, since we have not yet tackled aspect-based sentiment analysis. The aim of this task is the automatic classi cation of tweets in one of 4 levels: positive, negative, neutral, and none.
The neutral/none distinction introduces added di culty to the classi cation task. Tweets annotated as none are supposed to express no sentiment whatsoever, as in informative or declarative texts, whereas the neutral category of tweets is meant to qualify tweets where both positive and negative opinion is expressed, but they cancel each other out, resulting in a neutral overall message.
We believe this distinction is too fuzzy to be annotated reliably. First, precise balance of polarity is hardly ever found in any message where sentiment is expressed: the message is usually "negative/positive situation x, somehow counterbalanced by positive/negative situation y ", with an entailment that the result is tilted to either side. The following are examples of tweets tagged as neutral in the training set: 768547351443169284 Parece que las cosas no te van muy bien, espero que todo mejore, que todo el mundo merece ser feliz. 770417499317895168 No hay nada mas bonito q separarse d una persona y q al tiempo t diga q t echa de menos... pero a mi no m va a pasar We also found a number of examples where tweets that clearly fell into none cases, where wrongly annotated as neutral: 768588061496209408 Estas palabras, del Poema, INSTANTES, son de Nadine Stair. Escritora norteamericana, a la q le gustan los helados. 767846757996847104 pues imaginate en una casa muy grande 769993102442524674 Ninguno de los clubes lo hizo o cial pero se dice que s
These annotation issues are to be
expected, due to the added cognitive load that
is placed on the annotators, as other
researchers have pointed out
Within Sentiment Analysis it is common
to distinguish corpus-based approaches from
lexicon-based approaches. Although a
combination of both methods can be found in the
literature
Using sentiment dictionaries has a long
tradition in the eld. WordNet (Fellbaum,
1998) has been a recurrent source of lexical
information
However, the valence of a given word may
vary greatly from one domain to another, a
fact well recognized in the literature
Our approach to using a lexicon takes some ideas from the aforementioned approaches. We describe it in the next section. 2
Our system for this polarity classi cation task relies on the availability of rich sets of lexical, sentiment, and (formal) text features, rather than on highly sophisticated algorithms. We basically used a logistic regression classi er trained on the optimal set of features after many feature combinations were tried on the training set. We also tried a SVM classi er on the same feature sets, but we consistently obtained poorer results compared to the logistic regression classi er. Parameter netuning on each classi er was very limited; we simply performed a grid search on the C parameter, which threw 100 as optimal. For the SVM classi er we found the RBF kernel to perform better than the linear kernel2. We mostly focused on feature selection and combination.
We obtained good results on the three test datasets, with some important di erences between the InterTASS and General datasets. Results, however, were not as good as we had anticipated based on our experiments on the training datasets. We discuss this in section 3 below. Here we describe our general system architecture and feature sets.
This TASS shared task is our rst
experience with Twitter data sentiment classi
cation proper, although we had the related
experience from our recent participation in
WASSA-2017 Shared Task on Emotion
Intensity
Since our focus was on identifying the predictive power of classi cation features, and intended to perform many experiments with features combinations, we designed a simple tool to facilitate this.
This tool, Lingmotif Learn, is a
GUIenabled convenience tool that manages
datasets and uses the Python-based
scikitlearn
Sentiment features are returned by the
Lingmotif SA engine. Lingmotif
For large collections of short texts, such as Twitter datasets, it provides a multidocument mode whose default output is classi cation. In the current publicly available version this classi cation is entirely based on the Text Sentiment Score (TSS), which attempts to summarize the text's overall polarity on a 0-100 scale. TSS is calculated as a function of the text's positive and negative scores and the sentiment intensity, which re ects the proportion of sentiment to nonsentiment lexical items in the text. Speci c details on TSS calculation can be found in Moreno-Ortiz (2017a). A description of its applications is found in Moreno-Ortiz (2017b).
Lingmotif results are generated as a HTML/Javascript document, which is saved Name tss tsi sent.it pos.sc neg.sc pos.it neg.it neu.it split1.tss split2.tss sentences shifters
Description Text Sentiment Score Text Sentiment Intensity Number of lexical Items Positive score Negative score Number of positive items Number of negative items Number of neutral items TSS for split 1 of text TSS for split 2 of text Number of sentences
Number of sentiment shifters locally to a prede ned location and automatically sent to the user's default browser for immediate display. Internally, the application generates results as an XML document containing all the relevant data; this XML document is then parsed against one of several available XSL templates, and transformed into the nal HTML.
Lingmotif Learn simply plugs into the internally generated XML document to retrieve the desired sentiment analysis data, and appends the data to each tweet as features. 2.2
Table 1 summarizes the sentiment-related feature set generated by the Lingmotif engine.
Most of these features are included in the original Lingmotif engine, but for this occasion we experimented with text splits to test the relevance of the position of the sentiment words in the tweet. The features split1.tss and split2.tss are the combined sentiment score for each half of the tweet. The assumption was that sentiment words used towards the end of the tweet may have more weight on the overall tweet polarity. This might be helpful especially for the P/N/NEU distinction. Neutral tweets are supposed to have some balance between positivity and negativity. In our tests with the training set, however, adding these features did not improve results. We also experimented with 3 splits, with the same results. These features were thus discarded for test set classi cation.
Some of these features are in fact redundant. Notably, tss already encapsulates pos.sc, neg.sc, and neu.it. In our tests, the classi er performed better using just the pos.sc and neg.sc values, than our calculated tss, so we only used these two features.
Name sentences tt.ratio lex.items gram.items vb.items nn.items nnp.items jj.items rb.items chars intensifiers contrasters emoticons all.caps char.ngrams x.marks q.marks quote.marks susp.marks x.marks.seqs q.marks.seqs xq.marks.seqs handles hashtags urls
Description Number of sentences Type/Token ratio Number of lexical items Number of grammatical items Number of verbs Number of nouns Number of proper nouns Number of adjectives Number of adverbs Number of characters Number of intensi ers Number of contrast words Number of emoticons/emojis Number of upper case words Number of character ngrams Number of exclamation marks Number of question marks Number of quotation marks Number of suspension marks Number of x.marks sequences Number of q.marks sequences Number of x/q marks sequences Number of Twitter handles Number of hashtags
Number of URL's
Raw text features are commonly used in
sentiment analysis shared tasks successfully (e.g.
Table 2 contains the full list of text features we experimented with. 2.4
In order to account for words and expressions that convey sentiment but may not be included in the sentiment lexicon, we experimented with automatic keyword extraction for each of the classes in the training set. Automatic keyword and keyphrase extraction is a well developed eld and a number of tools and methodologies have been proName p.kw p.ng.kw p.handles n.kw n.ng.kw n.handles neu.kw neu.ng.kw neu.handles none.kw none.ng.kw none.handles
Description Positive keywords Positive ngram keywords Positive handles Negative keywords Negative ngram keywords Negative handles Neutral keywords Neutral ngram keywords Neutral handles None keywords None ngram keywords
None handles posed. Hasan and Ng (2014) provide a good overview of the state-of-the-art techniques for keyphrase extraction.
We used a very simple approach that consisted in comparing frequencies of single words and ngrams (2 to 4 words) on a onevs-rest basis for each of our four classes, for words and ngrams with a minimum frequency of 2. We calculated and ranked keyness based on the chi-square statistic, and then manually removed irrelevant results. We ended up with a list of 100 keywords and 100 keyphrases for each class. We did the same for Twitter handles.
Using the keywords feature set improved results considerably in our tests with the training set. However, this improvement did not transfer well to the test sets, especially in the case of the InterTASS dataset. We further discuss this issue in section 3. 3
Tables 4, 5, and 6 show our results for each of the test sets. Although performance is strong across all three, there clearly is a di erence between the General TASS datasets, on the one hand, and the InterTASS dataset on the other.
sent-only run3 sent-only- xed
Macro-F1 0.456 0.441 0.441
Accuracy 0.582 0.576 0.595
We believe this is due to two main reasons. First, the General training set (7,218 tweets) is much larger than the InterTASS training set (1,514 tweets, using both the training and development datasets). This of course provides a much more solid training base for
run3
nal no ngrams the former than the latter. All our models were trained on one dataset where both training datasets (General and InterTASS) where merged. Perhaps better results would have been obtained by training on each dataset separately.
The other reason for poorer performance on the InterTASS test set concerns the very di erent nature of the datasets. The General Corpus of TASS consists of tweets generated by public gures (artists, politicians, journalists) with a large number of followers. Such Twitter users are more predictable both in terms of the content of their tweets and the language they use. They are also Castilian Spanish speakers entirely. Most of these tweets contain very compact but carefully chosen language, expressing users' opinion or evaluation of polically or socially relevant events. On the other hand, the interTASS corpus shows much more variability; rst, the tweets were collected not only from Spain, but from several Latin American countries, which introduces important lexical variability. Second, no user selection is apparent. Tweets were randomly collected from the whole Spanish speaking user base. This introduces spelling errors and a much more colloquial and chatty language. Nonlexical linguistic features, such as exclamation marks, emojis or emoticons, are recurrent, as are, user-to user messages, which are of course hard-to-decode, since they presuppose certain privately shared knowledge. These issues have obviously a ected the performance of all TASS participants, as is clear from the nal leader board.
We obtained the best results for the General datasets with our run3 experiment, where we combined a selection of features from the three feature sets listed in tables pos.sc neg.sc vb.items jj.items rb.items gram.items n.chars intensifiers contrasters p.kw p.ng.kw p.handles n.kw n.ng.kw n.handles
Features neu.kw neu.ng.kw neu.handles none.kw none.ng.kw none.handles emoticons all.caps char.ngrams x.marks q.marks susp.marks hashtags handles urls 1, 2, and 3. This selection was in fact the optimal we found during our cross-validation tests on the training dataset. Table 7 lists the feature set used in this experiment.
Concerning the InterTass test set, the best results were obtained with the sent-only experiment, where a reduced set of features was used. We list these features in table 8.
We obtained better results for the InterTASS test set using this reduced set of features because the keyword sets were causing noise, since they were extracted using the whole training set, which contained a much larger proportion of tweets from the General TASS dataset.
Another important aspect is the large difference that we encountered between our own tests on the training datasets and our nal (o cial) results. For the General corpus of TASS, we consistently obtained very high F1 scores (upwards of 0.73) using the keyword set, but much closer to the o cial results without them. This is a clear indication of model over tting, with an obvious negative impact on the classi cation of the test set. After this became apparent on our rst results upload, we corrected by reducing the sets of keywords, keyphrases and user handles, which resulted in better overall results. 4
This shared task has served us to assess the usefulness of many di erent features as predictors of polarity classi cation in Spanish tweets. The di ering sizes and characteristics of the training and test datasets determined to some extent our results, but we also felt we over tted our model with too large a selection of keywords, which threw overoptimistic results in our tests.
Our results on par with other participants who used more sophisticated systems from the technical perspective, which is also an indication of the salient role that curated, high-quality lexical resources play in sentiment analysis.
We also experienced the negative impact of model over tting and learnt how to limit its e ects. We plan to use this knowledge in future versions of Lingmotif, which currently uses sentiment features exclusively. It is obvious that combining those with other formal features can improve results considerably.
This research was supported by Spain's MINECO through the funding of project Lingmotif2 (FFI2016-78141-P). corpus-based approaches to sentiment tagging. In Proceedings of the 4th International Workshop on Semantic Evaluations, SemEval '07, pages 117{120, Stroudsburg, PA, USA. Association for Computational Linguistics.
Aue, A. and M. Gamon. 2005. Customizing sentiment classi ers to new domains: A case study. Borovets, Bulgaria.
Ceron-Guzman, J. A. 2016. Jacerong at tass 2016: An ensemble classi er for sentiment analysis of spanish tweets at global level. In Proceedings of TASS 2016: Workshop on Sentiment Analysis at SEPLN co-located with 32nd SEPLN Conference (SEPLN 2016), pages 35{39, Salamanca, Spain. SEPLN.
Choi, Y. and C. Cardie. 2008. Learning with compositional semantics as structural inference for subsentential sentiment analysis. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP '08, pages 793{801, Stroudsburg, PA, USA.
Choi, Y., Y. Kim, and S.-H. Myaeng. 2009.
Domain-speci c sentiment analysis using contextual feature generation. In Proceeding of the 1st international CIKM workshop on Topic-sentiment analysis for mass opinion, pages 37{44, Hong Kong, China.
ACM.
Fellbaum, C., editor. 1998. WordNet An Electronic Lexical Database. The MIT Press, Cambridge, MA; London, May. Hasan, K. S. and V. Ng. 2014. Automatic keyphrase extraction: A survey of the state of the art. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1262{1273.
Hatzivassiloglou, V. and K. R. McKeown. 1997. Predicting the semantic orientation of adjectives. In Proceedings of the eighth conference on European chapter of the Association for Computational Linguistics, pages 174{181, Madrid, Spain. Association for Computational Linguistics.
Hu, M. and B. Liu. 2004. Mining and summarizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 168{177, Seattle, WA, USA. ACM.
Kennedy, A. and D. Inkpen. 2006. Sentiment classi cation of movie reviews using contextual valence shifters. Computational Intelligence, 22(2):110{125.
Kim, S.-M. and E. Hovy. 2004. Determining the sentiment of opinions. In Proceedings of the 20th international conference on Computational Linguistics, page 1367, Geneva, Switzerland. Association for Computational Linguistics.
Kiritchenko, S., X. Zhu, C. Cherry, and S. Mohammad. 2014. Nrc{canada-2014: Detecting aspects and sentiment in customer reviews. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 437{ 442, Dublin, Ireland, August. Association for Computational Linguistics and Dublin City University.
Lu, Y., M. Castellanos, U. Dayal, and C. Zhai. 2011. Automatic construction of a context-aware sentiment lexicon: An optimization approach. In Proceedings of the 20th International Conference on World Wide Web, WWW '11, pages 347{356, New York, NY, USA. ACM.