This paper describes the `oeg' submission to task 1 of the TASS 2017 workshop, focusing on Sentiment Analysis at tweet level. Di erent parameters and systems were tested in each one of the three corpora released for the task, including di erent Machine Learning algorithms and morphosyntactic analyses for negation detection, along with the use of lexicons and dedicated preprocessing techniques for detecting and correcting frequent errors and expressions in tweets. The obtained results o er a basis for the design of future strategies for systems to tackle Sentiment Analysis in Twitter.
Recent boom on Sentiment Analysis, partly
due to Natural Language Processing (NLP)
new techniques and to the wide use of
social networks in everyday life by Internet
users, has derived into the creation of new
resources and techniques to analyze opinions
in almost every possible eld. One of the
evidences of this growing importance of
Opinion Mining is its use by brands in order to
discover customer's opinions. These opinions
can be found in posts in the social media,
being possible to some extent to
automatically evaluate their polarity and the impact of
marketing campaigns. According to Nielsen
The OEG, together with Havas
Media, has participated in the LPS-BIGGER
project1, where software components have
been developed for the categorization of
brand-related messages into four categories
framed in marketing analysis, being one of
them a sentiment analysis task. This
software is capable of classifying Twitter
messages in Spanish and English into one or
more of eight pre-de ned emotions
(lovehate, satisfaction-dissatisfaction, trust-fear,
happiness-sadness). An adaptation of this
infrastructure has been used to detect polarity
in the Spanish messages proposed by Task
1 of the TASS 2017 workshop. The TASS
workshop
Copyright © 2017 by the paper's authors. Copying permitted for private and academic purposes. become one of the rst e orts to cover Sentiment Analysis in Spanish, challenging since 2012 both researchers and industry to analyze di erent annotated or tagged corpora.
In this edition, two tasks have been proposed, but the OEG participation just covers the
rst one, dealing with Sentiment Analysis at tweet level.
The reminder will be as follows. Section 2 covers related works in the area, including both general approaches and proposals. Section 3 exposes our proposals in detail. Section 4 includes the results and an analysis on them, and Section 5 presents our conclusions on our participation and future lines. 2
Several authors, such as Pang and Lee
Machine Learning systems
Hybrid Systems
All these kind of systems usually rely on
lexicons to enrich post representation, such as
SentiWordnet
Additionally, analyzing social media posts
has a complex plus due to the usual presence
of irony and other linguisitc phenomena of
the sort
Nevertheless studies and resources on
languages di erent from English are scarce.
Some examples of works for Spanish are the
adaptation performed in
The described system uses a Machine Learning approach with a Java-based standard pipeline. Some linguistic aspects such as negation and some di erent labeling strategies have been explored in more depth. 3.1
A binary classi cation system assigns or not a label to a text document, e.g. the classi cation of an email text as ham/spam. The labeling strategy is a straightforward operation {when a threshold is reached the message is categorised as spam, otherwise the message is classi ed as ham. In case of classifying the polarity of a text, the labeling strategy is somewhat more complex, as the number of possible results is at least three, positive, negative or neutral. For the TASS task 1, there is one category more: \no sentiment", which is di erent from neutral.
In order to classify a message as P (positive), N (negative), NEU (neutral) and NONE (no polarity), di erent strategies can be followed: 1. Labeling each document with one of the four categories independently and using four binary classi ers. The category is assigned to the class with the highest score among the four results. In order to have an optimal Bayesian classi er, the results for each class can be weighted by their a priori probabilities. 2. Labeling each document as either positive/non-positive and negative/nonnegative. Two binary classi ers are used, and depending on the individual classi er results (P and N) the classi ed category is determined as follows:
and N (this is, the distance d = jP N j is less than some threshold td), we can consider it as neutral (NEU).
P and N but not extremely close to zero (this is, N and P values area between some some thresholds tnmin < P < tnmax and tnmin < N < tnmax with 0 < tnmin ), we can consider it as neutral (NEU).
If a tweet has close-to-zero values
for P and N (this is, N and P are
such that P tnmin and N
tnmin ), we can consider it as NONE.
3. Labeling each document with one of the
three tags NEU, P and N, and then using
one single classi er. The classi er score
determines if the document is classi ed
as P (high values), as N (low values) or
as NEU (intermediate values). The
system would never return NONE (which is
only present in 14% of the documents in
the InterTASS corpus). Ignoring neutral
values has been reported as a bad
strategy, as studied by
During our processing, diverse linguistic features were taken into account. 3.2.1 Features and negation
treatment We tested two di erent lexical units in our systems: tokens and lemmas. In some tested systems we also considered as features words extracted from lexicons, and also the presence of negation in verbs, detected by using trees extracted by deep syntactic analysis.
Negation was tackled by detecting the
presence of \NEG" constituents in the
verbal groups, with the logic that if a negation
is present within a verbal group, the polarity
is inverted. Double negation was not
considered.
3.2.2 Preprocessing
For preprocessing the tweets, often full of
grammar errors and social networks
expressions that are highly decisive in polarity (such
as emoticons), we have developed a lter able
to detect these phenomena partly, similar to
the one described by Quiros et al.
(\hahaha", URL formats (in order to delete them, since they give no information).
Slang expressions and replacements in Spanish social networks, such as: { `q', `k', `qu', `ke', `qe' ! `que'. { `d' ! `de'. { `tb' ! `tambien'. { `lol' ! `ja'.
{ `xq', `pq', `porq' ! ` porque' .
Typos related to repeated letters (`LOOOOOL' ! `LOL' ) Suppression of numbers, as they just tend to carry polarity in concrete expressions.
Emoticons detection and polarization, such as: { positive polarity : f `:-)', `;)', `:D', `<3', `:<', `:P', `o:', `*.*' g. { negative polarity : f `:'(', `:S', `:$', `: (', `:C' g.
Additionally, for some systems a stopword lter was tested. This lter based on a list of common words that carry no semantic or polarity meaning, created combining the results of algorithms (such as TF-IDF) and manual revision. 3.3
3.3.1 External resources
IXA-pipes
In order to assess the a ectivity of
documents, a dataset of Spanish words and their
arousal (the level of activation or intensity
that a stimulus elicits) and valence (how
pleasant a stimulus is) was also used. The 875
words studied by Hinojosa et al.
Nave Bayes is a generative model assuming that features are independent given a class and that calculates the probability given a class using Bayes theorem. Even when independence does not happen in Natural Language, this technique has shown to deliver good results in NLP. Multinomial version of Nave Bayes is speci cally performant when dealing with language, being lexical units (words, lemmas...) frequency the data frequently used. Also SMO (Sequential Minimal Optimization for Support Vector Machines) classi ers have been tested, but results did not improve those from Nave Bayes. 4
We present now the con gurations that
turned out to be the best ones among the
di erent options exposed before:
laOEG implements the second
labelbased strategy described in section 3.1,
which performed better than any of the
other approaches. Several thresholds
were tested: Multinomial Nave Bayes
algorithm's threshold was set to 0:30
(also executions with values from 0:01 to
0:50 were performed), while di erent
values were tested for internal thresholds,
with values such as td = 0:10, tnmax =
0:15 and tnmin = 0:10. The feature
vector was built simply using tokens after
the pipeline described above. The
Multinomial Nave Bayes performed slightly
better than the SMO classi er, being
also one of its strengths its versatility
(training and features can be easily
managed in di erent manners) and its
velocity, being the fastest system among the
proposed; however, besides these strong
aspects, it is also the simplest approach,
processing just shallowly at token level
and being therefore unable to detect
nuances such as negation or irony.
victor0 Same as above, but using
lemmas in the feature vector and
considering negation. The presence of
negation in the verbal group at di erent
constituent levels led to the addition of extra
features (e.g. `don't like' is handled as a
single feature instead as three words or
lemmas in a bag) .
victor2 Same as victor0, but also
considers the presence of stopwords and
using the Hinojosa dataset
Numerical results of the systems proposed by OEG are exposed in Table 1 (for the InterTASS corpus), Table 2 (for the full test General Corpus of TASS) and Table 3 (for the 1k General Corpus of TASS), along with the highest and the lowest results of the overall of the participants for each corpus.
System victor2 victor0 laOEG Max. result Min. result The mere adaptation of a di erently purposed classi er does not yield optimal results for the TASS challenge. However, our results have proved to be one of the most stable among the three corpora for testing, since we obtained similar results with each of the systems in all of them, fact that is not common to other participant's proposals. Groups being the rst classi ed in a corpus can be in the last half of the ranking in another one. We acknowledge that corpora from previous TASS editions should have been used, and that additional machine learning approaches, 2https://www.ibm.com/watson/developercloud /doc/natural-language-understanding/
System victor2 laOEG Max. result Min. result
External out-of-the-box software did not prove to work any better. IBM Waton's Natural Language Understanding was tested, because no train is needed, Spanish language is covered and emotion and sentiment analysis functionality is ready to be used. However, its results did not prove any better than the system described in this paper.
The distinction between NEU and NON is a very speci c feature of this challenge that justi es speci c research on the strategies presented in Section 3.1. Also, in future TASS editions, special focus will be given to word sense disambiguation, introducing concepts as tokens rather than simple words or n-grams; and we will extensively use the broader sentilexicons newly appeared.
This work has been partially supported by LPS-BIGGER (IDI-20141259, Ministerio de Econom a y Competitividad), a research assistant grant by the Consejer a de Educacion, Juventud y Deporte de la Comunidad de Madrid partially founded by the European Social Fund (PEJ16/TIC/AI-1984) and a Juan de la Cierva contract.