This article describes our system presented at the workshop for sentiment analysis TASS 2015. Our system approaches the task 1 of the workshop, which consists on performing an automatic sentiment analysis to determine the global polarity of a set of tweets in Spanish. To do this, our system is based on a model supervised Linear Support Vector Machines combined with some polarity lexicons. The in uence of the di erent linguistic features and the di erent sizes of n-grams in improving algorithm performance. Also the results obtained, the various tests that have been conducted, and a discussion of the results are presented.
Since the origin of Web 2.0, Internet contains a very large amounts of user-generated information on an unlimited number of topics.
Many entities such as corporations or political groups try to learn about that knowledge to know the opinion of users. Social Media platforms such as Facebook or Twitter have proven to be useful for this tasks,due to the very high volume of messages that these platforms generate in real time and the very large number of users that use them everyday.
Faced with this challenge, in the last years the number of the Sentiment Analysis researches has increased appreciably, especially those based in Twitter and microblogging.
It should be taken into account that the
performance of these researches is
languagedependent, re ecting the considerable di
erences between languages and the di culty of
establish standard linguistic rules
In this context, the TASS1 workshop
This paper presents a global polarity clas1Taller de Analisis de Sentimientos en la SEPLN 2http://www.sepln.org/ Publicado en http://ceur-ws.org/Vol-1397/. CEUR-WS.org es una publicación en serie con ISSN reconocido si cation in Spanish tweets based on polarity lexicons and linguistic features. It is adapted to Spanish tweet texts, which involve particular linguistic characteristics like short length, limited to 140 characters, slang, spelling and grammatical errors and other user mentions.
The rest of the paper is organized as follows: the sentiment analysis related works are described in Section 2, the developed system's description is presented in Section 3, evaluation and results in Section 4 and conclusion and future work are discussed in Section 5. 2
There exists a large amount of literature
addressing the sentiment analysis eld,
especially applied to Twitter and microblogging
context. General surveys about Opinion
Mining and Sentiment Analysis may be found
Twitter has certain speci c
characteristics which distinguish them from other
social networks, e.g. short texts, @user
mentions, #hashtags and retweets. All of these
characteristics have been extensively studied
For several years we assist to an
exponential increase of studies based on
sentiment analysis and opinion mining in
Twitter. According to the state of art, two main
approaches exist in sentiment analysis:
supervised learning and unsupervised learning.
Supervised systems implement classi cation
models based on classi cation algorithms,
being the most frequent the Support Vector
Machine (SVM)
During the last TASS workshop in 2014
Several tools and datasets have been used
during the experiments to develop our nal
system. Because our system only approaches
the Task 1: Sentiment Analysis at global
level, this consists in a unique pipeline that
reaches the process completely. At the
beginning, a naive normalization system is
applied to the tweet texts with the purpose to
standardize several Twitter own features,
like #Hashtags or @User mentions. Then, the
Freeling language analysis tool3
During this step, based on a list of stop words for Spanish language, this words are annotated to be ignored by polarity ranking steps.
The task has been addressed as an
automatic multi-class classi cation job. For this
reason, it has been considered appropriate to
focus this problem with a one-vs-all strategy,
in a similar way to the presented by
To represent the text's as vectorized features, two main sources have been used: the polarity lexicon punctuations and the Okapi BM25 ranking function, to represent document's scoring (Robertson et al., 1995). BM25 is a bag-of-words retrieval function that ranks a set of documents based on the query terms appearing in each document. The formula used to implement BM25 in the system is de ned below: score(D; Q) = n X IDF (qi) T F (qi) i=1 f (qi; D) (k1 + 1) b + b ajvDgdjl ) (2) T F (qi) =
f (qi; D) + k1 (1 IDF (qi) = log
N
n(qi) + 0;5 n(qi) + 0;5
To calculate the score of a document D, f (qi; D) is the frecuency of each word lemma (qi), jDj is the length of the text D in words and avgdl is the average text length. After several experiments over the training corpus, the free parameters k1 and b have been optimized to k1 = 76 and b = 0;75. System develops one BM25 dictionary for each one-vs-all classi er.
In conjunction with the document's score, each tweet has been represented using different polarity lexicons in order to classify them into the six (P+, P, NEU, N, N+ and NONE) and the four (P, N, NEU and NONE) polarities. We use several datasets to score the polarity levels of words and lemmas. Owing to di erent characteristics of each dataset, such as semantic-orientation values, scores are calculated separately and considered as independent attributes in the system. (1) (3)
is an automatically-built polarity lexicon
for Spanish language that was created
by downloading messages from Twitter,
and includes both negative and
positive Spanish words
Polar polarity lexicon for Spanish was
created from di erent sources, and
includes both negative and positive words
The Spanish Opinion Lexicon
(SOL). The Spanish Opinion Lexicon
(SOL) is composed by 1,396 positive
and 3,151 negative words, thus in
total SOL has 4,547 opinion words4
Negation Words List. A list of negation spanish words has been created during the experiments. This list is used as a text feature in order to detect negative sentences and possible polarity inversions.
We also consider other text characteristics as classi er features, like text length in words quantity or a list of sentiments represented by emoticons using the Wikipedia's list of emoticons5. To conclude the system's prediction, another automatic classi er has been implemented, trained with the predictions of the binary results to select one label. 4
Our results are relative to the Task 1: Sentiment Analysis at global level of TASS 2015. This task consists on performing an automatic sentiment analysis to determine the global polarity of each message in the provided corpus. There are two di erent evaluations: one based on 6 di erent polarity labels (P+, P, NEU, N, N+, NONE) and another based on just 4 labels (P, N, NEU, NONE). Also there are two test sets: complete set and 1k
set, a subset of the rst one containing only 1000 tweets with a similar distribution to the training corpus was extracted to be used for an alternate evaluation of the performance of systems.
Tables 1 and 2 show the performance of di erent tested models using the full and 1k sets. For the rating of the developed system, 3 di erent systems have been presented for each subtask. Our submitted models consist in di erent features as follows:
Run 1: Words and lemmas based polarity dictionaries as features, di ering between positive and negative scores and between di erent datasets. Okapi BM25 scores of mono-grams used as features with the lemmas of the tweet texts. Binary classi ers were implemented using LinearSVC Machines and the global classi er uses their predictions (True or False).
Run 2: Words and lemmas based polarity dictionaries as features, di ering between positive and negative scores and between di erent datasets. Okapi BM25 scores of mono-grams and bi-grams used as features with the lemmas of the tweet texts. Binary classi ers were implemented using LinearSVC Machines and the global classi er uses their predictions (True or False).
Run 3: Similar to Run 2, with the exception of the binary classi ers that were implemented using Support Vector Regression (SVR) Machines and the global classi er uses their predictions (0 to 1 oat values).
Run1 Run2 Run3 Run1 Run2 Run3
Run1 Run2 Run3 Run1 Run2 Run3 based in SVR. This suggests that the precision of the regression values, in contrast with the binary values of the SVM classi ers, has a negative impact on the global classi er. However, the use of mono-grams and bi-grams as features presents di erent success rates depending of the test. This part of the system must be analysed in-depth in order to comprehend the performance di erence between both systems. 5
This paper describes the participation of the DeustoTech Internet research group in the Task 1: Sentiment Analysis at global level at TASS 2015. In our rst participation, our team presents a system based in Support Vector Machines in conjunction with several well established polarity lexicons. Experimental results present a good baseline to continue working through the development of new models and developing an structure able to take full advantage of multiple supervised learning systems.
As future work, we propose to research on di erent approaches to aboard the measure of sentiment analysis problems, especially those related to sentiment degrees with the aim to detect clearly di erences between different sentiment levels (Good vs Very Good, for example).
For further work, we would like to improve the present system including some steps previously to the classi er module, that have been demonstrated to improve the nal results like a normalization pipeline based on tweets. Also, the necessity of improving the tokenization module to include features like punctuation signs, web addresses, and named entities has become apparent.
The research activities described in this paper are funded by DeustoTech INTERNET, Deusto Institute of Technology, a research institute within the University of Deusto. Robertson, S.E., S. Walker, S. Jones, Hancock-Beaulieu M. M., and Gatford M. 1995. Okapi at trec-3. NIST SPECIAL PUBLICATION SP, 109-109.