This article presents the participation of the Intelligent Systems Group (GSI) at Universidad Politecnica de Madrid (UPM) in the Sentiment Analysis workshop focused in Spanish tweets, TASS2015. This year two challenges have been proposed, which we have addressed with the design and development of a modular system that is adaptable to di erent contexts. This system employs Natural Language Processing (NLP) and machine-learning technologies, relying also in previously developed technologies in our research group. In particular, we have used a wide number of features and polarity lexicons for sentiment detection. With regards to aspect detection, we have relied on a graph-based algorithm. Once the challenge has come to an end, the experimental results are promising.
The rst task of this challenge, Task
1
This task provides a corpus
The second and last task, Task 2
(VillenaRoman et al., 2015b), is aimed to detect
the sentiment polarity at an aspect level
using three labels (P, N and NEU). Within
this task, two corpora
The complexity presented by the challenge has taken us to develop a modular system, in which each component can work separately.
We have developed and experimented with each module independently, and later combine them depending on the Task (1 or 2) we want to solve.
The rest of the paper is organized as follows. First, Section 2 is a review of the research involving sentiment analysis in the Twitter domain. After this, Section 3 brie y describes the general architecture of the developed system. Following that, Section 4 describes the module developed in order to confront the Task 1 of this challenge. After this, Section 5 explains the other modules necessaries to address the Task 2. Finally, Section 6 concludes the paper and presents some conclusions regarding our participation in this challenge, as well as future works. 2
Centering the attention in the scope of TASS,
many researches have experimented, through
the TASS corpora, with di erent approaches
to evaluate the performance of these systems.
One of ours main goals is to design and
develop an adaptable system which can
function in a variety of situations. As we have
already mentioned, this has taken us to a
system composed of several modules that can
work separately. Since the challenge
proposes two di erent tasks
Our system is divided into three modules:
module. The NER module detects the entities within a text, and classi es them as one of the possibles entities. In the Section 5 a more detailed description of this module and the set of entities given is presented, as it is used in the Task 2. Aspect and Context detection module. This module is in charge of detecting the remaining aspects -aspects that are not entities and therefore can not be detected as such- and the contexts of all aspects. In the Section 5 this module is described in greater detail since it is only used for tackling the Task 2.
name suggests, the goal of this module is to classify the given texts using sentiment polarity labels. This module is based on combining NLP and machine learning techniques and is used in both Task 1 and 2. It is explained in more detail next. 3.1
The sentiment analysis module relies in a
SVM machine-learning model that is trained
with data composed of features extracted
1www.en.wikipedia.org/wiki/2014 Copa del Rey Final from the TASS dataset: General corpus for
the Task 1 and SocialTV corpus for Task
2
We have used di erent approaches to design the feature extraction. The reference document taken in the development of the features extraction was made by Mohammad, Kiritchenko, and Zhu (2013). With this in mind, the features extracted from each tweet to form a feature vector are:
N-grams, combination of contiguous sequences of one, two and three tokens consisting on words, lemmas and stem words. As this information can be difcult to handle due to the huge volume of N-grams that can be formed, we set a minimum frequency of three occurrences to consider the N-gram.
All-caps, the number of words with all characters in upper cases that appears in the tweets.
POS information, the frequency of each part-of-speech tag.
Hashtags, the number of hashtags terms. Punctuation marks, these marks are frequently used to increase the sentiment of a sentence, specially on the Twitter domain. The presence or absence of these marks (?!) are extracted as a new feature, as well as its relative position within the document.
Elongated words, the number of words that has one character repeated more than two times.
Emoticons, the system uses a Emoticons
Sentiment Lexicon, which has been
developed by
The best way to increase the coverage
range with respect to the detection of
words with polarity is to combine
several resources lexicon. The lexicons used
are: Elhuyar Polar Lexicon
Intensi ers, a intensi er
dictionary
Global Polarity, this score is the sum of the punctuations from the emoticon analysis and the lexicon resources.
An important feature that has been used to develop the classi er is the treatment of the negations. This approach takes into account the role of the negation words or phrases, as they can alter the polarity value of the words or phrases they precede.
The polarity of a word changes if it is
included in a negated context. For
detecting a negated context we have utilized a
set of negated words, which has been
manually composed by us. Besides, detecting the
context requires deciding how many tokens
are a ected by the negation. For this, we
have followed the proposal by
Once the negated context is de ned there
are two features a ected by this: N-grams
and lexicon. The negation feature is added
to these features, implying that its negated
(e.g. positive becomes negative, +1 becomes
-1). This approximation is based on the work
by
In this competition it is allowed for submission up to three experiments for each corpus. With this in mind, three experiments have been developed in this task attending to the lexicons that adjust better to the corpus: RUN-1, there is one lexicon that is adapted well to the corpus, the ElhPolar lexicon. It has been decided to use only this dictionary in the rst run. RUN-2, in this run the two lexicons that have the best results in the experiments have been combined, the ElhPolar and the ISOL.
RUN-3, the last run is a mix of all the lexicon used on the experiments.
6labels 6labels-1k 4labels 4labels-1k This task is an extension of the Task 1 in which sentiment analysis is made at the aspect level. The goal in this task is to detect the di erent aspects that can be in a tweet and afterwards analyze the sentiment associated with each aspect.
For this, we used a pipeline that takes the provided corpus as input and produces the sentiment annotated corpus as output. This pipeline can be divided into three major modules that work in a sequential manner: rst the NER, second the Aspect and Context detection, and third the Sentiment Analysis as described below. 5.1
NER The goal of this module is to detect the words that represent a certain entity from the set of entities that can be identi ed as a person (players and coaches) or an organization (teams).
For this module we used the Stanford CRF
NER
As the goal of the NER module is to detect the words that represent a speci c entity, we used a list of all the ways these entities were named. In this way, once the Stanford NER detect the general entity our improved NER module search in this list and decides the particular entity by matching the pattern of the entity words. 5.2
This module aims to detect the aspects that
are not entities, and thus have not been
detected by the NER module. To achieve
this, we have composed a dictionary using
the training dataset
As for the context detection, we have
implemented a graph based algorithm
Combining this two approaches -aspect and context detection- this module is able to detect the word or words which identify an aspect, and extract the context of this aspect. This context allows us to isolate the sentiment meaning of the aspect, fact that will be very interesting for the sentiment analysis at an aspect level.
We have obtained an accuracy of 93.21%
in this second step of the pipeline with
the training dataset
The sentiment analysis module is the end of the processing pipeline. This module is in charge of classifying the detected aspects in polarity values through the contexts of each aspect. We have used the same model used in Task 1 to analyse every detected aspect in Task 2, given that the detected aspect contexts in Task 2 are similar to the texts analysed in Task 1.
Nevertheless, though using the same model, it is needed to train this model with the proper data. For this, we extracted the aspects and contexts from the train dataset, process the corresponding features (explained in Section 3), and then train the model with these. In this way, the trained machine is fed contexts of aspects that will classify in one of the three labels (as mentioned: positive, negative and neutral). 5.4
By means of connecting these three modules together, we obtain a system that is able to recognize entities and aspects, detect the context in which they are enclosed, and classify them at an aspect level. The performance of this system is showed in the Table 4. The different RUNs represent separate adjustments of the same experiment, in which several parameters are controlled in order to obtain the better performance.
As can be seen in Table 4, the global performance obtained is fairly positive, as our 2We calculated this metric using the output granted by the TASS uploading page www.daedalus.es/TASS2015/private/evaluate.php.
RUN-2 RUN-3
62.1 55.7
In this paper we have described the
participation of the GSI in the TASS 2015 challenge
As future work, our aim is to improve
aspect detection by including semantic
similarity based on the available lexical resources in
the Linguistic Linked Open Data Cloud. To
this aim, we will integrate also vocabularies
such as Marl
This research has been partially funded and by the EC through the H2020 project MixedEmotions (Grant Agreement no: 141111) and by the Spanish Ministry of Industry, Tourism and Trade through the project Calista (TEC2012-32457). We would like to thank Maite Taboada as well as the rest of researchers for providing us their valuable lexical resources.