ISSN: 1613-0073
Organización Comité organizador Julio Villena-Román Miguel Á. García Cumbreras Eugenio Martínez Cámara Manuel C. Díaz Galiano M. Teresa Martín Valdivia L. Alfonso Ureña López CEUR Workshop Proceedings ISSN: 1613-0073 Editado en:Universidad de Jaén Año: 2016 Editores: Julio Villena-Román Sngular julio.villena@sngular.team Miguel Á. García Cumbreras Universidad de Jaén magc@ujaen.es Eugenio Martínez Cámara TU Darmstadt camara@ukp.informatik.tu-darmstadt.de Manuel C. Díaz Galiano Universidad de Jaén mcdiaz@ujaen.es M. Teresa Martín Valdivia Universidad de Jaén maite@ujaen.es
L. Alfonso Ureña López Universidad de Jaén laurena@ujaen.es Publicado por: CEUR Workshop Proceedings
Alexandra Balahur José Carlos Cortizo Jose María Gómez Hidalgo José Carlos González-Cristobal Lluís F. Hurtado Carlos A. Iglesias Fernández Zornitsa Kozareva Sara Lana Serrano Ruslan Mitkov Andrés Montoyo Rafael Muñoz Constantine Orasan Jose Manuel Perea Ortega Ferran Pla Santamaría María Teresa Taboada Gómez Mike Thelwall José Antonio Troyano Jiménez EC-Joint Research Centre (Italia) Universidad Europea de Madrid (España) Optenet (España) Universidad Politécnica de Madrid (España) Universidad de Valencia (España) Universidad Politécnica de Madrid (España) Information Sciences Institute (EE.UU.) Universidad Politécnica de Madrid (España) University of Wolverhampton (Reino Unido) Universidad de Alicante (España) Universidad de Alicante (España) University of Wolverhampton (Reino Unido) Universidad de Extremadura (España) Universidad de Valencia (España) Simon Fraser University (Canadá) University of Wolverhampton (Reino Unido) Universidad de Sevilla (España)
La organización de TASS ha contado con la colaboración de investigadores que participan en los siguiente proyectos de investigación: • REDES (TIN2015-65136-C2-1-R)
ISSN: 1613-0073 Preámbulo Actualmente el español es la segunda lengua materna del mundo por número de hablantes tras el chino mandarín, y la segunda lengua mundial en cómputo global de hablantes. Esa segunda posición se traduce en un 6,7% de población mundial que se puede considerar hispanohablante. La presencia del español en el mundo no tiene una correspondencia directa con el nivel de investigación en el ámbito del Procesamiento del Lenguaje Natural, y más concretamente en la tarea que nos atañe, el Análisis de Opiniones. Por consiguiente, el Taller de Análisis de Sentimientos en la SEPLN (TASS) tiene como objetivo la promoción de la investigación del tratamiento del español en sistemas de Análisis de Opiniones, mediante la evaluación competitiva de sistemas de procesamiento de opiniones.
En la edición de 2016 han participado 7 equipos, de los que 6 han enviado un artículo describiendo el sistema que han presentado, habiendo sido aceptados los 6 artículos tras ser revisados por el comité organizador. La revisión se llevó a cabo con la intención de publicar sólo aquellos que tuvieran un mínimo de calidad científica.
La edición de 2016 tendrá lugar en el seno del XXXII Congreso Internacional de la Sociedad Española para el Procesamiento del Lenguaje Natural, que se celebrará el próximo mes de septiembre en Salamanca (España) dentro del V Congreso Español de Informática (CEDI 2016). Septiembre de 2016
Los editores CEUR Workshop Proceedings Currently Spanish is the second native language in the world by number of speakers after the Mandarin Chinese. This second position means that the 6.7% of the world population is Spanish-speaking. The presence of the Spanish language in the world has not a direct correspondence with the number of research works related to the treatment of Spanish language in the context of Natural Language Processing, and specially in the field of Sentiment Analysis. Therefore, the Workshop on Sentiment Analysis at SEPLN (TASS) aims to promote the research of the treatment of texts written in Spanish in Sentiment Analysis systems by means of the competitive assessment of opinion processing systems.
Seven teams have participated in the 2016 edition of the workshop. Six of the seven teams have submitted a description paper of their systems. After a review process, the organizing committee has accepted the 6 papers, because all of them reached an acceptable scientific quality level. The 2016 edition will be held at the 32nd International Conference of the Spanish Society for Natural Language Processing (SEPLN 2016), which will take place at Salamanca in September framed by the 5th Spanish Conference of Computer Science (CEDI 2016).
September 2016
The editors
Artículos CEUR Workshop Proceedings Overview of TASS 2016 Miguel Ángel García Cumbreras, Julio Villena Román, Eugenio Martínez Cámara, M. Carlos Díaz Galiano, M. Teresa Martín Valdivia, L. Alfonso Ureña López ...................................................................13 Evaluación de Modelos de Representación del Texto con Vectores de Dimensión Reducida para Análisis de Sentimiento Edgar Casasola Murillo ..............................................................................................................................23 LABDA at the 2016 TASS challenge task: using word embeddings for the sentiment analysis task Antonio Quirós, Isabel Segura-Bedmar, Paloma Martínez.........................................................................29 JACERONG at TASS 2016: An Ensemble Classifier for Sentiment Tweets at Global Level Jhon Adrán Cerón-Guzmán .........................................................................................................................35 Participación de SINAI en TASS 2016 A. Montejo-Ráez, M. C. Díaz-Galiano.........................................................................................................41 ELiRF-UPV en TASS 2016: Análisis de Sentimientos en Twitter Lluís-F. Hurtado, Ferran Pla ......................................................................................................................47 GTI at TASS 2016: Supervised Approach for Aspect Based Sentiment Analysis in Twitter Tamara Álvarez-López, Milagros Fernández-Gavilanes, Silvia García-Méndez, Jonathan JuncalMartínez, Francisco Javier González-Castaño ...........................................................................................53 TASS 2016: Workshop on Sentiment Analysis at SEPLN, septiembre 2016, pág. 13-21
Overview of TASS 2016
Resumen de TASS 2016 Resumen: Este artículo describe la quinta edición del taller de evaluación experimental TASS 2016, enmarcada dentro del Congreso Internacional SEPLN 2016. El principal objetivo de TASS es promover la investigación y el desarrollo de nuevos algoritmos, recursos y técnicas para el análisis de sentimientos en medios sociales (concretamente en Twitter), aplicado al idioma español. Este artículo describe las tareas propuestas en TASS 2016, así como el contenido de los corpus utilizados, los participantes en las distintas tareas, los resultados generales obtenidos y el análisis de estos resultados.
Palabras clave: TASS 2016, análisis de opiniones, medios sociales Abstract: This paper describes TASS 2016, the fifth edition of the Workshop on Sentiment Analysis at SEPLN. The main aim is the promotion of the research and the development of new algorithms, resources and techniques on the field of sentiment analysis in social media (specifically Twitter) focused on the Spanish language. This paper presents the TASS 2016 proposed tasks, the description of the corpora used, the participant groups, the results and analysis of them.
Keywords: TASS 2016, sentiment analysis, social media. 1
TASS is an experimental evaluation workshop, a satellite event of the annual SEPLN Conference, with the aim to promote the research on Sentiment Analysis in social media focused on the Spanish language. The fifth edition will be held on September 13th, 2016 at the University of Salamanca, Spain.
Sentiment Analysis (SA) is traditionally defined as the computational treatment of opinion, sentiment and subjectivity in texts (Pang & Lee, 2008). However, Cambria and Hussain (2012) offer a more updated definition: Computational techniques for the extraction, classification, understanding and evaluation of opinions and comments published on the Internet and other kind of user generated contents. It is a hard task because even humans often disagree on the polarity of a given text. And it is a harder task when the text has only 140 characters (Twitter messages or tweets).
Although SA is not a new task, it is still
challenging, because the state of the art has not
yet resolved some problems related to
multilingualism, domain adaptation, text genre
adaptation and polarity classification at fine
grained level. Polarity classification has usually
been tackled following two main approaches.
The first one applies machine learning
algorithms in order to train a polarity classifier
using a labelled corpus
The aim of TASS is to provide a competitive forum where the newest research works in the field of SA in social media, specifically focused on Spanish tweets, are described and discussed by scientific and business communities.
The rest of the paper is organized as follows.
Section 2 describes the different corpus provided to participants. Section 3 shows the different tasks of TASS 2016. Section 4 describes the participants and the overall results are presented in Section 5. Finally, the last section shows some conclusions and future directions.
2
TASS 2016 experiments are based on two corpora, specifically built for the different editions of the workshop.
The two corpora will be made freely available to the community after the workshop. Please send an email to tass@sngularmeaning.team filling in the TASS Corpus License agreement with your email, affiliation (institution, company or any kind of organization) and a brief description of your research objectives, and you will be given a password to download the files in the password protected area. The only requirement is to include a citation to a relevant paper and/or the TASS website.
2.1
The General Corpus contains over 68.000 tweets, written in Spanish, about 150 wellknown personalities and celebrities of the world of politics, economy, communication, mass media and culture, between November 2011 and March 2012. Although the context of extraction has a Spanish-focused bias, the diverse nationality of the authors, including people from Spain, Mexico, Colombia, Puerto Rico, USA and many other countries, makes the corpus reach a global coverage in the Spanishspeaking world.
Each tweet includes its ID (tweetid), the creation date (date) and the user ID (user). Due to restrictions in the Twitter API Terms of Service (https://dev.twitter.com/terms/apiterms), it is forbidden to redistribute a corpus that includes text contents or information about users. However, it is valid if those fields are removed and instead IDs (including Tweet IDs and user IDs) are provided. The actual message content can be easily obtained by making queries to the Twitter API using the tweetid.
The general corpus has been divided into training set (about 10%) and test set (90%). The training set was released, so the participants could train and validate their models. The test corpus was provided without any tagging and has been used to evaluate the results. Obviously, it was not allowed to use the test data from previous years to train the systems.
Each tweet was tagged with its global polarity (positive, negative or neutral sentiment) or no sentiment at all. A set of 6 labels has been defined: strong positive (P+), positive (P), neutral (NEU), negative (N), strong negative (N+) and one additional no sentiment tag (NONE).
In addition, there is also an indication of the level of agreement or disagreement of the expressed sentiment within the content, with two possible values: AGREEMENT and DISAGREEMENT. This is especially useful to make out whether a neutral sentiment comes from neutral keywords or else the text contains positive and negative sentiments at the same time.
Moreover, the polarity values related to the entities that are mentioned in the text are also included for those cases when applicable. These values are similarly tagged with 6 possible values and include the level of agreement as related to each entity.
This corpus is based on a selection of a set of topics. Thematic areas such as “política” (“politics”), “fútbol” (“soccer”), “literatura” (“literature”) or “entretenimiento” (“entertainment”). Each tweet in the training and test set has been assigned to one or several of these topics (most messages are associated to just one topic, due to the short length of the text).
The annotation has been semi-automatically done: a baseline machine learning model is first run and then all tags are checked by human experts. In the case of the polarity at entity level, due to the high volume of data to check, the human annotation has only been done for the training set.
Table 1 shows a summary of the training and test corpora provided to participants.
Attribute Tweets Tweets (test) Tweets (test) Topics Users Date start (train) Date end (train) Date start (test) Date end (test) Value 68.017 60.798 (89%) 7.219 (11%) 10 154 2011-12-02 2012-04-10 2011-12-02 2012-04-10
Table 1: Corpus statistics
Users were journalists (periodistas), politicians (políticos) or celebrities (famosos). The only language involved was Spanish (es).
The list of topics that have been selected is the following: • Politics (política) • Entertainment (entretenimiento) • Economy (economía) • Music (música) • Soccer (fútbol) • Films (películas) • Technology (tecnología) • Sports (deportes) • Literature (literatura) • Other (otros)
The corpus is encoded in XML. Figure 1 shows the information of two tweets. The first tweet is only annotated with the polarity at tweet level because there is not any entity in the text. However, the second one is annotated with the global polarity of the message and the polarity associated to each of the entities that appear in the text (UPyD and Foro Asturias). STOMPOL (corpus of Spanish Tweets for Opinion Mining at aspect level about POLitics) is a corpus of Spanish tweets prepared for the research on the challenging task of opinion mining at aspect level. The tweets were gathered from 23rd to 24th of April 2015, and are related to one of the following political aspects that appear in political campaigns: • Economics (Economía): taxes, infrastructure, markets, labour policy... • Health System (Sanidad): hospitals, public/private health system, drugs, doctors... • Education (Educación): state school, private school, scholarships... • Political party (Propio_partido): anything good (speeches, electoral programme...) or bad (corruption, criticism) related to the entity • Other aspects (Otros_aspectos): electoral system, environmental policy...
Each aspect is related to one or several entities that correspond to one of the main political parties in Spain, which are: • Partido_Popular (PP) • Partido_Socialista_Obrero_Español (PSOE) • Izquierda_Unida (IU) • Podemos • Ciudadanos (C’s) • Unión_Progreso_y_Democracia (UPyD)
Each tweet in the corpus has been manually annotated by two annotators, and a third one in case of disagreement, with the sentiment polarity at aspect level. Sentiment polarity has been tagged from the point of view of the person who writes the tweet, using 3 levels: P, NEU and N. Again, no difference is made between no sentiment and a neutral sentiment (neither positive nor negative). Each political aspect is linked to its correspondent political party and its polarity. Entity PP PSOE C’s Podemos IU UPyD Total Since the first edition of TASS, a new task and a new corpus have been published. However, one of the aims of TASS is the evaluation of the progress of the research on SA. Thus, the edition of 2016 was focused on the analysis and the comparison of the systems with the submissions of previous editions.
The edition of 2016 was focused on two tasks: polarity classification at tweet level and polarity classification at entity level. The polarity classification task has been proposed with the same corpus since the first edition of TASS, but the polarity classification at aspect level has been proposed with a different corpus each edition. In the edition of 2016 the classification at aspect level uses the STOMPOL corpus, which was published the first time in the edition of 2015.
Participants are expected to submit up to 3 results of different experiments for one or both of these tasks, in the appropriate format described below.
Along with the submission of experiments, participants have been invited to submit a paper to the workshop in order to describe their experiments and discussing the results with the audience in a regular workshop session.
The two proposed tasks are described next. 3.1
This task consists on performing an automatic polarity classification to determine the global polarity of each message in the test set of the General Corpus. The training set of the corpus was provided to the participants with the aim they could train and validate their models with it. There were two different evaluations: one based on 6 different polarity labels (P+, P, NEU,
N, N+, NONE) and another based on just 4 labels (P, N, NEU, NONE).
Participants are expected to submit (up to 3) experiments for the 6-labels evaluation, and they are also allowed to submit (up to 3) specific experiments for the 4-labels scenario.
Results must be submitted in a plain text file with the following format:
tweetid \t polarity where polarity can be: • P+, P, NEU, N, N+ and NONE for the 6-labels case • P, NEU, N and NONE for the 4-labels case.
The same test corpus of previous years was used for the evaluation in order to develop a comparison among the systems. The accuracy is one of the measures used to evaluate the systems, however due to the fact that the training corpus is not totally balanced the systems were also assessed by the macroaveraged precision, macro-averaged recall and macro-averaged F1-measure. 3.2 Task 2: Aspect-based sentiment analysis A corpus with the entities and the aspect identified was provided to the participants, so the goal of the systems is the inference of the polarity at the aspect-level. As in 2015, STOMPOL corpus was the corpus used in this task. STOMPOL was divided in training and test set, the first one for the development and validation of the systems, and the second for evaluation.
Participants are expected to submit up to 3 experiments for each corpus, each in a plain text file with the following format: tweetid \t aspect-entity \t polarity
Allowed polarity values are: P, N and NEU. For the evaluation, a single label combining “aspect-polarity” has been considered. As in the first task, accuracy, macro-averaged precision, macro-averaged recall and macro-averaged F1measure have been calculated for the global result.
4
This year 7 (7 last year) groups submitted their systems The list of active participant groups is shown in Table 3, including the tasks in which they have participated.
Six of the seven participant groups sent a report describing their experiments and results achieved. Papers were reviewed and included in the workshop proceedings. References are listed in Table 4.
Group jacerong ELiRF-UPV LABDA INGEOTEC GASUCR GTI SINAI_w2v Total X 6 2 X X 1
Submitted runs and results for Task 1, evaluation based on 5 polarity levels with the whole General test Corpus are shown in Table 5. Accuracy, macro-averaged precision, macroaveraged recall and macro-averaged F1measure have been used to evaluate each individual label and ranking the systems.
In order to perform a more in-depth evaluation, results are calculated considering the classification only in 3 levels (POS, NEU, NEG) and no sentiment (NONE) merging P and P+ in only one category, as well as N and N+ in another one. The results reached by the submitted systems are shown in Table 6.
Run Id M-F1 ELiRF-UPV_1 0.518 ELiRF-UPV_2 0.496 jacerong_2 jacerong_3 jacerong_1 INGEOTEC LABDA_1 LABDA_2 LABDA_3 GASURC_3 GASURC_1 GASURC_2 ELiRF-UPV_1 0.549 ELiRF-UPV_2 0.548 Run Id jacerong_3 jacerong_2 jacerong_1 INGEOTEC LABDA_3 LABDA_2 LABDA_1 SINAI_w2v_1 SINAI_w2v_3 SINAI_w2v_4 SINAI_w2v_2 GASURC_1 GASURC_2 5.2
Submitted runs and results for Task 2, with the STOMPOL corpus, are shown in Table 7. Accuracy, macro-averaged precision, macroaveraged recall and macro-averaged F1measure have been used to evaluate each individual label and ranking the systems.
Run Id GTI ELiRF-UPV_1 0.526
M-F1
The systems submitted in the edition of 2016 represent the next step of the ones submitted in the previous edition. The systems may be cluster in two groups, those ones that rely on the classification power of the ensemble of several base classifiers, and those systems that change the use traditional Bag-of-Words model for the use of vectors of word embeddings in order to represent the meaning of each word. In the subsequent paragraphs the main features of the systems submitted are going to be depicted.
Hurtado and Pla (2016) describe the participation of the team ELiRF-UPV in the two tasks of TASS 2016. The only difference between the systems submitted for the two tasks is the fact that the one focused on the second task has a module for the identification of the context of each of the entities and aspects annotated on the tweets. The polarity classification system relies on the ensemble of 192 configurations of a SVM classifiers. For the combination of the set of classifiers they evaluate the performance of an approach based on voting and other on stacking.
The system depicted in (Cerón-Guzmán, 2016) is also based on an approach of ensemble classifiers. In this case the base classifiers used a classifier based on logistic regression and they are combined by voting.
Alvarez et al. (2016) exposed the participation of the team GTI on the task 2. The system is similar to the system of the team ELiRF-UPV in the sense that it is composed by two layers: context identification and polarity classification. Regarding the identification of the context, the authors design a heuristic method based on lexical markers. The polarity classification system is a SVM classifier that uses different type of features in order to represent the contexts of the entities and the aspects.
Montejo-Ráez and Díaz-Galiano (2016) introduce a system based on a supervised learning algorithm over vectors resulting from a weighted vector. This vector is computed using a Word2Vec algorithm. This method, which is inspired from neural-network language modelling, was executed with a collection of tweets written in Spanish and the Spanish Wikipedia in order to generate a set of word embeddings for the representation of the words of the General Corpus of TASS as dense vectors. The creation of the collection of tweets written in Spanish followed a distant supervision approach by means the assumption that tweets with happy and sad emoticons express emotions or opinions. Their experiments show massive data from Twitter can lead to a slight improvement in classification accuracy.
The system presented by the team LABDA
Casasola Murillo and Marín Reventós (2016) submitted an unsupervised system based on the system described in Turney (2002), but with a specific adaptation to the classification of tweets written in Spanish. 5.4
In Table 5 and Table 6 are shown the results of each system and they are ranked by the F1score reached, so it is not hard to know what is the best system in the edition of 2016.
On the other hand, how many tweets were rightly classified by the submitted systems? Is there a set of tweets that were not rightly classified by any system? What are the most difficult tweets to classify? These questions are going to be answered in the following paragraphs?
Table 8 shows the rate of tweets that are rightly classified by a number of systems. There are about a 6% of tweets whose polarity is not inferred by any of the submitted systems. In other words, the submitted systems in the edition of 2016 are able to classify about the 94% of the test set. So, what is the main features of that 6% of tweets that any system inferred their polarity? Number of systems 0 1 2 3 4 5 6 7 8 9 10 11 12 13 Sacarle 17 puntos en la final de Copa al Barça CB en el Palau Sant Jordi es una pasada.
Beating Barça by 17 points in the Copa is amazing
Polarity: P+
Figures Figure 3,Figure 4Figure 5 are three examples of tweets that were not rightly classified by any system. The common feature of the three tweets is that they do not have any lexical marker that express emotion or opinion. Moreover, the tweet of the Figure 4 is sarcastic, which means an additional challenging for SA because requires a deep understanding of the language. hahahahahaha “@Absolutexe: ¿Le han cambiado ya el nombre a la Junta de Andalucía por la Banda de Andalucía o aún no?” hahahahahaha “@Absolutexe: Has the Junta de Andalucía renamed Gang of Andalucía or not yet?”
Polarity: N+ Id: 177439342497767424 Rubalcaba pide a Rajoy que presente ya los Presupuestos y dice que no lo hace porque espera a las elecciones andaluzas Rubalcaba requires Rajoy to submit the Budget and says that he didn’t because he is waiting the results of the elections in Andalucia
Polarity: NONE
All the systems submitted are based on linear classifiers that do not take into account the context of each word, which means a big drawback for the understanding the meaning of a span of text.
The tweets of the Figures 3, 4 and 5 show that opinions and emotions are not only expressed by lexical markers, so the future participants should take into account the challenging task of implicit opinion analysis, irony and sarcasm detection. These new problems may be framed on the semantic level of Natural Language Processing and should be tackled by the research community in order to go a step further in the understanding of the subjective information, which is continuously published on the Internet. 6
TASS was the first workshop about SA focused on the processing of texts written in Spanish. In the three first editions of TASS, the research community were mainly formed by Spanish researchers, however since the last edition, the researchers that come from South America is making bigger, so it is an evidence that the research community of Sentiment Analysis in Spanish is not only located in Spain and is formed by the Spanish speaking countries.
Anyway, the developed corpus and gold standards, and the reports from participants will for sure be helpful for knowing the state of the art in SA in Spanish.
The future work will be mainly focused on the definition of a new General Corpus because of the following reasons: 1. The language used on Twitter changes faster than the language used in traditional genres of texts, so the update of the corpus is required in order to cover a real used of the language on Twitter. 2. After several editions of the workshop, we realize that the quality of the annotation is not extremely good, so it is required to define a new corpus with a high quality annotation in order to provide a real gold standard for Spanish SA on Twitter. 3. The research community deeply know the General Corpus of TASS and it wants a new challenge.
A significant amount of new tasks is currently being defined in Natural Language Processing, so some of them, such as stance classification, will be studied to be proposal for the next edition of TASS.
This work has been partially supported by a
grant from the Fondo Europeo of Desarrollo
Regional (FEDER) and REDES project
(TIN2015-65136-C2-1-R) from the Spanish
Government.
OOV word. These candidates are
elements of the union of a dictionary
of Spanish standard word forms and a
gazetteer of proper nouns. The best
normalization candidate for the OOV word
is which best ts a statistical language
model. The language model was
estimated from the Spanish Wikipedia
corpus. Lastly, the selected candidate is
capitalized according to the
capitalization rules of the Spanish language.
Extensive research on lexical normalization
of Spanish tweets can be read in
Negation handling. Inspired by the
approach proposed by Pang et al.
Feature Extraction In this stage, the normalized tweet text is transformed into a feature vector that feeds the machine learning classi er. The features are grouped into basic features and n-gram features. 2.2.1 Basic Features Some of these features are computed before the process of text cleaning and normalization is performed.
The number of words completely in uppercase.
The number of words with more than two consecutive repetitions of a same character.
The number of consecutive repetitions of exclamation marks, question marks, and both punctuation marks (e.g., \!!", \??", \?!") and whether the text ends with an exclamation or question mark.
The number of occurrences of each class of emoticons (i.e., positive and negative) and whether the last token of the tweet is an emoticon.
The number of positive and negative
words, relative to the ElhPolar lexicon
The number of negated contexts.
The number of occurrences of each
Partof-Speech tag.
2.2.2 N-gram Features
The xed-length set of basic features is
always extracted from tweets. However, the
tweet text varies from another in terms of
length, number of tokens, and vocabulary
used. For that reason, a process that
transforms textual data into numerical feature
vectors of xed length is required. This process,
known as vectorization, is performed by
applying the tf-idf weighting scheme
Classi cation
At the last stage, the sentiment analysis
system classi es a given tweet as either P+, P,
NEU, N, N+, or NONE, or assigns
probabilities for each class. After receiving as input
the feature vector, a L2-regularized Logistic
Regression classi er assigns a class label to
the tweet or a probability to be of a certain
class. The classi er was trained on the
training set, using the Scikit-learn
Experiments
1,720 di erent sentiment analysis systems
were trained on the training set via 5-fold
cross validation, in order to nd the best
parameter settings, namely: negation handling,
run-1
run-2
run-3
run-1
run-2
run-3
Class
P
NEU
N
NONE
polarity lexicon, order of word and
character n-grams, and others parameters related
to the vectorization process (e.g.,
lowercasing, frequency thresholds, etc.). The systems
were sorted by their mean cross-validation
score, and thus the top 50 ranked were
ltered to build the ensemble. The training
set is a collection of 7,219 tweets, each of
which is tagged with one of six labels (i.e.,
P+, P, NEU, N, N+, and NONE). Note that
the systems were trained for the six-labels
evaluation, and therefore the P+ and P
labels were merged into P, as well as the N+
and N labels were merged into N, to produce
an output in accordance with the four-labels
evaluation. Further description of the
provided corpus, as well as of the training and
test sets, can be read in
Next, the top 50 systems assigned a class label to each tweet in a collection of 1,000, which was drawn from the untagged test set with a similar class distribution to the training set. In this stage, the objective was to nd the systems with the lowest absolute correlation with each other; therefore, the performance was not evaluated. Then, the less-correlated combinations of 5, 10, and 25 systems, were used to build the ensembles, whose outputs correspond to the submitted experiments. These experiments are described below: run-1: the less-correlated combination of 5 systems, which chooses the class label that represents the majority in the predictions made by the ensemble members. run-2: the less-correlated combination of 10 systems, which chooses the class with the highest unweighted average probability. run-3: the less-correlated combination of 25 systems, which chooses the class with the highest unweighted average probability.
Tables 1 and 2 show the performance evaluation on the test set (i.e., a collection of 60,798 tweets) for six and four labels, respectively. Accuracy has been de ned as the o cial metric for ranking the systems. In summary, the main gain occurs among the \run1" and \run-2" experiments, with an increment of 0.5% in accuracy in the six-labels
Table 3: Discriminative power for each class in the four-labels evaluation evaluation, and of 0.2% in the four-labels evaluation; instead, a negligible gain occurs among the \run-2" and\ run-3" experiments, taking additionally into account the computational cost of running the latter.
As a nal point, Table 3 shows how the overall performance is a ected by the low discriminative power of the ensembles (in this case, the one that correspond to \run-3") for the NEU class. With this in mind, it is proposed as future work to deal with the low representativeness of the NEU class in the training data (i.e., 9.28% of tweets), in order to properly characterize this kind of tweets. 4
Conclusion This paper has described an ensemble-based approach for sentiment analysis of Spanish Twitter data at global level, developed in order to participate in Task 1 proposed by the organization of TASS workshop. Three ensembles were built on the combination of sentiment analysis systems with the lowest absolute correlation with each other. The systems were adapted to the informal genre and the free writing style that characterize Twitter, in order to improve the quality of natural language analysis. In this way, the predicted class label for a particular tweet was based on a majority rule or on the highest average probability. Experimental results showed that the less-correlated combination of 25 systems, which chose the class with the highest unweighted average probability, was the setting that best suited to the task. However, there is a great room for improvement in the learning of a proper characterization of neutral tweets.
Participacion de SINAI en TASS 2016
SINAI participation in TASS 2016 A. Montejo-Raez University of Jaen 23071 Jaen (Spain) amontejo@ujaen.es
M.C. D az-Galiano
University of Jaen 23071 Jaen (Spain) mcdiaz@ujaen.es Resumen: Este art culo describe el sistema de clasi cacion de la polaridad utilizado por el equipo SINAI en la tarea 1 del taller TASS 2016. Como en participaciones anteriores, nuestro sistema se basa en un metodo supervisado con SVM a partir de vectores de palabras. Dichos vectores se calculan utilizando la tecnicas de deeplearning Word2Vec, usando modelos generados a partir de una coleccion de tweets expresamente generada para esta tarea y el volcado de la Wikipedia en espan~ol. Nuestros experimentos muestran que el uso de colecciones de datos masivos de Twitter pueden ayudar a mejorar sensiblemente el rendimiento del clasi cador.
Palabras clave: Analisis de sentimientos, clasi cacion de la polaridad,
deeplearning, Word2Vec
Abstract: This paper introduces the polarity classi cation system used by the
SINAI team for the task 1 at the TASS 2016 workshop. Our approach is based on a
supervised learning algorithm over vectors resulting from a weighted vector. This
vector is computed using a deep-learning algorithm called Word2Vec. The algorithm
is applied so as to generate a word vector from a deep neural net trained over a
speci c tweets collection and the Spanish Wikipedia. Our experiments show massive
data from Twitter can lead to a slight improvement in classi caciones accuracy.
Keywords: Sentiment analysis, polarity classi cation, deep learning, Word2Vec,
Doc2Vec
1
En este trabajo describimos las
aportaciones realizadas para participar en la
tarea 1 del taller TASS (Sentiment
Analysis at global level), en su edicion de 2016
Este estudio esta parcialmente nanciado por el proyecto TIN2015-65136-C2-1-R otorgado por el Ministerio de Econom a y Competitividad del Gobierno de Espan~a. de dichos vectores para obtener una unica representacion vectorial. Nuestros resultados demuestran que el rendimiento del sistema de clasi cacion puede verse sensiblemente mejorado gracias a la introduccion de estos datos en la generacion del modelo de palabras, no as en el entrenamiento del clasi cador de polaridad nal.
La tarea del TASS en 2016 denominada Sentiment Analysis at global level consiste en el desarrollo y evaluacion de sistemas que determinan la polaridad global de cada tweet del corpus general. Los sistemas presentados deben predecir la polaridad de cada tweet utilizando 6 o 4 etiquetas de clase (granularidad na y gruesa respectivamente).
El resto del art culo esta organizado de la siguiente forma. El apartado 2 describe el estado del arte de los sistemas de clasi cacion de polaridad en espan~ol. A continuacion, se describe la coleccion de tweets con emoticonos utilizada para entrenar el clasi cador. En el apartado 4 se describe el sistema desarrollado y en el apartado 5 los experimentos realizados, los resultados obtenidos y el analisis de los mismos. Finalmente, en el ultimo apartado exponemos las conclusiones y el trabajo futuro. 2
Clasi cacion de la polaridad en
espan~ol
La mayor parte de los sistemas de clasi
cacion de polaridad estan centrados en textos
en ingles, y para textos en espan~ol el sistema
mas completo, en cuanto a tecnicas lingu
sticas aplicadas, posiblemente sea The Spanish
SO Calculator
Los algoritmos de aprendizaje profundo
(deep-learning en ingles) estan dando buenos
resultados en tareas donde el estado del
arte parec a haberse estancado
Estas tecnicas tambien son de aplicacion en
el procesamiento del lenguaje natural
(Collobert y Weston, 2008), e incluso ya existen
sistemas orientados al analisis de sentimientos,
como el de Socher et al.
Los algoritmos de aprendizaje automatico no son nuevos, pero s estan resurgiendo gracias a una mejora de las tecnicas y la disposicion de grandes volumenes de datos necesarios para su entrenamiento efectivo.
En la edicion de TASS en 2012 el equipo
que obtuvo mejores resultados
En la edicion de TASS en 2013 el mejor equipo (Fernandez et al., 2013) tuvo todos sus experimentos en el top 10 de los resultados, y la combinacion de ellos alcanzo la primera posicion. Presentaron un sistema con dos variantes: una version modi cada del algoritmo de ranking (RA-SR) utilizando bigramas, y una nueva propuesta basada en skipgrams. Con estas dos variantes crearon lexicones sobre sentimientos, y los utilizaron junto con aprendizaje automatico (SVM) para detectar la polaridad de los tweets.
En 2014 el equipo con mejores resultados
en TASS se denominaba ELiRF-UPV
El equipo ELiRF-UPV
Coleccion de tweets con emoticonos Los algoritmos de deep-learning necesitan grandes volumenes de datos para su entrenamiento. Por ese motivo se ha creado una coleccion de tweets espec ca para la deteccion de polaridad. Para crear dicha coleccion se han recuperado tweets con las siguientes caracter sticas:
Que contengan emoticonos que expresen la polaridad del tweet. En este caso se han utilizado los siguientes emoticonos:
Positivos: :) :-) :D :-D
Negativos: :( :-( Que los tweets no contengan URLs, para evitar tweets cuyo contenido principal se encuentra en el enlace.
Que no sean retweets, para reducir el numero de tweets repetidos.
La captura de dichos tweets se realizo durante 22 d as, del 18/07/2016 hasta el 9/08/2016, recuperando unos 100.000 tweets diarios aproximadamente. Tal y como se ve en la Figura 1 la recuperacion fue muy homogenea y se obtuvieron mas de 2.000.000 de tweets. Figura 1: Numero de tweets recuperados cada 12 horas
Posteriormente, se realizo un ltrado de dichos tweets eliminando aquellos que contubieran menos de 5 palabras, teniendo en cuenta que consideramos palabra todo termino que solo contenga letras (sin numeros, ni caracteres especiales).
Al nal quedaron 1.777.279 clasi cados segun el emoticono que contienen de la siguiente manera:
Positivos: 869.339 tweets
Negativos: 907.940 tweets
Por ultimo, se realiza la siguiente limpieza de tweets:
Convertir el texto a minusculas.
Sustituir letras acentuadas por sus versiones sin acentuar.
Quitar las palabras vac as de contenido (stopwords).
Normalizar las palabras para que no contengan letras repetidas, sustituyendo las repeticiones de letras contiguas para dejar solo 3 repeticiones. 4
Descripcion del sistema
Word2Vec1 es una implementacion de la
arquitectura de representacion de las palabras
mediante vectores en el espacio continuo,
basada en bolsas de palabras o n-gramas
concebida por Tomas Mikolov et al.
Tal y como se ha indicado, para obtener
los vectores Word2Vec representativos para
cada palabra tenemos que generar un modelo
a partir de un volumen de texto grande. Para
ello hemos utilizado los parametros que
mejores resultados obtuvieron en nuestra
participacion del 2014
Como puede verse en la Figura 2, nuestro sistema realiza la clasi cacion de los tweets utilizando dos fases de aprendizaje, una en la que entrenamos el modelo Word2Vec haciendo uso de un volcado de la enciclopedia on-line Wikipedia, en su version en espan~ol, como hemos indicado anteriormente. De esta forma representamos cada tweet con el vector resultado de calcular la media de los vectores Word2Vec de cada palabra en el tweet y su desviacion t pica (por lo que cada vector de palabras por modelo es de 600 dimensiones).
Se lleva a cabo una simple normalizacion
previa sobre el tweet, eliminando repeticion de
letras y poniendo todo a minusculas. La
segunda fase de entrenamiento utiliza el
algoritmo SVM y se entrena con la coleccion de
tweets con emoticonos explicada en el
apartado 3. La implementacion de SVM utilizada es
la basada en kernel lineal con entrenamiento
SGD (Stochastic Gradient Descent)
proporcionada por la biblioteca Sci-kit Learn3
Esta solucion es la utilizada en las dos variantes de la tarea 1 del TASS con prediccion 2http://dumps.wikimedia.org/eswiki 3http://scikit-learn.org/ de 4 clases: la que utiliza el corpus de tweets completo (full test corpus) y el que utiliza el corpus balanceado (1k test corpus).
Figura 2: Flujo de datos del sistema completo 5
Resultados obtenidos Hemos experimentado con el efecto que tienen en el rendimiento del sistema el uso de una coleccion de datos generada a partir de la captura de tweets y que han sido etiquetados segun los emoticonos que contienen en la forma comentada anteriormente. La coleccion de mas de 1,7 millones de tweets ha sido utilizada al completo para generar un modelo de vectores de palabras, cuya combinacion con el de Wikipedia se ha analizado. Tambien hemos comprobado como el uso de dicha coleccion de tweets afecta cuando se usa para el entrenamiento del modelo de clasi cacion de la polaridad. Para ello se han seleccionado 500,000 tweets aleatoriamente de esta coleccion, con sus correspondientes etiquetas P (positivo) o N (negativo) y se han combiando con la colecciond de entrenamiento de TASS.
Los resultados segun las medidaas de Accuracy y Macro F1 obtenidas se muestran en la tabla 1. La primera columna nos indica a partir de cuales datos se han generado los modelos de vectores de palabras, bien solo con Wikipedia (W) o como combinacion de esta con los tweets del corpus construido (W+T). La segunda columna indica como se ha entrenado el clasi cador de polaridad a partir de los textos etiquetados vectorizados con los modelos generados en el paso previo, bien solo usando los datos de entrenamiento proporcionados por la organizacion (TASS) o incorporando los etiquetados a partir de emoticonos (TASS+T).
Como podemos observar, el uso de una coleccion de tweets para ampliar la capacidad de representar un modelo basado en vectores de palabras mejora sensiblemente al ge
Tabla 1: Resultados obtenidos sobre el conjunto full w2v SVM Accuracy W TASS 61,31 % W+T TASS 62,39 % W TASS+T 49,28 % W+T TASS+T 53,72 %
Macro-F1 48,55 % 50,44 % 40,20 % 44,10 % nerado solamente con Wikipedia, pasando de 61,31 % de ajuste a un 62,39 %. En cambio, utilizar los tweets capturados para la fase de entrenamiento supervisado no lleva sino a una ca da del rendimiento del sistema.
Esto nos lleva a plantearnos la pregunta de que ocurrir a si utilizaramos solo los tweets recopilados para generar un modelo de vectores de palabras. Los resultados que se obtienen son un 59,05 % de ajuste y un 44,43 % de F1. No cabe duda de que conviene explorar el uso de modelos de generacion de caracter sticas a partir de vectores de palabras.
Estos resultados mejoran nuestros datos del an~o pasado, en los que obtuvimos un ajuste del 61,19 % combinando vectores de palabras (Word2Vec) y vectores de documentos (Doc2Vec). 6
Conclusiones y trabajo futuro A partir de los resultados obtenidos, encontramos que resulta interesante la incorporacion de texto no formal (tweets) para la generacion de los modelos de palabras, lo cual tiene su sentido en una tarea de clasi cacion que, precisamente, trabaja sobre textos no formales que tienen la misma red social como fuente. En cambio, el considerar que los emoticonos en un tweet pueden ayudar a un clasi cador como SVM a mejorar en la determinacion de la polaridad ha resultado una hipotesis fallida. Esto puede entenderse echando un vistazo a algunos de los tweets capturados por el sistema, donde se evidencia la di cultad, incluso para una persona, de poner en contexto el sentido del tweet y su consideracion como positivo o negativo si no disponemos de un emoticono asociado.
Como trabajo futuro nos proponemos disen~ar una red neuronal profunda mas elaborada, pero que parta tambien de textos de entrenamiento tanto formales como no formales, si bien teniendo en cuanta informacion lingu stica mas avanzada como la sintactica, en lugar de trabajar con simples bolsas de palabras. Tambien queremos explorar el uso de redes de este tipo en el proceso de clas cacion en s , y no solo en la generacion de caracter sticas. Una posibilidad es utilizar una red de tipo DBN (Deep Belief Network) (Hinton y Salakhutdinov, 2006) en la que se an~ade una ultima fase donde se realiza el etiquetado de los ejemplos.
Bibliograf a Bengio, Yoshua. 2009. Learning deep architectures for ai. Foundations and trends in
Machine Learning, 2(1):1{127.
Brooke, Julian, Milan To loski, y Maite Taboada. 2009. Cross-linguistic sentiment analysis: From english to spanish. En Galia Angelova Kalina Bontcheva Ruslan Mitkov Nicolas Nicolov, y Nikolai Nikolov, editores, RANLP, paginas 50{54. RANLP 2009 Organising Committee / ACL.
Collobert, Ronan y Jason Weston. 2008.
A uni ed architecture for natural language processing: Deep neural networks with multitask learning. En Proceedings of the 25th International Conference on Machine Learning, ICML '08, paginas 160{167,
New York, NY, USA. ACM.
D az-Galiano, M.C. y A. Montejo-Raez.
2015. Participacion de SINAI DW2Vec en TASS 2015. En In Proc. of TASS 2015: Workshop on Sentiment Analysis at
SEPLN. CEUR-WS.org, volumen 1397.
Fernandez, Javi, Yoan Gutierrez, Jose M.
Gomez, Patricio Mart nez-Barco, Andres Montoyo, y Rafael Mun~oz. 2013. Sentiment analysis of spanish tweets using a ranking algorithm and skipgrams. En In Proc. of the TASS workshop at SEPLN 2013.
Garc a-Cumbreras, Miguel Angel, Julio
Villena-Roman, Eugenio Mart nezCamara, Manuel Carlos D az-Galiano, Ma. Teresa Mart n-Valdivia, y L. Alfonso Uren~a-Lopez. 2016. Overview of tass 2016. En Proceedings of TASS 2016: Workshop on Sentiment Analysis at SEPLN co-located with the 32nd SEPLN Conference (SEPLN 2016), Salamanca,
Spain, September.
Hinton, Geo rey E y Ruslan R Salakhutdinov. 2006. Reducing the dimensionality of data with neural networks. Science, 313(5786):504{507. ELiRF-UPV en TASS 2016: Analisis de Sentimientos en Twitter ELiRF-UPV at TASS 2016: Sentiment Analysis in Twitter
Llu s-F. Hurtado y Ferran Pla Universitat Politecnica de Valencia
Cam de Vera s/n
46022 Valencia flhurtado, fplag@dsic.upv.es Resumen: En este trabajo se describe la participacion del equipo del grupo de investigacion ELiRF de la Universitat Politecnica de Valencia en el Taller TASS2016. Este taller es un evento enmarcado dentro de la XXXII edicion del Congreso Anual de la Sociedad Espan~ola para el Procesamiento del Lenguaje Natural. Este trabajo presenta las aproximaciones utilizadas para las dos tareas planteadas en el taller, los resultados obtenidos y una discusion de los mismos. Nuestra participacion se ha centrado principalmente en explorar diferentes aproximaciones para combinar un conjunto de sistemas con lo que se ha obtenido los mejores resultados en ambas tareas.
Palabras clave: Twitter, Analisis de Sentimientos.
Abstract: This paper describes the participation of the ELiRF research group of the Universitat Politecnica de Valencia at TASS2016 Workshop. This workshop is a satellite event of the XXXII edition of the Annual Conference of the Spanish Society for Natural Language Processing. This work describes the approaches used for the two tasks of the workshop, the results obtained and a discussion of these results. Our participation has focused primarily on exploring di erent approaches for combining a set of systems. Using these approaches we have achieved the best results in both tasks.
Keywords: Twitter, Sentiment Analysis.
Introduccion
El Taller de Analisis de Sentimientos (TASS) en sus cinco ediciones ha venido planteando tareas relacionadas con el analisis de sentimientos en Twitter. El objetivo principal es el de comparar y evaluar diferentes aproximaciones a estas tareas. Ademas, desarrolla recursos de libre acceso, basicamente, corpora anotados con polaridad, tematica, tendencia pol tica, aspectos, que son de gran utilidad para la comparacion de diferentes aproximaciones a las tareas propuestas.
En esta quinta edicion del TASS se
proponen dos tareas de ediciones anteriores
El presente art culo resume la participacion del equipo ELiRF-UPV de la Universitat Politecnica de Valencia en todas las tareas planteadas en este taller. Primero se describen las aproximaciones y recursos utilizados en cada tarea. A continuacion se presenta la evaluacion experimental realizada y los resultados obtenidos. Finalmente se muestran las conclusiones y posibles trabajos futuros.
Descripcion de los sistemas
Los sistemas presentados en el TASS 2016
se basan en el sistema desarrollado en la
edicion anterior del TASS 2015
Esta consiste basicamente en la adaptacion para el castellano del tokenizador de tweets Tweetmotif (Connor, Krieger, y Ahn, 2010).
Tambien se ha usado Freeling
Todas las tareas se han abordado como
un problema de clasi cacion. Se han
utilizado Maquinas de Soporte Vectorial (SVM) por
su capacidad para manejar con exito
grandes cantidades de caracter sticas. En concreto
usamos dos librer as (LibSVM2 y LibLinear3)
que han demostrado ser e cientes
implementaciones de SVM que igualan el estado del
arte. El software esta desarrollado en Python
y para acceder a las librer as de SVM se ha
utilizado el toolkit scikit-learn4.
En este trabajo se ha explotado la
tecnica de combinacion de diferentes con
guraciones de clasi cadores para aprovechar su
complementariedad. Se ha utilizado la tecnica de
votacion simple utilizada en trabajos
anteriores
Cada tweet se ha representado como un vector que contiene los coe cientes tf-idf de las caracter sticas consideradas. En toda la experimentacion realizada, las caracter sticas y los parametros de los clasi cadores se han elegido mediante una validacion cruzada de 10 iteraciones (10-fold cross-validation) sobre el conjunto de entrenamiento.
1http://nlp.lsi.upc.edu/freeling/ 2http://www.csie.ntu.edu.tw/~cjlin/libsvm/ 3http://www.csie.ntu.edu.tw/~cjlin/liblinear/ 4http://scikit-learn.org/stable/
Tarea 1: Analisis de sentimientos en tweets
Esta tarea consiste en determinar la polaridad de los tweets y la organizacion ha de nido dos subtareas. La primera distingue seis etiquetas de polaridad: N y N+ que expresan polaridad negativa con diferente intensidad, P y P+ para la polaridad positiva con diferente intensidad, NEU para la polaridad neutra y NONE para expresar ausencia de polaridad. La segunda solo distinguen 4 etiquetas de polaridad: N, P, NEU y NONE.
El corpus proporcionado por la organizacion del TASS consta de un conjunto de entrenamiento, compuesto por 7219 tweets etiquetados con la polaridad usando seis etiquetas, y un conjunto de test, de 60798 tweets, al cual se le debe asignar la polaridad. La distribucion de tweets segun su polaridad en el conjunto de entrenamiento se muestra en la Tabla 1.
Polaridad N N+ NEU NONE P P+ TOTAL # tweets 1335 847 670 1483 1232 1652 7219 Tabla 1: Distribucion de tweets en el conjunto de entrenamiento segun su polaridad.
A partir de la tokenizacion propuesta se
realizo un proceso de validacion cruzada
(10fold cross validation) para determinar el
mejor conjunto de caracter sticas y los
parametros del modelo. Como caracter sticas se
probaron diferentes taman~os de n-gramas de
palabras y de lemas. Tambien se exploro la
combinacion de los modelos mediante diferentes
tecnicas de votacion para aprovechar su
complementariedad y mejorar las prestaciones
nales. Algunas de estas tecnicas
proporcionaron mejoras signi cativas sobre el mismo
conjunto de datos, como se muestra en
Se han considerado dos alternativas para abordar la tarea: run1 La primera alternativa combina mediante un sistema de votacion ponderada la salida de 192 clasi cadores basados en el uso de SVM. La diferencia entre los clasi cadores radica en el preprocesado y la tokenizacion utilizada, las caracter sticas seleccionadas y los valores de los parametros del propio modelo SVM.
En concreto se realizaron todas las combinaciones posibles entre 8 tokenizaciones (lemas o palabras, detectar NE o no, detectar menciones a usuarios y hashtags, ...); 4 conjuntos distinto de caracter sticas (palabras o bigramas con y sin diccionarios de polaridad) y 6 valores distintos del parametro c del modelo SVM con kernel lineal.
La clase asignada a cada tweet t viene determinada por la siguiente formula.
c^ = argmax(Nt(c) P (c)) c2C Donde C es el conjunto de todas las clases, Nt(c) es el numero de clasi cadores que asignan la clase c al tweet t, y P (c) es la probabilidad a priori de la clase c calculada utilizando el corpus de entrenamiento. run2 La segunda alternativa explora la combinacion de modelos mediante el aprendizaje de un metaclasi cador. Utilizando las salidas de los mismos 192 clasi cadores que en el run anterior, se ha aprendido un segundo modelo SVM que sirve para proporcionar la nueva salida combinada. Se ha destinado una parte del corpus de entrenamiento para ajustar los parametros del metamodelo. Esta aproximacion es la misma que la utilizada en la edicion del TASS 2015.
Para la subtarea de 4 etiquetas el run1 se ha aprendido utilizando el corpus de aprendizaje con 4 etiquetas mientras que el run2, dada la complejidad del ajuste de parametros del metamodelo se ha optado por adaptar el resultado de la subtarea de 6 etiquetas uniendo P y P+ como P y N y N+ como N.
En la Tabla 2 se muestran los valores de Accuracy obtenidos para las dos subtareas.
Los sistemas presentados han obtenido las dos primeras posiciones en las dos subtareas consideradas.
6-ETIQUETAS 4-ETIQUETAS
Run run1 run2 run1 run2 Tabla 2: Resultados o ciales del equipo ELiRF-UPV en la Tarea 1 de la competicion TASS-2016 sobre el conjunto de test para 6 y 4 etiquetas.
Tarea 2: Analisis de Polaridad de Aspectos en Twitter
Esta tarea consiste en asignar la
polaridad a los aspectos que aparecen marcados en
el corpus. Una de las di cultades de la tarea
consiste en de nir que contexto se le asigna a
cada aspecto para poder establecer su
polaridad. Para un problema similar, deteccion de
la polaridad a nivel de entidad, en la edicion
del TASS 2013, propusimos una
segmentacion de los tweets basada en un conjunto de
heur sticas
En este trabajo se propone una aproximacion mas simple que consiste en determinar el contexto de cada aspecto a traves de una ventana ja de nida a la izquierda y derecha de la instancia del aspecto. Esta aproximacion es la que se utilizo en nuestro sistema del TASS 2015 la cual utiliza ventanas de diferente longitud. La longitud de la ventana optima se ha determinado experimentalmente sobre el conjunto de entrenamiento mediante una validacion cruzada. Para entrenar nuestro sistema, se ha considerado el conjunto de entrenamiento unicamente, se han determinado los segmentos para cada aspecto y se ha seguido una aproximacion similar a la Tarea 1.
El corpus de la tarea, corpus STOMPOL, se compone de un conjunto de tweets relacionados con una serie de aspectos pol ticos (como econom a, sanidad, etc.) enmarcados en la campan~a pol tica de las elecciones andaluzas de 2015. Cada aspecto se relaciona con una o varias entidades que se corresponden con uno de los principales partidos pol ticos en Espan~a (PP, PSOE, IU, UPyD, Cs y Podemos). El corpus consta de 1.284 tweets, y ha sido dividido en un conjunto de entrenamiento (784 tweets) y un conjunto de evaluacion (500 tweets). 4.1.
Aproximacion y resultados
A continuacion presentamos una pequen~a descripcion de las caracter sticas de nuestro sistema as como el proceso seguido en la fase de entrenamiento. El sistema utiliza un clasi cador basado en SVM. Para aprender los modelos solo se utiliza el conjunto de entrenamiento proporcionado para la tarea y los diccionarios de polaridad previamente descritos. Antes de abordar el entrenamiento se determinan los segmentos de tweet que constituyen el contexto de cada una de los aspectos presentes. Se ha tenido en cuenta tres taman~os de ventana de longitudes 5, 7 y 10 palabras a la izquierda y derecha del aspecto. Cada uno de los segmentos se tokeniza y se utiliza Freeling para determinar sus lemas y ciertas entidades. A continuacion se aprenden diferentes modelos combinando taman~os de ventana, parametros del modelo y diferentes caracter sticas (palabras, lemas, NE, etc).
Mediante validacion cruzada se elige el mejor modelo. Para esta tarea solo hemos presentado un modelo.
STOMPOL
Run run1
Accuracy
0.633 Tabla 3: Resultados o ciales del equipo ELiRF-UPV en la Tarea 2 de la competicion TASS-2016 para el corpus STOMPOL.
En la Tabla 3 se presentan los resultados obtenidos para la Tarea 2 con lo que nuestra aproximacion ha obtenido la primera posicion en dicha tarea.
Conclusiones y trabajos futuros
En este trabajo se ha presentado la participacion del grupo ELiRF-UPV en las 2 tareas planteadas en TASS 2016. Nuestro equipo ha utilizado aproximaciones basadas en maquinas de soporte vectorial y se ha centrado principalmente en combinar diferentes sistemas.
Haciendo un analisis del numero de participantes y de los resultados obtenidos en las dos ultimas ediciones del TASS, creemos que se esta cerca de alcanzar los mejores resultados posibles en la tarea de Analisis de sentimientos tal y como se ha venido planteando hasta el momento.
A la vista de los buenos resultados que se han obtenido mediante la combinacion de sistemas, como trabajo futuro nos planteamos desarrollar nuevos metodos de combinacion de sistemas mas so sticados as como la inclusion de otros paradigmas de clasi cacion mas hetereogeneos (distintos de los SVM) para aumentar la complementariedad de los sistemas combinados.
Ademas, se pretende extender el sistema para otros idiomas. El sistema descrito ya ha sido utilizado, con ligeras modi caciones, en tareas de analisis de sentimientos para el Ingles en la competicion Semeval (Mart nez, Pla, y Hurtado, 2016) aunque con resultados no tan satisfactorios como en las tareas del TASS.
Agradecimientos
Este trabajo ha sido parcialmente subvencionado por el MINECO mediante el proyecto ASLP-MULAN: Audio, Speech and Language Processing for Multimedia Analytics (TIN2014-54288-C4-3-R).
Bibliograf a Connor, Brendan O, Michel Krieger, y David Ahn. 2010. Tweetmotif: Exploratory search and topic summarization for twitter. En William W. Cohen y Samuel Gosling, editores, Proceedings of the Fourth International Conference on Weblogs and Social Media, ICWSM 2010, Washington, DC, USA, May 23-26, 2010. The AAAI
Press.
Garc a-Cumbreras, Miguel Angel, Julio
Villena-Roman, Eugenio Mart nezCamara, Manuel Carlos D az-Galiano, Ma. Teresa Mart n-Valdivia, y L. Alfonso Uren~a-Lopez. 2016. Overview of tass 2016. En Proceedings of TASS 2016: Workshop on Sentiment Analysis at SEPLN co-located with the 32nd SEPLN Conference (SEPLN 2016), Salamanca,
Spain, September.
Hansen, Lars Kai, Adam Arvidsson,
Finn Arup Nielsen, Elanor Colleoni, y Michael Etter. 2011. Good friends, bad news-a ect and virality in twitter. En Future information technology. Springer, paginas 34{43.
Hurtado, Llu s F., Ferran Pla, y Davide Buscaldi. 2015. Elirf-upv en tass 2015: Analisis de sentimientos en twitter. En SEPLN.
Hurtado, LLu s F y Ferran Pla. 2014. Elirfupv en tass 2014: Analisis de sentimientos, deteccion de topicos y analisis de sentimientos de aspectos en twitter. En
TASS2014.
Mart nez, V ctor, Ferran Pla, y Llu s-F Hurtado. 2016. Dsic-elirf at semeval-2016 task 4: Message polarity classi cation in twitter using a support vector machine approach.
Mart nez-Camara, E., M. T. Mart n
Valdivia, M. D. Molina-gonzalez, y L. A. Uren~a-lopez. 2013. Bilingual Experiments on an Opinion Comparable Corpus. En Proceedings of the 4th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media
Analysis, pagina 87{93.
Padro, Llu s y Evgeny Stanilovsky. 2012.
Freeling 3.0: Towards wider multilinguality. En Proceedings of the Language Resources and Evaluation Conference (LREC 2012), Istanbul, Turkey, May. EL
RA.
Pedregosa, F., G. Varoquaux, A. Gramfort,
V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, y E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825{2830.
Pla, Ferran y Llu s-F Hurtado. 2013. Tass2013: Analisis de sentimientos en twitter.
En Proceedings of the TASS workshop at SEPLN 2013. IV Congreso Espan~ol de Informatica.
Pla, Ferran y Llu s-F. Hurtado. 2014a. Political tendency identi cation in twitter using sentiment analysis techniques. En Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, paginas 183{192, Dublin, Ireland, August. Dublin City University and Association for Computational Linguistics. GTI at TASS 2016: Supervised Approach for Aspect Based
Sentiment Analysis in Twitter GTI en TASS 2016: Una aproximacion supervisada para el analisis de sentimiento basado en aspectos en Twitter Tamara Alvarez-Lopez, Milagros Fernandez-Gavilanes, Silvia Garc a-Mendez, Jonathan Juncal-Mart nez, Francisco Javier Gonzalez-Castan~o
GTI Research Group, AtlantTIC
University of Vigo, 36310 Vigo, Spain ftalvarez,mfgavilanes,sgarcia,jonijmg@gti.uvigo.es, javier@det.uvigo.es Resumen: Este art culo describe la participacion del grupo de investigacion GTI, del centro AtlantTIC, perteneciente a la Universidad de Vigo, en el tass 2016. Este taller es un evento enmarcado dentro de la XXXII edicion del Congreso Anual de la Sociedad Espan~ola para el Procesamiento del Lenguaje Natural. En este trabajo se propone una aproximacion supervisada, basada en clasi cadores, para la tarea de analisis de sentimiento basado en aspectos. Mediante esta tecnica hemos conseguido mejorar las prestaciones de ediciones anteriores, obteniendo una solucion acorde con el estado del arte actual.
Palabras clave: Analisis de sentimiento, aspectos, SVM, aprendizaje automatico, Twitter Abstract: This paper describes the participation of the GTI research group of AtlantTIC, University of Vigo, in tass 2016. This workshop is framed within the XXXII edition of the Annual Congress of the Spanish Society for Natural Language Processing event. In this work we propose a supervised approach based on classi ers, for the aspect based sentiment analysis task. Using this technique we managed to improve the performance of previous years, obtaining a solution re ecting the actual state-of-the-art.
Keywords: Sentiment analysis, aspects, SVM, machine learning, Twitter 1 The social media activity is being profused in the recent years, users post opinions and comments in Twitter and in other social platforms. Due to this, there is a huge amount of information available that could be useful for business, in order to design marketing campaigns or to apply any kind of business analysis.
As a consequence, the research on text mining and also on the eld of Sentiment Analysis (sa) has grown considerably these days. sa is the part of Natural Language Processing (nlp) responsible for determining the polarity of a text or a whole sentence. The sa applied to Twitter has to be conducted in a restricted scenario due to the maxi
This work was partially supported by the Ministerio de Econom a y Competitividad under project COINS (TEC2013-47016-C2-1-R) and by Xunta de Galicia (GRC2014/046). mum length of the post. However, tweets have other elements we have to consider, like hashtags, mentions and retweets. More concretely, aspect-based sentiment analysis (absa) consists of extracting opinions, i.e. determining the sentiment polarity, from speci c entities in the text (Liu, 2012). Therefore, this task becomes a challenge on the eld of nlp.
The tass Workshop
Many approaches applied to sa can be
found in the literature, where it is
possible to distinguish between knowledge based
approaches (Brooke, To loski, and Taboada,
2009; Fernandez-Gavilanes et al., 2016),
using grammars and thesaurus and others
based on machine learning approaches
We present our supervised machine learning (ml) system which consists of a Support Vector Machine (svm) classi er. Our objective is to conduct the sa process at an aspect level, task 2, determining the polarity of a speci c given part of a sentence.
The article is structured as follows. Section 2 is a review of the research involving sa in the Twitter domain. Then, the Section 3 describes the applied approach and the implemented system. In Section 4, we show the experimental results of our system. Finally, in Section 5 we present the conclusions and future works.
Related work A large amount of literature related to Opinion Mining (om) and sa can be found (Pang and Lee, 2008; Mart nez-Camara et al., 2016). Most of the systems are applied to Twitter. However others are applied to social media platforms within the micro-blog context. Due to this, the approaches are varied technically and in connection with the purpose.
Two main approaches exist in sa: supervised and unsupervised learning ones. Supervised systems implement classi cation methods like svm, Logistic Regression (lr), Conditional Random Fields (crf), K-Nearest Neighbors (knn), etc. Cui, Mittal, and Datar (2006) a rmed that svm are more appropriate for sentiment classi cation than generative models, due to their capability for working with ambiguity, that is, dealing with mixed feelings. Supervised algorithms are used when the number of classes, as well as the representative members of each class, are known.
Unsupervised systems are based on
linguistic knowledge like lexicons, and syntactic
features in order to infer the polarity
As noted earlier, the special case of
applying sa to Twitter has been fully
addressed
According to the purpose of the developed
systems, it is possible to nd applications
like classi cation of product reviews and
political sentiment and election results
prediction
Preprocessing Before applying any supervised approach to our corpus, some preprocessing is needed.
First of all, we have to normalize the text, since in Twitter language we can nd abbreviations, mentions, hashtags, URLs or misspellings. In order to do that, we replace the URLs with the \URL" tag and we replace the abbreviations or misspellings with the correct entire word. For mentions and hashtags, we keep them unchanged but deleting the \@" or \#" symbols. Moreover, when a hashtag is composed of several words, we split and treat them as di erent tokens.
After this, a lexical analysis is carried out.
It consists of lemmatization and POS tagging, which are performed by means of Freeling tool (Atserias et al., 2006).
Once we have analysed lexically the texts, we decided to separate the sentences by the di erent aspects. For doing that, the scope of each aspect is determined, applying the following rules, which are adapted from our English aspect based sentiment anaylisis system (Alvarez-Lopez et al., 2016)
If there is only one aspect in the sentence, we keep the sentence unchanged, and introduce it entirely as input for the next step. If there are multiple aspects, we separate the sentences by punctuation marks, conjunctions or other aspects found.
If there are several aspects with no words between them, we consider that they belong to the same context, and assign the same polarity to all of them.
SVM classi er In this section we describe the strategy followed to determine the sentiment (positive, negative or neutral) for each aspect predened in corpus.
We develop a svm classi er, using the
libsvm library
Word tokens of nouns, adjectives and verbs in the sentence.
Lemmas of verbs, nouns and adjectives that appear in each sentence.
POS tags of nouns, adjectives and verbs.
N-grams of di erent length, grouping the words in each sentence.
Aspects appearing in the sentence. We join \aspect"-\entity", de ned in each target as a feature.
Negations. We create a negation dictionary, which contains several particles indicating negation, such as \no", \nunca", etc.
The previous features are all binary ones, assigning the value 1 if the current feature is present in the tweet and the value 0, if not. 4 The Task 2: Sentiment Analysis at the aspect level consists of assigning a polarity label to each aspect, which were initially marked in the stompol corpus (Mart nez-Camara et al., 2016) raised by the tass organization. In this way, this corpus provides both polarity labels and the identi cation of the aspects that appear in each tweet. The aim is to be able to correctly assign to each aspect a positive, negative or neutral polarity.
In this regard, the stompol corpus consists of a set of Spanish tweets related to a number of political issues, such as health or economy, among others. These issues are framed in the political campaign of Andalusian elections in 2015, where each aspect relates to one or several entities that correspond to one of the main political parties in Spain (PP, PSOE, IU, UPyD, Cs and Podemos). The corpus is composed by 1,284 tweets, and has been divided into a training set (784 tweets) and a set of evaluation (500 tweets).
In order to evaluate the performance of the various features for polarity classi cation at an aspect-based level, we perform a series of ablation experiments as shown in Table 1. We start with the word token baseline classi er, and then add all four sets of features that help to increase performance as measured by accuracy. As we might expect, including the aspect feature has the most marked e ect on the performance of polarity classi cation, although all the features contributed to improving overall performance on stompol corpus.
Accuracy
Improvement Type Word token +Lemmas +pos tags +Aspects +Negations +1.52% +0.62% +1.68% +0.66%
Table 1: Results for polarity feature ablation experiments on stompol corpus
Due to the low participation of research teams in task 2 this year, we decided to compare our proposal to the systems presented this year and also to that ones of last year, because of the use of the same dataset.
For this reason, Table 2 compares results for our approach with di erent o cial ones submitted in 2015 and 2016 tass editions.
In this way, we compared our results for a ml approach based on well-known squaredregularised logistic regression with a snippet of length 4 (Lys-2) described in Vilares et al. (2015), a clustering method focused on grouping authors with similar sociolinguistic insights (TID-spark) described in Park (2015), a recurrent neural network composed of a single long short term memory and a logistic function (Lys-1) described in Vilares et al. (2015), a ml approach based on a svm with a snipped of length 5,7 and 10 (ELiRF) described in Hurtado, Pla, and Buscaldi (2015), and the best performing run of the actual task 2 tass edition (ELiRF-UPV).
Experiment
Task edition 2016 2015 2016 2015 2015 2015
Comparing the results, the performance of our current model is close from the top ranking systems of this and last year. 5
Conclusions and future works This paper describes the participation of the GTI group in the tass 2016, Task 2: AspectBased Sentiment Analysis. We developed a supervised system based on a svm classi er for the aspect-based sentiment analysis. The performance of our approach has been compared to that ones submitted this year but also to that ones submitted last year. Experimental results suggest that we need to include explore new features, such as word embedding representations or paraphrase (Zhao and Lan, 2015), in order to improve the performance.
As future work we plan to include new features explained before and to develop a new system which combines di erent ml classi cation methods. We are also interested in considering di erent paradigms of heterogeneous classi cation, such as deep learning to increase the performance.
References Alvarez-Lopez, T., J. Juncal-Mart nez,
M. Fernandez-Gavilanes, E. CostaMontenegro, and F. J. Gonzalez-Castano. 2016. Gti at semeval-2016 task 5: Svm and crf for aspect detection and unsupervised aspect-based sentiment analysis.
Proceedings of SemEval, pages 306{311.
Atserias, J., B. Casas, E. Comelles,
M. Gonzalez, L. Padro, and M. Padro. 2006. Freeling 1.3: Syntactic and semantic services in an open-source NLP library.
In Proceedings of LREC, volume 6, pages 48{55.
Bermingham, A. and A. F. Smeaton. 2011.
On using Twitter to monitor political sentiment and predict election results.
Brooke, J., M. To loski, and M. Taboada.
2009. Cross-linguistic sentiment analysis: From english to spanish. In G. Angelova, K. Bontcheva, R. Mitkov, N. Nicolov, and N. Nikolov, editors, RANLP, pages 50{ 54. RANLP 2009 Organising Committee / ACL.
Chang, C.-C. and C.-J. Lin. 2011. Libsvm: a library for support vector machines. ACM Transactions on Intelligent Systems and
Technology (TIST), 2(3):27.
Cui, H., V. Mittal, and M. Datar. 2006.
Comparative experiments on sentiment classi cation for online product reviews.
In Proceedings of the 21st National Conference on Arti cial Intelligence - Volume 2, AAAI'06, pages 1265{1270. AAAI
Press. dos Santos, C. N. and M. Gatti. 2014. Deep convolutional neural networks for sentiment analysis of short texts. In COLING, pages 69{78.
Fabo, P. R., M. Cuadros, and T. Etchegoyhen. 2013. Lexical normalization of spanish tweets with preprocessing rules, domain-speci c edit distances, and language models. In Proceedings of the Tweet Normalization Workshop co-located with 29th Conference of the Spanish Society for Natural Language Processing (SEPLN 2013), Madrid, Spain, September 20th, 2013., pages 59{63.
Fernandez-Gavilanes, M., T. Alvarez-Lopez,
J. Juncal-Mart nez, E. Costa-Montenegro, and F. J. Gonzalez-Castan~o. 2016. Unsupervised method for sentiment analysis in online texts. Expert Systems with Applications, 58:57{75.
Garc a-Cumbreras, M. A., J. Villena-Roman,
E. Mart nez-Camara, M. C. D az-Galiano, M. T. Mart n-Valdivia, and L. A. Uren~aLopez. 2016. Overview of tass 2016. In
Proceedings of TASS 2016: Workshop on Sentiment Analysis at SEPLN co-located with the 32nd SEPLN Conference (SEPLN 2016), Salamanca, Spain, September.
Han, B. and T. Baldwin. 2011. Lexical normalisation of short text messages: Makn sens a #twitter. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume 1, HLT '11, pages 368{378, Stroudsburg, PA, USA. Association for Computational Linguistics.
Hurtado, L. F., F. Pla, and D. Buscaldi. 2015. ELiRF-UPV en TASS 2015: Analisis de sentimientos en Twitter. In Proceedings of TASS 2015: Workshop on Sentiment Analysis at SEPLN colocated with 31st SEPLN Conference (SEPLN 2015), Alicante, Spain, September 15, 2015., pages 75{79.
Li, G. and F. Liu. 2010. A clustering-based approach on sentiment analysis. In Intelligent Systems and Knowledge Engineering (ISKE), 2010 International Conference on, pages 331{337. IEEE.
Liu, B. 2012. Sentiment Analysis and Opinion Mining. Synthesis Lectures on Human Language Technologies. Morgan & Claypool Publishers.
Mart nez-Camara, E., M. A. Garc a
Cumbreras, J. Villena-Roman, and J. Garc a-Morera. 2016. Tass 2015 - the evolution of the spanish opinion mining systems. Procesamiento del Lenguaje
Natural, 56:33{40.
Mohammad, S. M., S. Kiritchenko, and
X. Zhu. 2013. Nrc-canada: Building the state-of-the-art in sentiment analysis of tweets. In Proceedings of the seventh international workshop on Semantic Evaluation Exercises (SemEval-2013), Atlanta,
Georgia, USA, June.
Pak, A. and P. Paroubek. 2010. Twitter as a corpus for sentiment analysis and opinion mining. In N. C. C.
Chair), K. Choukri, B. Maegaard, J. Mariani, J. Odijk, S. Piperidis, M. Rosner, and D. Tapias, editors, Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10), Valletta, Malta, may. European Language Resources Association (ELRA).
Paltoglou, G. and M. Thelwall. 2012. Twitter, myspace, digg: Unsupervised sentiment analysis in social media. ACM Transactions on Intelligent Systems and
Technology (TIST), 3(4):66.
Pang, B. and L. Lee. 2008. Opinion mining and sentiment analysis. Found. Trends
Inf. Retr., 2(1-2):1{135, January.
Park, S. 2015. Sentiment classi cation using sociolinguistic clusters. In Proceedings of TASS 2015: Workshop on Sentiment Analysis at SEPLN co-located with 31st SEPLN Conference (SEPLN 2015), Alicante, Spain, September 15, 2015., pages 99{104.
Vilares, D., Y. Doval, M. A. Alonso, and
C. Gomez-Rodr guez. 2015. Lys at TASS 2015: Deep learning experiments for sentiment analysis on spanish tweets.
In Proceedings of TASS 2015: Workshop on Sentiment Analysis at SEPLN colocated with 31st SEPLN Conference (SEPLN 2015), Alicante, Spain, September 15, 2015., pages 47{52.
Wang, X., F. Wei, X. Liu, M. Zhou, and
M. Zhang. 2011. Topic sentiment analysis in Twitter: A graph-based hashtag sentiment classi cation approach. In Proceedings of the 20th ACM International Conference on Information and Knowledge Management, CIKM '11, pages 1031{ 1040, New York, NY, USA. ACM.
Zhao, J. and M. Lan. 2015. Ecnu: Leveraging word embeddings to boost performance for paraphrase in Twitter. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 34{39, Denver, Colorado, June. Association for Computational Linguistics.