This paper describes TASS 2016, the fifth edition of the Workshop on Sentiment Analysis at SEPLN. The main aim is the promotion of the research and the development of new algorithms, resources and techniques on the field of sentiment analysis in social media (specifically Twitter) focused on the Spanish language. This paper presents the TASS 2016 proposed tasks, the description of the corpora used, the participant groups, the results and analysis of them.
TASS is an experimental evaluation workshop, a satellite event of the annual SEPLN Conference, with the aim to promote the research on Sentiment Analysis in social media focused on the Spanish language. The fifth edition will be held on September 13th, 2016 at the University of Salamanca, Spain.
Sentiment Analysis (SA) is traditionally
defined as the computational treatment of
opinion, sentiment and subjectivity in texts
Although SA is not a new task, it is still
challenging, because the state of the art has not
yet resolved some problems related to
multilingualism, domain adaptation, text genre
adaptation and polarity classification at fine
grained level. Polarity classification has usually
been tackled following two main approaches.
The first one applies machine learning
algorithms in order to train a polarity classifier
using a labelled corpus
The aim of TASS is to provide a competitive forum where the newest research works in the field of SA in social media, specifically focused on Spanish tweets, are described and discussed by scientific and business communities.
The rest of the paper is organized as follows.
Section 2 describes the different corpus provided to participants. Section 3 shows the different tasks of TASS 2016. Section 4 describes the participants and the overall results are presented in Section 5. Finally, the last section shows some conclusions and future directions.
2
TASS 2016 experiments are based on two corpora, specifically built for the different editions of the workshop.
The two corpora will be made freely available to the community after the workshop. Please send an email to tass@sngularmeaning.team filling in the TASS Corpus License agreement with your email, affiliation (institution, company or any kind of organization) and a brief description of your research objectives, and you will be given a password to download the files in the password protected area. The only requirement is to include a citation to a relevant paper and/or the TASS website.
2.1
The General Corpus contains over 68.000 tweets, written in Spanish, about 150 wellknown personalities and celebrities of the world of politics, economy, communication, mass media and culture, between November 2011 and March 2012. Although the context of extraction has a Spanish-focused bias, the diverse nationality of the authors, including people from Spain, Mexico, Colombia, Puerto Rico, USA and many other countries, makes the corpus reach a global coverage in the Spanishspeaking world.
Each tweet includes its ID (tweetid), the creation date (date) and the user ID (user). Due to restrictions in the Twitter API Terms of Service (https://dev.twitter.com/terms/apiterms), it is forbidden to redistribute a corpus that includes text contents or information about users. However, it is valid if those fields are removed and instead IDs (including Tweet IDs and user IDs) are provided. The actual message content can be easily obtained by making queries to the Twitter API using the tweetid.
The general corpus has been divided into training set (about 10%) and test set (90%). The training set was released, so the participants could train and validate their models. The test corpus was provided without any tagging and has been used to evaluate the results. Obviously, it was not allowed to use the test data from previous years to train the systems.
Each tweet was tagged with its global polarity (positive, negative or neutral sentiment) or no sentiment at all. A set of 6 labels has been defined: strong positive (P+), positive (P), neutral (NEU), negative (N), strong negative (N+) and one additional no sentiment tag (NONE).
In addition, there is also an indication of the level of agreement or disagreement of the expressed sentiment within the content, with two possible values: AGREEMENT and DISAGREEMENT. This is especially useful to make out whether a neutral sentiment comes from neutral keywords or else the text contains positive and negative sentiments at the same time.
Moreover, the polarity values related to the entities that are mentioned in the text are also included for those cases when applicable. These values are similarly tagged with 6 possible values and include the level of agreement as related to each entity.
This corpus is based on a selection of a set of topics. Thematic areas such as “política” (“politics”), “fútbol” (“soccer”), “literatura” (“literature”) or “entretenimiento” (“entertainment”). Each tweet in the training and test set has been assigned to one or several of these topics (most messages are associated to just one topic, due to the short length of the text).
The annotation has been semi-automatically done: a baseline machine learning model is first run and then all tags are checked by human experts. In the case of the polarity at entity level, due to the high volume of data to check, the human annotation has only been done for the training set.
Table 1 shows a summary of the training and test corpora provided to participants.
Attribute Tweets Tweets (test) Tweets (test) Topics Users Date start (train) Date end (train) Date start (test) Date end (test) Value 68.017 60.798 (89%) 7.219 (11%) 10 154 2011-12-02 2012-04-10 2011-12-02 2012-04-10
Users were journalists (periodistas), politicians (políticos) or celebrities (famosos). The only language involved was Spanish (es).
The list of topics that have been selected is the following: • Politics (política) • Entertainment (entretenimiento) • Economy (economía) • Music (música) • Soccer (fútbol) • Films (películas) • Technology (tecnología) • Sports (deportes) • Literature (literatura) • Other (otros)
The corpus is encoded in XML. Figure 1 shows the information of two tweets. The first tweet is only annotated with the polarity at tweet level because there is not any entity in the text. However, the second one is annotated with the global polarity of the message and the polarity associated to each of the entities that appear in the text (UPyD and Foro Asturias). gathered from 23rd to 24th of April 2015, and are related to one of the following political aspects that appear in political campaigns: • Economics (Economía): taxes, infrastructure, markets, labour policy... • Health System (Sanidad): hospitals, public/private health system, drugs, doctors... • Education (Educación): state school, private school, scholarships... • Political party (Propio_partido): anything good (speeches, electoral programme...) or bad (corruption, criticism) related to the entity • Other aspects (Otros_aspectos): electoral system, environmental policy...
Each aspect is related to one or several entities that correspond to one of the main political parties in Spain, which are: • Partido_Popular (PP) • Partido_Socialista_Obrero_Español (PSOE) • Izquierda_Unida (IU) • Podemos • Ciudadanos (C’s) • Unión_Progreso_y_Democracia (UPyD)
Each tweet in the corpus has been manually annotated by two annotators, and a third one in case of disagreement, with the sentiment polarity at aspect level. Sentiment polarity has been tagged from the point of view of the person who writes the tweet, using 3 levels: P, NEU and N. Again, no difference is made between no sentiment and a neutral sentiment (neither positive nor negative). Each political aspect is linked to its correspondent political party and its polarity. Since the first edition of TASS, a new task and a new corpus have been published. However, one of the aims of TASS is the evaluation of the progress of the research on SA. Thus, the edition of 2016 was focused on the analysis and the comparison of the systems with the submissions of previous editions.
The edition of 2016 was focused on two tasks: polarity classification at tweet level and polarity classification at entity level. The polarity classification task has been proposed with the same corpus since the first edition of TASS, but the polarity classification at aspect level has been proposed with a different corpus each edition. In the edition of 2016 the classification at aspect level uses the STOMPOL corpus, which was published the first time in the edition of 2015.
Participants are expected to submit up to 3 results of different experiments for one or both of these tasks, in the appropriate format described below.
Along with the submission of experiments, participants have been invited to submit a paper to the workshop in order to describe their experiments and discussing the results with the audience in a regular workshop session.
The two proposed tasks are described next. 3.1
This task consists on performing an automatic polarity classification to determine the global polarity of each message in the test set of the General Corpus. The training set of the corpus was provided to the participants with the aim they could train and validate their models with it. There were two different evaluations: one based on 6 different polarity labels (P+, P, NEU,
N, N+, NONE) and another based on just 4 labels (P, N, NEU, NONE).
Participants are expected to submit (up to 3) experiments for the 6-labels evaluation, and they are also allowed to submit (up to 3) specific experiments for the 4-labels scenario.
Results must be submitted in a plain text file with the following format:
tweetid \t polarity where polarity can be: • P+, P, NEU, N, N+ and NONE for the 6-labels case • P, NEU, N and NONE for the 4-labels case.
The same test corpus of previous years was used for the evaluation in order to develop a comparison among the systems. The accuracy is one of the measures used to evaluate the systems, however due to the fact that the training corpus is not totally balanced the systems were also assessed by the macroaveraged precision, macro-averaged recall and macro-averaged F1-measure.
A corpus with the entities and the aspect identified was provided to the participants, so the goal of the systems is the inference of the polarity at the aspect-level. As in 2015, STOMPOL corpus was the corpus used in this task. STOMPOL was divided in training and test set, the first one for the development and validation of the systems, and the second for evaluation.
Participants are expected to submit up to 3 experiments for each corpus, each in a plain text file with the following format: tweetid \t aspect-entity \t polarity
Allowed polarity values are: P, N and NEU. For the evaluation, a single label combining “aspect-polarity” has been considered. As in the first task, accuracy, macro-averaged precision, macro-averaged recall and macro-averaged F1measure have been calculated for the global result.
4
This year 7 (7 last year) groups submitted their systems The list of active participant groups is shown in Table 3, including the tasks in which they have participated.
Six of the seven participant groups sent a report describing their experiments and results achieved. Papers were reviewed and included in the workshop proceedings. References are listed in Table 4.
Group jacerong ELiRF-UPV LABDA INGEOTEC GASUCR GTI SINAI_w2v Total Submitted runs and results for Task 1, evaluation based on 5 polarity levels with the whole General test Corpus are shown in Table 5. Accuracy, macro-averaged precision, macroaveraged recall and macro-averaged F1measure have been used to evaluate each individual label and ranking the systems.
In order to perform a more in-depth evaluation, results are calculated considering the classification only in 3 levels (POS, NEU, NEG) and no sentiment (NONE) merging P and P+ in only one category, as well as N and N+ in another one. The results reached by the submitted systems are shown in Table 6. 5.2
Submitted runs and results for Task 2, with the STOMPOL corpus, are shown in Table 7. Accuracy, macro-averaged precision, macroaveraged recall and macro-averaged F1measure have been used to evaluate each individual label and ranking the systems.
Run Id GTI ELiRF-UPV_1 0.526
M-F1
The systems submitted in the edition of 2016 represent the next step of the ones submitted in the previous edition. The systems may be cluster in two groups, those ones that rely on the classification power of the ensemble of several base classifiers, and those systems that change the use traditional Bag-of-Words model for the use of vectors of word embeddings in order to represent the meaning of each word. In the subsequent paragraphs the main features of the systems submitted are going to be depicted.
The system depicted in (
Alvarez et al. (2016) exposed the participation of the team GTI on the task 2. The system is similar to the system of the team ELiRF-UPV in the sense that it is composed by two layers: context identification and polarity classification. Regarding the identification of the context, the authors design a heuristic method based on lexical markers. The polarity classification system is a SVM classifier that uses different type of features in order to represent the contexts of the entities and the aspects.
The system presented by the team LABDA
Casasola Murillo and Marín Reventós
(2016) submitted an unsupervised system based
on the system described in
In Table 5 and Table 6 are shown the results of each system and they are ranked by the F1score reached, so it is not hard to know what is the best system in the edition of 2016.
On the other hand, how many tweets were rightly classified by the submitted systems? Is there a set of tweets that were not rightly classified by any system? What are the most difficult tweets to classify? These questions are going to be answered in the following paragraphs?
Table 8 shows the rate of tweets that are rightly classified by a number of systems. There are about a 6% of tweets whose polarity is not inferred by any of the submitted systems. In other words, the submitted systems in the edition of 2016 are able to classify about the 94% of the test set. So, what is the main features of that 6% of tweets that any system inferred their polarity? Number of systems 0 1 2 3 4 5 6 7 8 9 10 11 12 13
Figures Figure 3,Figure 4Figure 5 are three examples of tweets that were not rightly classified by any system. The common feature of the three tweets is that they do not have any lexical marker that express emotion or opinion. Moreover, the tweet of the Figure 4 is sarcastic, which means an additional challenging for SA because requires a deep understanding of the language.
Id: 177439342497767424 hahahahahaha “@Absolutexe: ¿Le han cambiado ya el nombre a la Junta de Andalucía por la Banda de Andalucía o aún no?” hahahahahaha “@Absolutexe: Has the Junta de Andalucía renamed Gang of Andalucía or not yet?”
Polarity: N+
All the systems submitted are based on linear classifiers that do not take into account the context of each word, which means a big drawback for the understanding the meaning of a span of text.
The tweets of the Figures 3, 4 and 5 show that opinions and emotions are not only expressed by lexical markers, so the future participants should take into account the challenging task of implicit opinion analysis, irony and sarcasm detection. These new problems may be framed on the semantic level of Natural Language Processing and should be tackled by the research community in order to go a step further in the understanding of the subjective information, which is continuously published on the Internet. 6
TASS was the first workshop about SA focused on the processing of texts written in Spanish. In the three first editions of TASS, the research community were mainly formed by Spanish researchers, however since the last edition, the researchers that come from South America is making bigger, so it is an evidence that the research community of Sentiment Analysis in Spanish is not only located in Spain and is formed by the Spanish speaking countries.
Anyway, the developed corpus and gold standards, and the reports from participants will for sure be helpful for knowing the state of the art in SA in Spanish.
The future work will be mainly focused on the definition of a new General Corpus because of the following reasons: 1. The language used on Twitter changes faster than the language used in traditional genres of texts, so the update of the corpus is required in order to cover a real used of the language on Twitter. 2. After several editions of the workshop, we realize that the quality of the annotation is not extremely good, so it is required to define a new corpus with a high quality annotation in order to provide a real gold standard for Spanish SA on Twitter. 3. The research community deeply know the General Corpus of TASS and it wants a new challenge.
A significant amount of new tasks is currently being defined in Natural Language Processing, so some of them, such as stance classification, will be studied to be proposal for the next edition of TASS.
This work has been partially supported by a grant from the Fondo Europeo of Desarrollo Regional (FEDER) and REDES project (TIN2015-65136-C2-1-R) from the Spanish Government.