The Task on Semantic Analysis at SEPLN (TASS task within IberLEF 2020 workshop) took place on September 22, reaching its ninth edition. Due to the COVID-19 pandemic, the number of participants is lower compared to past campaigns. Also, the organizers decided to held it remotely. In this edition, the classical polarity classification subtask was, again, organized. As a novelty, a second subtask was proposed to foster research in emotion detection of Spanish texts on a new dataset. This paper summarizes the diferent approaches of the teams who participated, the key insights of their systems and the results obtained for all the proposed solutions.
Sentiment Analysis (SA) is still a challenging task, because the dificulties of processing some underlying linguistic phenomena as negation, irony, sarcasm and more broadly speaking the subjectivity of the opinions. However, there are additional uses of subjective language to express private states beyond opinions1][. Humans also express their private states through the description of their emotional state. Hence, emotion analysis (EA) represents an additional challenge to the processing of subjective language.
The task on semantic analysis at SEPLN (TASS), belonging to the IberLEF workshop, addressed
as a shared task the main challenges in SA in Spanish. The edition of TASS 2020 has as main
novelty to face up the task of EA in Spanish. Specifically, TASS 2020 proposes two subtasks:
1. General polarity at three levels. It corresponds to the default subtask of TASS, which
consists of classifying the opinion meaning of tweets written in Spanish. Since the edition
of 2017, one of the aims of TASS is the elaboration of a representative Spanish dataset
for SA in Twitter. Accordingly, the InterTASS dataset was presented and used for the
ifrst time in the edition of 2017 [
Three research teams submitted several classification results to the Subtask 1, and two teams submitted to the Subtask 2.
The details of the systems submitted are described in Sectio2n.2s and2.3.
The International Conference of the Spanish Society for Natural Language Processing (SEPLN) supports it and the workshop is held within the SEPLN 2020 event.
Spanish is the fourth most spoken language and the second one on Twitter. These facts inspire TASS to create new systems of language comprehension and new resources for NLP and, more specifically, for sentiment analysis.
Many resources have been developed within the framework of the TASS tasks. In this edition,
InterTASS [
Task 1, General polarity at three levels, with two sub-tasks: • Monolingual, in which only the dataset of the variety of Spanish to be undertaken has been used. • Multivariant, in which the test dataset has been built with a set of tweets from all the varieties in the corpus.
Task 2, Emotion detection. It has been designed to encourage research in the understanding the emotions expressed by users on social media. This is a hard task due to the absence of voice modulations and facial expressions.
The task consists of classifying the emotion expressed in a tweet anseu“tral or no emotion” or as one of the six Ekman’s basic emotionsa:nger, disgust, fear, joy, sadness andsurprise.
In this section we describe in detail the corpus used and the two proposed tasks.
In this edition, only three teams presented their systems and results, whose main features are detailed below.
Palomino Team [
EliRF-UPV [
International TASS Corpus (InterTASS) was released in 2091]7, u[pdated in 2018 [
In this edition, we have introduced a novelty: the delivered corpora contains only 3 diferent tags or levels of opinion intensity (P, N, NEU), where NEU includes NONE.
Evaluation measures: accuracy and the macro-averaged versions of Precision, Recall, and F1 have been used as evaluation measures. Systems were ranked by the Macro-F1 and Accuracy measures.
For sub-task 1.1, monolingual classification, the training and evaluation used each data set of the InterTASS varieties (ES-Spain, CR-Costa Rica, PE-Peru, UY-Uruguay and MX-Mexico), but, unlike in past editions, the use of any other corpora or linguistic resources has been allowed.
For sub-task 1.2, multivariant classification, a new test dataset was delivered, with tweets extracted from the diferent test subsets of Spain, Costa Rica, Peru, Uruguay, and Mexico. These tweets were specially selected from the most dificult tweets of the last edition (tweets that were missed by most teams). Again, unlike in past editions, the use of any other corpora or linguistic resources has been allowed.
The submitted systems have had to face up with the following challenges: • Lack of context: the source elements are tweets. • Informal language: misspelling, emojis and onomatopoeia are common. • Multivariety: the datasets have been developed with tweets written in the Spanish language spoken in Spain, Peru, Costa Rica, Uruguay and Mexico. Team ELiRF-UPV Palomino-Ochoa UMUTeam Team ELiRF-UPV Palomino-Ochoa UMUTeam Team ELiRF-UPV Palomino-Ochoa UMUTeam
Team Palomino-Ochoa ELiRF-UPV UMUTeam
Set ES ES ES Set CR CR
CR Set PE PE PE
Set UY UY UY • Generalization: the systems will be assessed with several datasets of tweets written in the Spanish language spoken in diferent countries
The goal with this task is to evaluate how systems classify polarity at tweet level, for tweets written in Spanish, in an environment of a previously known variety of Spanish. That is, each variety of Spanish has a test corpus that must be evaluated using any source.
Tables1, 2, 3, 4 and 5 show the final results for Spanish, Costa Rican, Peruvian, Uruguayan and Mexican variant sub-tasks.
The goal with this task is to evaluate how systems classify polarity at tweet level, for tweets written in Spanish, in a multivariant environment.
This task has been introduced for the first time this year (TASS 2020) with the purpose of encouraging research in the emotion detection task for Spanish. While polarity classification is a well-established task with many standard datasets and well-defined methodologies, emotion detection has received less attention due to its complexity. In fact, it can be considered a further step in the task of polarity classification since it consists of detecting fine-grained emotions in text, not just positive or negative polarity.
The goal of Task 2: Emotion detection is to classify the main emotion expressed in a tweet as “neutral or no emotion” or as one of the six Ekman’s basic emotions3[] that best represent the mental state of the tweeter. The emotion categories are listed below, along with some synonyms: • Joy, including serenity and ecstasy. • Sadness, including pensiveness and grief. • Anger, including annoyance and rage. • Surprise, including distraction and amazement. • Disgust, including disinterest, dislike, and loathing. • Fear, including apprehension, anxiety, concern and terror.
• Others, no emotion or neutral.
For this task, we use EmoEvent 5[], a multilingual emotion corpus of tweets based on events that took place in April 2019. They are related to diferent domains such as entertainment, catastrophe, political, global commemoration, and global strike. Each instance in the dataset were labeled with the main emotion expressed in the tweet by three annotators according to the following categorieas:nger, disgust, fear, joy, sadness, surprise, “neutral or no emotion”. The authors decided the final emotion label of the tweet as the majority emotion labeled by the annotators, but in case the three annotators labeled the tweet with diferent emotions, the ifnal label is “neutral or no emotion”. In particular, for this task we use the Spanish version of EmoEvent that contains a total of 8,409 tweets written in Spanish.
With the purpose of providing the dataset to the participants, we decided to replace the hashtags by the keywordHASHTAG in order to prevent the automatic classifier from relying on hashtags to categorize the emotion associated with a tweet. Moreover, we replaced the user mentions by @USER to anonymize mentions to users. Finally, training, development and test sets have been released for the participants. Ta7blsehows the number and percentage of tweets corresponding to each partition by emotion.
The systems submitted have had to cope with the following challenges: • Lack of context: the source elements are tweets and the length of each tweet is limited (up to 240 characters). • Informal language: misspellings, emojis, onomatopoeias are common in tweets.
For Task 2: Emotion Detection, two teams have submitted their systems. Tab8leshows the final results obtained by the participant systems on the test set of Task 2. The ELiRF-UPV team obtained the best macro averaged F1-score of 0.447. They take advantage of BERT by adapting the model on Spanish tweets, then they established a baseline based on Deep Averaging Networks to compare their results. The UMUTeam has achieved the performance of 0.379, presenting a system based on the combination of linguistic features and word embeddings. As a classifier, they use the sequential minimal optimisation algorithm which is based on support vector machines.
The 2020 edition of TASS continues to work with the InterTASS dataset, introducing a new task that works with a multivariant test set, common to all participating teams, and introduces as a novelty a new task: emotion detection.
Although 14 participating teams were registered, in the end only three presented results. In the diferent tasks with the InterTASS corpus we highlight the robustness of the systems, without significant variations between monolingual tasks and the new multivariate task.
Regarding the task of emotion detection, taking into account that the dataset provided is not balanced among the eight categories of emotion raised, the systems presented results that pose many improvements in the future to address this complex task, pending an in-depth analysis of each system, which will be reflected in the paper of each participant.
The future of TASS is to establish as a main line of work this detection of emotions as a natural evolution of the analysis of feelings.
This work has been partially supported by a grant from the Spanish Government under the LIVING-LANG project (RTI2018-094653-B-C21) . Eugenio Martínez Cámara was supported by the Spanish Government Programme Juan de la Cierva Formación (FJCI-2016-28353).