Overview of TASS 2019: One More Further for the Global Spanish Sentiment Analysis Corpus?

Abstract

In September 2019, the eighth edition of TASS workshop (Task of Sentiment Analysis at SEPLN) was held in Bilbao, Spain as part of the rst edition of IberLEF (Iberian Languages Evaluation Forum), which joined the e orts of the IberEval and TASS workshops. In this edition, the natural evolution from previous editions was proposed: sentiment analysis at tweet level. It includes two subtasks, monolingual and cross-lingual sentiment analysis, with di erent subsets of the InterTASS corpus (ES-Spain, PE-Peru, CR-Costa Rica, UR-Uruguay and MX-Mexico). This paper summarizes the approaches and the results of the submitted systems of the di erent groups for each task.

Keywords

• Sentiment Analysis Opinion Mining Social Media

1 Introduction

is the result of the association of some workshops in the Natural Language Processing (NLP) domain for Spanish and other languages spoken in the Iberian peninsula, and the aim of joining forces of di erent NLP research communities in order to provide a common forum for assessing NLP systems and interchanging research ideas, issues, challenges and experiences.

Since the edition of TASS 2017 [ 3 ], the main aim is the elaboration of a corpus of sentiment tweets written in di erent Spanish variants, in order to provide a representative corpus of Spanish posts written in microblogs all over the world and not only of the usage of the Spanish language in Spain. Accordingly, the International TASS corpus was released for the rst time in the edition of 2017, and it was only composed of tweets written in the Spanish used in Spain. The second version of InterTASS was released in the edition of TASS 2018 [ 4 ], and set of tweets written in the Spanish used in Peru and Costa Rica were added. The novelty of the edition of TASS 2019 lies in the incorporation of two new Spanish variants, namely the Spanish written in social media in Mexico and Uruguay.

The goal behind the aim of compiling a global Spanish corpus of tweets is to study the di erences among di erent versions of Spanish, and fostering the crosslingual research on the Spanish language. Consequently, the TASS 2019 proposed two subtasks, speci cally a mono-lingual polarity classi cation task (Subtask 1) and a cross-lingual polarity classi cation task (Subtask 2) (see Section 2.1).

Seven research teams submitted several classi cation results to the Subtask 1, and four teams submitted to the Subtask 2. The systems submitted go in the line of the state of the art in similar workshops, and the participants developed classi cation systems based on Recurrent Neural Networks, Transformer Networks and ne-tunning models built upon BERT [ 2 ]. The details of the systems submitted are described in Sections 2.2 and 2.3. 2

Spanish Semantic Analysis Tasks

The workshop \Sentiment Analysis at SEPLN (TASS)" has been held since 2012, under the umbrella of the International Conference of the Spanish Society for Natural Language Processing (SEPLN).

Spanish is the second language used in Twitter, what calls for the development of new language comprehension systems and the opportunity of creation of resources for NLP and, more speci cally, for sentiment analysis.

Many resources have been developed under TASS tasks. In this edition, we have completed the InterTASS corpus [ 4 ] with Uruguayan and Mexican Spanish variants. The workshop has been built over 2 general task: monolingual and multilingual approaches, over combinations of the ve di erent datasets of Spanish language variants.

In this section we describe the entire InterTASS corpus and the two proposed tasks. 2.1

Corpus datasets International TASS Corpus (InterTASS) is a corpus released in 2017 [ 3 ] that was updated in 2018 [ 4 ]. In this edition, it has been extended with new texts written in two new Spanish variants: Uruguayan and Mexican.

Therefore, this last version contains tweets written in ve di erent variants of Spanish from Spain, Peru, Costa Rica, Uruguay and Mexico, and it exhibits a large amount of lexical and even structural di erences in each variant. In this edition, participants have had to face this ve di erent variants of Spanish to train and tests their systems.

Spanish dataset The Spanish dataset was released in 2017 as the rst version of InterTASS. Its contains 3,401 tweets in Spanish by users from Spain, and it is a subset of a biggest corpus collected from July 2016 to January 2017. Each tweet was labeled with its level of polarity, which can be positive (p), neutral (neu), negative (n) and no sentiment tag (none). Each tweet was annotated at least by three annotators. The dataset was originally split into three datasets that have been reorganized this year, in order to homogenize all the datasets. The new partitions contain a training set with 1,126 tweets, a development set with 569 tweets and a test set with 1,706 tweets. Table 1 shown the general statistics of Spanish dataset. Costa Rican dataset The Costa Rican dataset was created in 2018. It contains 2,363 tweets. The annotation methodology replicated the one used to label the Spanish dataset. Each tweet was labeled as positive (p), neutral (neu), negative (n) and no sentiment tag (none). Every tweet was labeled by three annotators. Agreement was reached for 2,048 tweets. For the extra tweets, two more annotators were required to obtain agreement. The dataset also has been reorganized in order to homogenize the entire corpus. Table 2 shows the new composition. Peruvian dataset This dataset is comprised by 3,005 tweets in the Peruvian Spanish variant. The annotation of the dataset was performed as follows. First, three annotators labeled all the tweets independently. Then, tweets with total or partial agreement (with agreement between two annotators at least) were included into the dataset. Tweets where annotators totally disagreed were labeled by two additional annotators. After this, the rst annotator decided the label of the tweets where the disagreement continued. Finally, all tweets were included into the dataset. This partition has also been re-balanced. Table 3 shows the distribution of tweets according to classes in the Peruvian Spanish variant. Uruguayan dataset The Uruguayan dataset is comprised of 2,857 tweets in the Uruguayan Spanish variant. The annotation process consisted of two phases. First, of all three annotators independently labeled all the tweets. After this rst step, two more annotators relabeled the tweets that got three di erent votes in the rst round. The few tweets that were still ambiguous after this process were discussed between the annotators in order to get a consensus. Table 4 shows the distribution of tweets in the Uruguayan Spanish variant. Mexican dataset The Mexican dataset contains 3,000 tweets in the Mexican Spanish variant. It was generated through a labeling process done with four annotators. This labeling process consisted of the following steps: 1) for each tweet, each annotator assigned a polarity of the set fP, N, NEU, NONEg; 2) from the labels assigned by all annotators, if there is a predominant label, this is assigned as the class of the tweet, and 3) in the case of no predominant label, another annotator intervened to obtain a predominant label for nal assignment. The resulting distribution of tweets can be seen in Table 5. The main goal of this task is the evaluation of polarity classi cation systems at tweet level for tweets written in Spanish in a monolingual environment. That is, the aim is to evaluate systems designed and trained for each individual variant.

The submitted systems will have to face up with the following challenges: { Lack of context: the source elements are tweets. { Informal language: misspelling, emojis and onomatopoeia are common. { Multilinguality (local): the datasets have been developed with tweets written in the Spanish language spoken in Spain, Peru, Costa Rica, Uruguay and Mexico. { Generalization: the systems will be assessed with several datasets of tweets written in the Spanish language spoken in di erent countries.

In this task, the participating teams could only perform monovariety experiments using InterTASS dataset (ES-Spain, PE-Peru, CR-Costa Rica, URUruguay and MX-Mexico), so ve rankings have been prepared, one for each Spanish variant.

Systems presented Seven teams presented their systems and results for this rst task, whose main features are detailed below.

Atalaya Team [ 8 ] System inspired in [ 9 ]. Di erent representations of the data have been used, such as bag-of-words, bag-of-characters and tweet embeddings and they have trained robust subword-aware word embeddings and computed tweet representations using a weighted-averaging strategy. The novelty of the system is the use of two data augmentation techniques to deal with data scarcity: two-way translation augmentation, and a novel technique that generates new instances by combining halves of tweets.

LaSTUS/TALN Team [ 1 ] The system proposes a deep learning approach based on bidirectional LSTM (biLSTM) models to face both sub-tasks. The tweets are tokenized keeping emojis and full hashtags and they are transformed in a embedding process.

GTH-UPM Team [ 6 ] The developed system consisted of three classi ers: a system based on feature vectors extracted from the tweets, a neural-based classier using FastText and a deep neural network classi er using contextual vector embeddings created using BERT (Bidirectional Encoder Representations from Transformers). The averaged probability of the three classi ers was calculated to get the nal score.

ELiRF-UPV [ 7 ] proposed a system focused mainly on employing the encoders of the Transformer model, based on self-attention mechanisms. The Transformer model dispenses with convolution and recurrences to learn long-range relationships. They use only the encoder part in order to extract vector representations that are useful to perform sentiment analysis. They denote this encoding part of the Transformer model as Transformer Encoder. The results obtained were very promising, being the rst or second ranked system on almost all the Spanish variants.

The Titans [ 5 ] use a bidirectional LSTM based approach to capture information from both the past and future context followed by an attention layer consisting of initializers and regularizers.

RETUYT-InCo [ 10 ] presents three approaches for classifying the sentiment of tweets for di erent Spanish variants. In the rst one, they consider multiple variants to perform a classi cation of the sentence word vectors mean, performing the classi cation through layered fully connected neural networks and support vector machines. The second approach relies on transfer learning from a pretrained Spanish BERT. The third approach is based on the use of FastText embeddings as input to an LSTM neural network. The MLP based approach achieved good results in monolingual experiments while the BERT based system performed better in the crosslingual task.

ITAINNOVA [ 11 ] explores two di erent deep learning approaches. The rst one with an embedding-based strategy combined with bidirectional recurrent neural networks (an architecture that learns the representation of input documents as a concatenation of self-learned char-embeddings with sequence word-embeddings), and the second one using the new method of pre-trained BERT. Although the performance of the second approach has not been presented as o cial results, it is reasonably remarkable and higher than the winner approach.

Tables 6, 7, 8, 9 and 10 show the results obtained on the Spain, Peru, Costa Rica, Uruguay and Mexico test datasets respectively. ELiRF-UPV team obtained the overall best results. The purpose of this task is similar to that of Task 1, but systems must be trained with one or more Spanish variants and tested with a di erent Spanish variant. The Spanish variant of training set had to be di erent from the evaluation one, in order to test the dependency of systems on a language.

Six teams have participated in this task: Atalaya Team, LaSTUS/TALN Team, GTH-UPM Team, The Titans Team, ITAINNOVA Team and RETUYTInCo. The systems are the same as those described in section 2.2. 0.462 0.456 0.437 0.588 0.498 0.472

Tables 11, 12, 13, 14 and 15 show the results obtained on the test sets for Spanish variants of Spain, Peru, Costa Rica, Uruguay and Mexico respectively. In three of the ve experiment results the Atalaya team obtained the best results, being the second in the evaluation of Spanish for Costa Rica.

The values obtained in the evaluation of this task are very similar to those of Task 1, although slightly lower, which is reasonable as no training data from the target Spanish variant was allowed. The 2019 edition of TASS has attracted the participation of 13 systems, seven for the rst task (monolingual sentiment analysis), and six for the second task (crosslingual). Seven papers with the description of the evaluated systems were presented. This year, new datasets for the InterTASS corpus have been added, enlarging this reference corpus for the Spanish sentiment analysis task.

The submitted systems are in the line the state-of-the-art approaches in other similar workshops, and most of them are grounded in Deep Learning and the use of hand-crafted linguistic features.

As future work, we plan to consolidate the InterTASS corpus to the Spanishspeaking community, with new challenges for the next year. Moreover, we will 0.517 0.458 0.450 0.474 0.465 0.455 keep working in the development of new corpora and linguistic resources for the research community.

References

1. Altin , L.S.M. , Bravo , A. , Saggion , H.: Lastus/taln at tass 2019: Sentiment analysis for spanish language variants with neural networks . In: D az- Galiano , M.C. , Garc a-Vega, M. , Casasola , E. , Chiruzzo , L. , Garc a-Cumbreras, M.A. , Mart nezCamara , E., Moctezuma , D. , Montejo Raez , A. , Sobrevilla Cabezudo , M.A. , Tellez , E. , Gra , M. , Miranda , S. (eds.) Proceedings of TASS 2019: Workshop on Semantic Analysis at SEPLN (TASS 2019 ). CEUR Workshop Proceedings , vol. ??? CEURWS, Bilbao, Spain ( September 2019 )

2. Devlin , J. , Chang , M.W. , Lee , K. , Toutanova , K. : BERT: Pre-training of deep bidirectional transformers for language understanding . In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , Volume 1 (Long and Short Papers). pp. 4171 { 4186 . Association for Computational Linguistics, Minneapolis, Minnesota (Jun 2019 ), https://www.aclweb.org/anthology/N19-1423

3. D az-Galiano , M.C. , Mart nez-Camara, E. , Garc a Cumbreras, M.A. , Garc a Vega, M. , Villena Roman , J.: The democratization of deep learning in TASS 2017. Procesamiento del Lenguaje Natural 60 , 37 { 44 ( 2018 ), http://journal.sepln.org/sepln/ojs/ojs/index.php/pln/article/view/5556

4. Daz-Galiano , M.C. , Garca-Cumbreras , M.. , Garca-Vega , M. , Gutirrez , Y. , Martnez Cmara , E. , Piad-Mor s , A. , Villena-Romn , J. : Tass 2018 : The strength of deep learning in language understanding tasks . Procesamiento del Lenguaje Natural 62 , 77 { 84 ( 2019 ), http://journal.sepln.org/sepln/ojs/ojs/index.php/pln/article/view/5955

5. Garain , A. , Mahata , S.K. : Sentiment analysis at sepln (tass)-2019: Sentiment analysis at tweet level using deep learning . In: D az- Galiano , M.C. , Garc a-Vega, M. , Casasola , E. , Chiruzzo , L. , Garc a-Cumbreras, M.A. , Mart nez-Camara, E. , Moctezuma , D. , Montejo Raez , A. , Sobrevilla Cabezudo , M.A. , Tellez , E. , Gra , M. , Miranda , S. (eds.) Proceedings of TASS 2019: Workshop on Semantic Analysis at SEPLN (TASS 2019 ). CEUR Workshop Proceedings , vol. ??? CEUR-WS , Bilbao, Spain ( September 2019 )

6. Godino , I.G. , DHaro , L.F.: Gth-upm at tass 2019: Sentiment analysis of tweets for spanish variants . In: D az- Galiano , M.C. , Garc a-Vega, M. , Casasola , E. , Chiruzzo , L. , Garc a-Cumbreras, M.A. , Mart nez-Camara, E. , Moctezuma , D. , Montejo Raez , A. , Sobrevilla Cabezudo , M.A. , Tellez , E. , Gra , M. , Miranda , S. (eds.) Proceedings of TASS 2019: Workshop on Semantic Analysis at SEPLN (TASS 2019 ). CEUR Workshop Proceedings , vol. ??? CEUR-WS , Bilbao, Spain ( September 2019 )

7. Angel Gonzalez , J. , Hurtado , L.F. , Pla , F. : Elirf-upv at tass 2019: Transformer encoders for twitter sentiment analysis in spanish . In: D az- Galiano , M.C. , Garc aVega , M. , Casasola , E. , Chiruzzo , L. , Garc a-Cumbreras, M.A. , Mart nez-Camara, E. , Moctezuma , D. , Montejo Raez , A. , Sobrevilla Cabezudo , M.A. , Tellez , E. , Gra , M. , Miranda , S. (eds.) Proceedings of TASS 2019: Workshop on Semantic Analysis at SEPLN (TASS 2019 ). CEUR Workshop Proceedings , vol. ??? CEUR-WS , Bilbao, Spain ( September 2019 )

8. Luque , F.M. : Atalaya at tass 2019: Data augmentation and robust embeddings for sentiment analysis . In: D az- Galiano , M.C. , Garc a-Vega, M. , Casasola , E. , Chiruzzo , L. , Garc a-Cumbreras, M.A. , Mart nez-Camara, E. , Moctezuma , D. , Montejo Raez , A. , Sobrevilla Cabezudo , M.A. , Tellez , E. , Gra , M. , Miranda , S. (eds.) Proceedings of TASS 2019: Workshop on Semantic Analysis at SEPLN (TASS 2019 ). CEUR Workshop Proceedings , vol. ??? CEUR-WS , Bilbao, Spain ( September 2019 )

9. Luque , F.M. , Perez , J.M. : Atalaya at tass 2018: Sentiment analysis with tweet embeddings and data augmentation . In: Mart nez- Camara , E. , Almeida Cruz , Y. , D az- Galiano , M.C. , Estevez Velarde , S. , Garc a-Cumbreras, M.A. , Garc a-Vega, M. , Gutierrez Vazquez , Y. , Montejo Raez , A. , Montoyo Guijarro , A. , Mun~oz Guillena, R. , Piad Mor s, A. , Villena-Roman , J . (eds.) Proceedings of TASS 2018: Workshop on Semantic Analysis at SEPLN (TASS 2018 ). CEUR Workshop Proceedings , vol. 2172 . CEUR-WS , Sevilla, Spain ( September 2018 )

10. Pastorini , M. , Pereira , M. , Zeballos , N. , Chiruzzo , L. , Rosa , A. , Etcheverry , M. : Retuyt-inco at tass 2019: Sentiment analysis in spanish tweets . In: D az- Galiano , M.C. , Garc a-Vega, M. , Casasola , E. , Chiruzzo , L. , Garc a-Cumbreras, M.A. , Mart nez-Camara, E. , Moctezuma , D. , Montejo Raez , A. , Sobrevilla Cabezudo , M.A. , Tellez , E. , Gra , M. , Miranda , S. (eds.) Proceedings of TASS 2019: Workshop on Semantic Analysis at SEPLN (TASS 2019 ). CEUR Workshop Proceedings , vol. ??? CEUR-WS , Bilbao, Spain ( September 2019 )

11. nes Salas , R.M.M. , del Hoyo-Alonso, R. , Aznar-Gimeno , R. : From recurrency to attention in opinion analysis. comparing rnn vs transformer models . In: D az- Galiano , M.C. , Garc a-Vega, M. , Casasola , E. , Chiruzzo , L. , Garc a-Cumbreras, M.A. , Mart nez-Camara, E. , Moctezuma , D. , Montejo Raez , A. , Sobrevilla Cabezudo , M.A. , Tellez , E. , Gra , M. , Miranda , S. (eds.) Proceedings of TASS 2019: Workshop on Semantic Analysis at SEPLN (TASS 2019 ). CEUR Workshop Proceedings , vol. ??? CEUR-WS , Bilbao, Spain ( September 2019 )