UMUTeam at TASS 2020: Combining Linguistic Features and Machine-learning Models for Sentiment Classification José Antonio García-Díaza , Ángela Almelab and Rafael Valencia-Garcíaa a Facultad de Informática, Universidad de Murcia, Campus de Espinardo, 30100, Spain b Facultad de Letras, Universidad de Murcia, Campus de La Merced, 30001, Murcia, Spain Abstract This paper describes the participation of the UMUTeam at the TASS’2020 Workshop on Sentiment Anal- ysis, in which two tracks were proposed. The first track consists in the classification of tweets according to general sentiments of tweets written in several Spanish varieties, whereas the second task consists in a fine-grained distinction between the six basic emotions described by Ekman (2009). Our proposal is based on the usage of linguistic features alone or in combination with word-embeddings. Specifi- cally, we test Convolutional Neural Networks and Support Vector Machines with sentence embeddings. Although our proposal did not achieve the best results, we obtained the best precision rate regarding emotion detection (Task 2) and competitive results with respect to the general sentiment classification in which tweets written in different varieties of Spanish were mixed. We consider that our proposal, despite its limitations, provides substantial benefits such as the interpretability of the results. Keywords Sentiment Analysis, Supervised learning, Deep Learning, Machine learning 1. Introduction This paper describes the participation of the UMUTeam in the TASS’2020 Workshop on Senti- ment Analysis (SA). SA is a trending task of Natural Language Processing (NLP) which consists in the extraction and classification of subjective opinions in texts written in natural language. Two tasks were proposed to the participants. The first one involved extracting the subjectivity polarity from tweets written in different varieties of Spanish, including European Spanish and different varieties from Latin America. This task was divided into two sub-tasks: (1) monolingual classification, in which several training and testing splits from different Spanish varieties were considered separately: Spain (es), Costa Rica (cr), Uruguay (uy), and Mexico (mx); and (2) multi-variant classification, in which a testing dataset consisting of mixed tweets from the above Spanish varieties were mixed. In both sub-tasks the participants were required to determine if tweets express positive, negative or neutral sentiments. The second task consisted in a multi- class classification of emotions based on six of the basic emotions proposed by Ekman (2009) Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2020) email: joseantonio.garcia8@um.es (J.A. García-Díaz); angelalm@um.es (.́ Almela); valencia@um.es (R. Valencia-García) orcid: 0000-0002-3651-2660 (J.A. García-Díaz); 0000-0002-1327-8410 (.́ Almela); 0000-0003-2457-1791 (R. Valencia-García) © 2020 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings http://ceur-ws.org ISSN 1613-0073 CEUR Workshop Proceedings (CEUR-WS.org) [1]: anger, disgust, fear, joy, sadness and surprise along with others in order to classify those tweets which do not agree with the aforementioned labels. All datasets, regardless of the task, were unbalanced: Task 1 contained more tweets labelled as neutral and Task 2 contained more tweets labelled as others. We participated both in Task 1 and Task 2. In a nutshell, our proposal consisted in testing the reliability of linguistic features alone or in combination with well-known machine-learning and deep-learning models. Linguistic features, in comparison with statistical approaches, ease the interpretability of the results because they are conceptually higher-level features than words. With our participation, we attempted to answer the following research questions: • RQ1. Is the usage of linguistic features enough to compete with state-of-the-art approaches based on statistical methods? • RQ2. Can linguistic features be combined with statistical methods in order to improve the accuracy of the results while keeping interpretability? • RQ3. Is the reliability of linguistic features affected by different cultural background in the same language? To answer these research questions, we extracted linguistic features from the datasets by using a self-developed tool designed from scratch for the Spanish language and which is part of the first author’s doctoral thesis. These linguistic features were evaluated separately or in combination with word-embeddings with a Convolutional Neural Network (CNN) and with sentence-embeddings with Support Vector Machines (SVM). The rest of the paper is organised as follows. In Section 2, the datasets provided to the participants are described, whereas Section 3 describes the materials and methods used in our proposal. Then, Section 4 shows the results from each model and task along with an analysis of the results. Finally, Section 5 summarises the results according to the research questions and describes further improvement actions. 2. Corpus description The datasets involved in these tasks were provided by the organisers of the TASS workshop [2]. The corpus was compiled from Twitter in April 2019 and it contains topics from different domains, such as politics, entertainment, catastrophes, and global events, among others. The corpus contains approximately 16k tweets for Task 1, and 8.4k tweets for Task 2 [3]. For each task, the corpus was released as three datasets, namely, training, development, and testing. Table 1 shows the distribution of the datasets for each task. Note that the size of the training and development datasets of Subtask 1.2 is 0 because participants were expected to use the training and development datasets of Subtask 1.1. It is worth noting that hashtags and mentions were replaced by the tokens HASHTAG and @USER in order to prevent that automatic classifiers could overfit their results based on wrong assumptions. Another significant fact was that the label Neutral also referred to those tweets where no sentiment was assigned during the manual classification of the corpus. This is a 188 Table 1 Corpus size distribution for each dataset of the TASS’2020 corpus Model Training Development Testing Total Subtask 1.1 (es) 1127 582 1707 3416 Subtask 1.1 (pe) 967 499 1465 2931 Subtask 1.1 (cr) 778 391 1167 2336 Subtask 1.1 (uy) 944 487 1429 2860 Subtask 1.1 (mx) 991 511 1501 3003 Subtask 1.2 0 0 1501 1501 Subtask 2 5886 857 1666 8409 novelty with respect to previous editions of the TASS Workshop, in which tweets had been classified as neutral and none separately. 3. Materials and methods In this section, we describe (1) the preprocessing techniques applied to the corpus according to our proposal (see Section 3.1), (2) the linguistic features extracted (see Section 3.2), and (3) the machine-learning models used for training the sentiment classifiers (see Section 3.3). 3.1. Preprocessing stage Our preprocessing stage involved (1) encoding each letter to its lowercase form, (2) fixing misspelling and typos by using the Aspell library1 , (3) contracting white-space characters, such as spaces, tabs or new lines, and (4) removing expressive lengthening (the intentional elongation of letters in a word to emphasise it). The normalised version of each tweet is used as input for extracting those linguistic features that are dictionary based. However, we also kept the original version of the tweet to extract certain features regarding the usage of uppercase letters (which may be indicative of shouting or emphasis) or to identify the number of misspellings among other features. 3.2. Linguistic feature extraction For the extraction of linguistic features we used UMUTextStats [4], a self-developed linguistic tool for text analysis that is based on Linguistic Inquiry and Word Count (LIWC) [5]. LIWC is an analysis tool that counts words which belong to pre-established categories based on part-of- speech categories as well as other categories (family, sex, and death, to name but a few). One of LIWC’s greatest strengths is that has been validated under different domains. For example, it has been used in opinion mining [6], in the analysis of suicide notes [7], in cyber-bullying detection [8], or satire identification [9]. Although LIWC was originally conceived for the English language, it has a translated version available for Spanish. However, in contrast with 1 http://aspell.net/ 189 LIWC, our proposal handles specific phenomena of the Spanish language such as grammatical gender, as well as drawing a fine-grained distinction among PoS categories, such as adjectives, adverbs, verbs, and suffixes, among others. A further strength of UMUTextStats over LIWC is that UMUTextStats allows for complex regular expressions that are helpful in order to capture complex features such as discourse markers, by means of which different arguments connected in a text can be captured. The current version of UMUTextStats captures a total of 311 different linguistic features organised in the following categories: • Grammatical features (GRA). The features within this category are organised accord- ing to the Part-of-Speech (PoS) taxonomy. It includes verbs, adjectives, adverbs, determin- ers, pronouns, and conjunctions, to name but a few. Moreover, this category is organised into a more fine-grained distinction than LIWC. This decision was made because Spanish, as a highly inflected language, makes use of gender and number matching rules which may indicate when users are reporting facts, desires or hypothetical events. • Morphological features (MOR). The features within this category extract information from components of the words including prefixes and suffixes (distinguishing between different types, namely denominal morphemes, deverbal morphemes, deadjectival mor- phemes, and evaluative suffixation by means of diminutives, augmentatives, and pejora- tives). This category also contains features to match grammatical number (singular or plural). • Spelling and stylistic mistakes (ERR). In this category, we distinguish between stylis- tic patterns, which denote the usage of colloquial language in order to detect atypical patterns, and linguistic errors, which can either denote the low cultural level of the writers, that is to say, errors in competence, or that they have failed to properly revise their writings before publishing them, that is, errors in performance. • Figurative language (FIG). The features within this category are related to the usage of idioms, understatements, hyperboles or any other rhetorical device whose intention is to deviate the meaning of an utterance from its literal meaning [10]. • Linguistic processes (LPR). This category contains stylometric features such as num- ber of words, syllables, and sentences. It also distinguishes between different types of sentences, such as interrogatory, exclamatory or literal quotation. • Symbols (SYM). The features within this category aims to capture sentence dividers, such as spaces, colons, semicolons among other general-purpose symbols. • Entity and topic extraction (ENT). These contain a list of general topics, including animals, food, jobs, clothes or body-parts, in order to determine the general topics of a text. In this category, we include the usage of inclusive language, analytic thinking, achievements and failures, and risk perception. • Sentiments (SEN). The features within this category aims to determine general words and expressions related to positive and negative feelings. 190 3.3. Models based on word-embeddings The state of the art regarding SA makes use of word-embeddings and deep-learning methods. Word-embeddings, compared with traditional methods such as those models based on n-grams and one-hot-vectors, have two significant advantages. On the one hand, they represent words as dense vectors rather than sparse vectors. The main idea beyond this approach is that dense vectors allow clustering words with similar meanings. On the other, word-embeddings can be initialised by applying unsupervised techniques from general purpose trained sources, such as social networks, free encyclopedias or news sites, instead of being initialised with random values. Moreover, the idea of word-embeddings could be extended from words to whole texts, in order to represent sentences of paragraphs as dense vectors [11]. In this sense, sentence-embeddings are calculated by averaging the embeddings of the words that compose it. Our participation in the TASS-2020 workshop involved three runs. The first run (LF + WE) consisted in linguistic features trained with a Multilayer Perceptron in combination with word- embeddings trained with a Convolutional Neural Network (CNN), the second run (LF) entailed the linguistic features trained with Support Vector Machines, and the third run (LF + SE) involved combining the linguistic features with sentence embeddings. For the first run we used the functional API of Keras [12] to create a classifier that combines the inputs from a CNN [13], composed by a Embedding Layer with Spanish pre-trained word- embeddings from fastText [14] and a multilayer perceptron for training the linguistic features. The outputs for both deep-learning models where concatenated and combined with two more deep-learning layers and the the output layer for the final prediction. The architecture diagram of the first run is shown in Figure 1. We proposed the usage of CNN in order to exploit the spatial dimension of word-embeddings because these kinds of networks are able to find common patterns of words regardless of their position in the text, which can help solve some NLP problems such as polysemy. In this run, the training dataset was used for training and the development dataset for evaluating our proposal. For the second and the third runs, we used the Weka platform [15] to evaluate the reliability of using linguistic features separately (LF) and in combination with sentence-embeddings (LF + SE). Both runs were trained with Sequential Minimal Optimisation (SMO), a machine-learning classifier which is based on Support Vector Machines (SVMs). Specifically, we set this SVM to use a polynomial kernel in order to learn from non-lineal models. These runs were trained with the combination of the training and development datasets. 4. Results Our team participated in all the tasks proposed in the TASS-2020 workshop. All runs were evaluated using the macro-averaged versions of F-measure (F1), Precision (P), and Recall (R). First, Table 2 contains the results of our runs for Subtask 1.1 regarding monolingual classification. For the first run, with the combination of CNN and linguistic features, the European Spanish dataset (es) achieved the best result with a macro F1-measure of 0.503311. As precision was higher than recall, we can assume that our proposal fails regarding class detection, but when this detection is successful, the results are quite reliable. However, the rest of the Spanish varieties achieved significantly worse results between a macro F1-measure of 0.322492 for the 191 Figure 1: Model architecture of the first run, consisting in the combination of a Convolutional Neural Work for word-embeddings and a Multilayer perceptron for the linguistic features Costa Rica dataset and 0.391288 for the Uruguay dataset. We can also observe that precision and recall were more similar between them. However, for runs 2 and 3, where the training was performed by combining the training and development datasets, the results varied and both runs achieved their best result with the Uruguay dataset (0.49832 for run 2 and 0.520168 for run 3). However, with the European Spanish dataset the results are lower than the ones achieved for the first run, owing to the lack of the development dataset during the training stage. When our results are compared with the rest of the participants, it will be observed that they achieved more stable results and their macro-F1 measure is much more similar across the different datasets. For example, the best result, achieved by daniel.palomino.paucar, obtained a macro-F1 measure of 0.64667, which outperforms the results from our proposal. Next, the results achieved by our proposal for Subtask 1.2 regarding multilingual classification are shown in Table 3. In this subtask, we merged the datasets provided for Subtask 1.1 in order to create a multilingual classifier. It will be observed in the results that the runs for this subtask were similar to the ones achieved in Subtask 1.1, achieving our best macro F1-measure with 0.357876 for run 1, but obtaining similar precision and recall rates. The best result achieved in this subtask among all the participants was 0.497966. However, as the number of participants for this subtask was low, this result must be taken with caution. It is worth noting that the rules 192 Table 2 Results of the UMUTeam in Task 1.1 Monolingual classification Model Dataset F1 P R CNN (LF + WE) es 0.503311 0.561294 0.456186 CNN (LF + WE) pe 0.379346 0.399611 0.361037 CNN (LF + WE) cr 0.322492 0.321570 0.323420 CNN (LF + WE) uy 0.391288 0.393571 0.389031 CNN (LF + WE) mx 0.373712 0.377036 0.370446 SMO (LF) es 0.373223 0.373757 0.372691 SMO (LF) pe 0.379505 0.393181 0.366748 SMO (LF) cr 0.333147 0.333461 0.332834 SMO (LF) uy 0.498352 0.498630 0.498075 SMO (LF) mx 0.373502 0.373762 0.373241 SMO (LF + SE) es 0.378384 0.378490 0.378278 SMO (LF + SE) pe 0.389992 0.396927 0.383295 SMO (LF + SE) cr 0.349812 0.349505 0.350119 SMO (LF + SE) uy 0.520168 0.518072 0.522281 SMO (LF + SE) mx 0.396592 0.397157 0.396030 Table 3 Results of the UMUTeam in Task 1.2: Multilingual classification Model F1 P R CNN (LF + WE) 0.336824 0.337451 0.336198 SMO (LF) 0.357876 0.358662 0.357093 SMO (LF + SE) 0.334466 0.335797 0.333146 of the tasks allowed participants to use external corpora or linguistic resources. However, our participation was limited to the datasets provided. In this sense, we could not provide a fair comparison with the rest of the participants until we analyse their approaches in detail. Compared with purely statistical models such as the ones based on word-embeddings, linguis- tic features provide interpretability. It is possible, therefore, to obtain the most discriminatory linguistic features by calculating Information Gain (IG), which is a metric used by ensemble methods such as decision trees in order to determine when a new branch must be created. Figure 2 shows the 20 best features with major information for the combination of the training datasets of the different datasets provided for Subtask 1.1. We can observe that Sentiments (SEN) is the linguistic category which provides the largest number of linguistic features including positive, negative, anger, offensive and sad. Out of these features, positive is, by far, the most discriminatory feature. Grammatical features (GRA) is another linguistic feature that provides several features, such as adverbs-negation and adjectives-qualifying. It is worth noting that these linguistic features are hard to obtain by applying models based on word-embeddings like those linguistic features regarding stylistic patterns (misspellings, linguistic errors, and swear), as well as other features used to add emphasis such as the number of exclamatory sentences. 193 Figure 2: Information gain of the 20 best linguistic features for the Task 1.2 Table 4 Results of the UMUTeam in Task 2: Emotion detection Model F1 P R CNN (LF + WE) 0.242799 0.243873 0.241734 SMO (LF) 0.372774 0.490520 0.300614 SMO (LF + SE) 0.378694 0.419558 0.345084 Finally, the results from Task 2, emotion detection, are shown in Table 4. As can be observed, the usage of linguistic features and sentence embeddings (run 3) achieved the best result over the rest of our runs. It worth noting that run 3 only achieves slightly better results than the run 2 (LF), but with lower precision and higher recall. Our best macro F1-measure is 0.378694, which is reasonably successful, in view of the complexity of the task although the best score was achieved by jogonba2elirf with a macro F1-measure of 0.446582. 5. Conclusions and further work In this paper, the participation of the UMUTeam in the TASS’2020 Workshop on Sentiment Classification applying linguistic features has been described. For Subtask 1.1, our proposal achieved decent results only for one dataset: European Spanish with CNN and linguistic features, and for the Uruguay dataset with SMO and linguistic features with average word-embeddings. For Subtask 1.2, which consisted in the combination of the datasets from different Spanish varieties, we achieved more stable results irrespective of the model applied. However, the low 194 number of participants hindered the achievement of better insights. Finally, for Task 2, which consisted in emotion detection, we achieved reasonably good results with a macro F1 of 0.372774 over six different classes. As regards RQ1 and RQ2, we have observed that our proposal achieves good results when comparing the usage of LF separately with the rest of our runs, but the results are far from being the best results. In this sense, we need to perform a more detailed analysis of the rest of the participants in order to identify the weakness of our proposal. Regarding RQ3, we have observed that our proposal does not provide stable results for Subtask 1.1 when the different dialects are considered separately, but the results were more stable when training and testing contained tweets from the different dialects as the same time. This fact suggests that some of the linguistic features between each dialect and cultural background are complementary, but not enough to perform a reliable sentiment classification. We are well satisfied with our participation for the first time in a TASS workshop and our competition in challenging NLP tasks. We are, however, aware of the limitations of our proposal and the long way to go. To our mind, the main drawback is that we focused only on European Spanish during the design of the linguistic features. We will, therefore, focus on improvements for other varieties enabling the adaptation of the system. Regarding the technological aspect, we will include the hyper-parameter tuning in our pipeline, in order to choose the optimal hyper-parameters for a learning algorithm. We will also try to incorporate other pre-trained and contextualised word-embeddings such as ELMo [16] and BERT [17]. Acknowledgments This work has been supported by the Spanish National Research Agency (AEI) and the European Regional Development Fund (FEDER/ERDF) through projects KBS4FIA (TIN2016-76323-R) and LaTe4PSP (PID2019-107652RB-I00). In addition, José Antonio García-Díaz has been supported by Banco Santander and University of Murcia through the Doctorado industrial programme. References [1] P. Ekman, Lie catching and microexpressions, The philosophy of deception 1 (2009) 5. [2] M. García-Vega, M. C. Díaz-Galiano, M. A. García-Cumbreras, A. Montejo Ráez, S. M. Jiménez Zafra, E. Martínez-Cámara, C. A. Murillo, E. Casasola Murillo, L. Chiruzzo, D. Moctezuma, Sobrevilla, Overview of tass 2020: Introduction emotion detection, in: Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2020), volume 2664 of CEUR Workshop Proceedings, CEUR-WS, Málaga, Spain, 2020. [3] F. M. Plaza del Arco, C. Strapparava, L. A. Urena Lopez, M. Martin, EmoEvent: A multilin- gual emotion corpus based on different events, in: Proceedings of The 12th Language Re- sources and Evaluation Conference, European Language Resources Association, Marseille, France, 2020, pp. 1492–1498. URL: https://www.aclweb.org/anthology/2020.lrec-1.186. [4] J. A. García-Díaz, M. Cánovas-García, R. Valencia-García, Ontology-driven aspect-based sentiment analysis classification: An infodemiological case study regarding infectious 195 diseases in latin america, Future Generation Computer Systems 112 (2020) 614–657. doi:https://doi.org/10.1016/j.future.2020.06.019. [5] Y. R. Tausczik, J. W. Pennebaker, The psychological meaning of words: Liwc and com- puterized text analysis methods, Journal of language and social psychology 29 (2010) 24–54. [6] M. del Pilar Salas-Zárate, E. López-López, R. Valencia-García, N. Aussenac-Gilles, Á. Almela, G. Alor-Hernández, A study on LIWC categories for opinion mining in spanish reviews, J. Inf. Sci. 40 (2014) 749–760. URL: https://doi.org/10.1177/0165551514547842. doi:10.1177/ 0165551514547842. [7] B. O’dea, M. E. Larsen, P. J. Batterham, A. L. Calear, H. Christensen, A linguistic analysis of suicide-related twitter posts., Crisis: The Journal of Crisis Intervention and Suicide Prevention 38 (2017) 319. [8] V. Singh, S. Ghosh, C. Jose, Toward multimodal cyberbullying detection, in: Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems, volume Part F127655, 2017, pp. 2090–2099. doi:10.1145/3027063.3053169. [9] M. del Pilar Salas-Zárate, M. A. Paredes-Valverde, M. Á. Rodríguez-García, R. Valencia- García, G. Alor-Hernández, Automatic detection of satire in twitter: A psycholinguistic- based approach, Knowl. Based Syst. 128 (2017) 20–33. URL: https://doi.org/10.1016/j.knosys. 2017.04.009. doi:10.1016/j.knosys.2017.04.009. [10] M. del Pilar Salas-Zárate, G. Alor-Hernández, J. L. Sánchez-Cervantes, M. A. Paredes- Valverde, J. L. García-Alcaraz, R. Valencia-García, Review of english literature on figurative language applied to social networks, Knowl. Inf. Syst. 62 (2020) 2105–2137. URL: https: //doi.org/10.1007/s10115-019-01425-3. doi:10.1007/s10115-019-01425-3. [11] D. Cer, Y. Yang, S.-y. Kong, N. Hua, N. Limtiaco, R. S. John, N. Constant, M. Guajardo- Cespedes, S. Yuan, C. Tar, et al., Universal sentence encoder, arXiv preprint arXiv:1803.11175 (2018). [12] F. Chollet, et al., Keras (2015), 2017. [13] Y. Kim, Convolutional neural networks for sentence classification, CoRR abs/1408.5882 (2014). URL: http://arxiv.org/abs/1408.5882. arXiv:1408.5882. [14] E. Grave, P. Bojanowski, P. Gupta, A. Joulin, T. Mikolov, Learning word vectors for 157 languages, CoRR abs/1802.06893 (2018). URL: http://arxiv.org/abs/1802.06893. arXiv:1802.06893. [15] M. Hall, E. Frank, G. Holmes, B. Pfahringer, P. Reutemann, I. H. Witten, The weka data mining software: an update, ACM SIGKDD explorations newsletter 11 (2009) 10–18. [16] M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, L. Zettlemoyer, Deep contextualized word representations, CoRR abs/1802.05365 (2018). URL: http://arxiv.org/ abs/1802.05365. arXiv:1802.05365. [17] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, 2018. arXiv:1810.04805. 196