Introduction

This paper describes the participation of the UMUTeam in the TASS'2020 Workshop on Sentiment Analysis (SA). SA is a trending task of Natural Language Processing (NLP) which consists in the extraction and classification of subjective opinions in texts written in natural language.

Two tasks were proposed to the participants. The first one involved extracting the subjectivity polarity from tweets written in different varieties of Spanish, including European Spanish and different varieties from Latin America. This task was divided into two sub-tasks: (1) monolingual classification, in which several training and testing splits from different Spanish varieties were considered separately: Spain (es), Costa Rica (cr), Uruguay (uy), and Mexico (mx); and (2) multi-variant classification, in which a testing dataset consisting of mixed tweets from the above Spanish varieties were mixed. In both sub-tasks the participants were required to determine if tweets express positive, negative or neutral sentiments. The second task consisted in a multiclass classification of emotions based on six of the basic emotions proposed by Ekman (2009) [1]: anger, disgust, fear, joy, sadness and surprise along with others in order to classify those tweets which do not agree with the aforementioned labels. All datasets, regardless of the task, were unbalanced: Task 1 contained more tweets labelled as neutral and Task 2 contained more tweets labelled as others.

We participated both in Task 1 and Task 2. In a nutshell, our proposal consisted in testing the reliability of linguistic features alone or in combination with well-known machine-learning and deep-learning models. Linguistic features, in comparison with statistical approaches, ease the interpretability of the results because they are conceptually higher-level features than words. With our participation, we attempted to answer the following research questions:

• RQ1. Is the usage of linguistic features enough to compete with state-of-the-art approaches based on statistical methods?

• RQ2. Can linguistic features be combined with statistical methods in order to improve the accuracy of the results while keeping interpretability?

• RQ3. Is the reliability of linguistic features affected by different cultural background in the same language?

To answer these research questions, we extracted linguistic features from the datasets by using a self-developed tool designed from scratch for the Spanish language and which is part of the first author's doctoral thesis. These linguistic features were evaluated separately or in combination with word-embeddings with a Convolutional Neural Network (CNN) and with sentence-embeddings with Support Vector Machines (SVM).

The rest of the paper is organised as follows. In Section 2, the datasets provided to the participants are described, whereas Section 3 describes the materials and methods used in our proposal. Then, Section 4 shows the results from each model and task along with an analysis of the results. Finally, Section 5 summarises the results according to the research questions and describes further improvement actions.

Corpus description

The datasets involved in these tasks were provided by the organisers of the TASS workshop [2]. The corpus was compiled from Twitter in April 2019 and it contains topics from different domains, such as politics, entertainment, catastrophes, and global events, among others. The corpus contains approximately 16k tweets for Task 1, and 8.4k tweets for Task 2 [3]. For each task, the corpus was released as three datasets, namely, training, development, and testing. Table 1 shows the distribution of the datasets for each task. Note that the size of the training and development datasets of Subtask 1.2 is 0 because participants were expected to use the training and development datasets of Subtask 1.1.

It is worth noting that hashtags and mentions were replaced by the tokens HASHTAG and @USER in order to prevent that automatic classifiers could overfit their results based on wrong assumptions. Another significant fact was that the label Neutral also referred to those tweets where no sentiment was assigned during the manual classification of the corpus. This is a novelty with respect to previous editions of the TASS Workshop, in which tweets had been classified as neutral and none separately.

Materials and methods

In this section, we describe (1) the preprocessing techniques applied to the corpus according to our proposal (see Section 3.1), (2) the linguistic features extracted (see Section 3.2), and (3) the machine-learning models used for training the sentiment classifiers (see Section 3.3).

Preprocessing stage

Our preprocessing stage involved (1) encoding each letter to its lowercase form, (2) fixing misspelling and typos by using the Aspell library1 , (3) contracting white-space characters, such as spaces, tabs or new lines, and (4) removing expressive lengthening (the intentional elongation of letters in a word to emphasise it). The normalised version of each tweet is used as input for extracting those linguistic features that are dictionary based. However, we also kept the original version of the tweet to extract certain features regarding the usage of uppercase letters (which may be indicative of shouting or emphasis) or to identify the number of misspellings among other features.

Linguistic feature extraction

For the extraction of linguistic features we used UMUTextStats [4], a self-developed linguistic tool for text analysis that is based on Linguistic Inquiry and Word Count (LIWC) [5]. LIWC is an analysis tool that counts words which belong to pre-established categories based on part-ofspeech categories as well as other categories (family, sex, and death, to name but a few). One of LIWC's greatest strengths is that has been validated under different domains. For example, it has been used in opinion mining [6], in the analysis of suicide notes [7], in cyber-bullying detection [8], or satire identification [9]. Although LIWC was originally conceived for the English language, it has a translated version available for Spanish. However, in contrast with LIWC, our proposal handles specific phenomena of the Spanish language such as grammatical gender, as well as drawing a fine-grained distinction among PoS categories, such as adjectives, adverbs, verbs, and suffixes, among others. A further strength of UMUTextStats over LIWC is that UMUTextStats allows for complex regular expressions that are helpful in order to capture complex features such as discourse markers, by means of which different arguments connected in a text can be captured. The current version of UMUTextStats captures a total of 311 different linguistic features organised in the following categories:

• Grammatical features (GRA). The features within this category are organised according to the Part-of-Speech (PoS) taxonomy. It includes verbs, adjectives, adverbs, determiners, pronouns, and conjunctions, to name but a few. Moreover, this category is organised into a more fine-grained distinction than LIWC. This decision was made because Spanish, as a highly inflected language, makes use of gender and number matching rules which may indicate when users are reporting facts, desires or hypothetical events.

• Morphological features (MOR). The features within this category extract information from components of the words including prefixes and suffixes (distinguishing between different types, namely denominal morphemes, deverbal morphemes, deadjectival morphemes, and evaluative suffixation by means of diminutives, augmentatives, and pejoratives). This category also contains features to match grammatical number (singular or plural).

• Spelling and stylistic mistakes (ERR). In this category, we distinguish between stylistic patterns, which denote the usage of colloquial language in order to detect atypical patterns, and linguistic errors, which can either denote the low cultural level of the writers, that is to say, errors in competence, or that they have failed to properly revise their writings before publishing them, that is, errors in performance.

• Figurative language (FIG). The features within this category are related to the usage of idioms, understatements, hyperboles or any other rhetorical device whose intention is to deviate the meaning of an utterance from its literal meaning [10].

• Linguistic processes (LPR). This category contains stylometric features such as number of words, syllables, and sentences. It also distinguishes between different types of sentences, such as interrogatory, exclamatory or literal quotation.

• Symbols (SYM). The features within this category aims to capture sentence dividers, such as spaces, colons, semicolons among other general-purpose symbols.

• Entity and topic extraction (ENT). These contain a list of general topics, including animals, food, jobs, clothes or body-parts, in order to determine the general topics of a text. In this category, we include the usage of inclusive language, analytic thinking, achievements and failures, and risk perception.

• Sentiments (SEN). The features within this category aims to determine general words and expressions related to positive and negative feelings.

Models based on word-embeddings

The state of the art regarding SA makes use of word-embeddings and deep-learning methods. Word-embeddings, compared with traditional methods such as those models based on n-grams and one-hot-vectors, have two significant advantages. On the one hand, they represent words as dense vectors rather than sparse vectors. The main idea beyond this approach is that dense vectors allow clustering words with similar meanings. On the other, word-embeddings can be initialised by applying unsupervised techniques from general purpose trained sources, such as social networks, free encyclopedias or news sites, instead of being initialised with random values. Moreover, the idea of word-embeddings could be extended from words to whole texts, in order to represent sentences of paragraphs as dense vectors [11]. In this sense, sentence-embeddings are calculated by averaging the embeddings of the words that compose it.

Our participation in the TASS-2020 workshop involved three runs. The first run (LF + WE) consisted in linguistic features trained with a Multilayer Perceptron in combination with wordembeddings trained with a Convolutional Neural Network (CNN), the second run (LF) entailed the linguistic features trained with Support Vector Machines, and the third run (LF + SE) involved combining the linguistic features with sentence embeddings.

For the first run we used the functional API of Keras [12] to create a classifier that combines the inputs from a CNN [13], composed by a Embedding Layer with Spanish pre-trained wordembeddings from fastText [14] and a multilayer perceptron for training the linguistic features. The outputs for both deep-learning models where concatenated and combined with two more deep-learning layers and the the output layer for the final prediction. The architecture diagram of the first run is shown in Figure 1. We proposed the usage of CNN in order to exploit the spatial dimension of word-embeddings because these kinds of networks are able to find common patterns of words regardless of their position in the text, which can help solve some NLP problems such as polysemy. In this run, the training dataset was used for training and the development dataset for evaluating our proposal.

For the second and the third runs, we used the Weka platform [15] to evaluate the reliability of using linguistic features separately (LF) and in combination with sentence-embeddings (LF + SE). Both runs were trained with Sequential Minimal Optimisation (SMO), a machine-learning classifier which is based on Support Vector Machines (SVMs). Specifically, we set this SVM to use a polynomial kernel in order to learn from non-lineal models. These runs were trained with the combination of the training and development datasets.

Results

Our team participated in all the tasks proposed in the TASS-2020 workshop. All runs were evaluated using the macro-averaged versions of F-measure (F1), Precision (P), and Recall (R). First, Table 2 contains the results of our runs for Subtask 1.1 regarding monolingual classification. For the first run, with the combination of CNN and linguistic features, the European Spanish dataset (es) achieved the best result with a macro F1-measure of 0.503311. As precision was higher than recall, we can assume that our proposal fails regarding class detection, but when this detection is successful, the results are quite reliable. However, the rest of the Spanish varieties achieved significantly worse results between a macro F1-measure of 0.322492 for the Costa Rica dataset and 0.391288 for the Uruguay dataset. We can also observe that precision and recall were more similar between them. However, for runs 2 and 3, where the training was performed by combining the training and development datasets, the results varied and both runs achieved their best result with the Uruguay dataset (0.49832 for run 2 and 0.520168 for run 3). However, with the European Spanish dataset the results are lower than the ones achieved for the first run, owing to the lack of the development dataset during the training stage.

When our results are compared with the rest of the participants, it will be observed that they achieved more stable results and their macro-F1 measure is much more similar across the different datasets. For example, the best result, achieved by daniel.palomino.paucar, obtained a macro-F1 measure of 0.64667, which outperforms the results from our proposal.

Next, the results achieved by our proposal for Subtask 1.2 regarding multilingual classification are shown in Table 3. In this subtask, we merged the datasets provided for Subtask 1.1 in order to create a multilingual classifier. It will be observed in the results that the runs for this subtask were similar to the ones achieved in Subtask 1.1, achieving our best macro F1-measure with 0.357876 for run 1, but obtaining similar precision and recall rates. The best result achieved in this subtask among all the participants was 0.497966. However, as the number of participants for this subtask was low, this result must be taken with caution. It is worth noting that the rules of the tasks allowed participants to use external corpora or linguistic resources. However, our participation was limited to the datasets provided. In this sense, we could not provide a fair comparison with the rest of the participants until we analyse their approaches in detail.

Compared with purely statistical models such as the ones based on word-embeddings, linguistic features provide interpretability. It is possible, therefore, to obtain the most discriminatory linguistic features by calculating Information Gain (IG), which is a metric used by ensemble methods such as decision trees in order to determine when a new branch must be created. Figure 2 shows the 20 best features with major information for the combination of the training datasets of the different datasets provided for Subtask 1.1. We can observe that Sentiments (SEN) is the linguistic category which provides the largest number of linguistic features including positive, negative, anger, offensive and sad. Out of these features, positive is, by far, the most discriminatory feature. Grammatical features (GRA) is another linguistic feature that provides several features, such as adverbs-negation and adjectives-qualifying. It is worth noting that these linguistic features are hard to obtain by applying models based on word-embeddings like those linguistic features regarding stylistic patterns (misspellings, linguistic errors, and swear), as well as other features used to add emphasis such as the number of exclamatory sentences. Finally, the results from Task 2, emotion detection, are shown in Table 4. As can be observed, the usage of linguistic features and sentence embeddings (run 3) achieved the best result over the rest of our runs. It worth noting that run 3 only achieves slightly better results than the run 2 (LF), but with lower precision and higher recall. Our best macro F1-measure is 0.378694, which is reasonably successful, in view of the complexity of the task although the best score was achieved by jogonba2elirf with a macro F1-measure of 0.446582.

Conclusions and further work

In this paper, the participation of the UMUTeam in the TASS'2020 Workshop on Sentiment Classification applying linguistic features has been described. For Subtask 1.1, our proposal achieved decent results only for one dataset: European Spanish with CNN and linguistic features, and for the Uruguay dataset with SMO and linguistic features with average word-embeddings. For Subtask 1.2, which consisted in the combination of the datasets from different Spanish varieties, we achieved more stable results irrespective of the model applied. However, the low number of participants hindered the achievement of better insights. Finally, for Task 2, which consisted in emotion detection, we achieved reasonably good results with a macro F1 of 0.372774 over six different classes.

As regards RQ1 and RQ2, we have observed that our proposal achieves good results when comparing the usage of LF separately with the rest of our runs, but the results are far from being the best results. In this sense, we need to perform a more detailed analysis of the rest of the participants in order to identify the weakness of our proposal. Regarding RQ3, we have observed that our proposal does not provide stable results for Subtask 1.1 when the different dialects are considered separately, but the results were more stable when training and testing contained tweets from the different dialects as the same time. This fact suggests that some of the linguistic features between each dialect and cultural background are complementary, but not enough to perform a reliable sentiment classification.

We are well satisfied with our participation for the first time in a TASS workshop and our competition in challenging NLP tasks. We are, however, aware of the limitations of our proposal and the long way to go. To our mind, the main drawback is that we focused only on European Spanish during the design of the linguistic features. We will, therefore, focus on improvements for other varieties enabling the adaptation of the system. Regarding the technological aspect, we will include the hyper-parameter tuning in our pipeline, in order to choose the optimal hyper-parameters for a learning algorithm. We will also try to incorporate other pre-trained and contextualised word-embeddings such as ELMo [16] and BERT [17].