This paper describes the participation of the UMUTeam at the TASS'2020 Workshop on Sentiment Analysis, in which two tracks were proposed. The first track consists in the classification of tweets according to general sentiments of tweets written in several Spanish varieties, whereas the second task consists in a fine-grained distinction between the six basic emotions described by Ekman (2009). Our proposal is based on the usage of linguistic features alone or in combination with word-embeddings. Specifically, we test Convolutional Neural Networks and Support Vector Machines with sentence embeddings. Although our proposal did not achieve the best results, we obtained the best precision rate regarding emotion detection (Task 2) and competitive results with respect to the general sentiment classification in which tweets written in diferent varieties of Spanish were mixed. We consider that our proposal, despite its limitations, provides substantial benefits such as the interpretability of the results.
[
We participated both in Task 1 and Task 2. In a nutshell, our proposal consisted in testing the reliability of linguistic features alone or in combination with well-known machine-learning and deep-learning models. Linguistic features, in comparison with statistical approaches, ease the interpretability of the results because they are conceptually higher-level features than words. With our participation, we attempted to answer the following research questions: • RQ1. Is the usage of linguistic features enough to compete with state-of-the-art approaches based on statistical methods? • RQ2. Can linguistic features be combined with statistical methods in order to improve the accuracy of the results while keeping interpretability? • RQ3. Is the reliability of linguistic features afected by diferent cultural background in the same language?
To answer these research questions, we extracted linguistic features from the datasets by using a self-developed tool designed from scratch for the Spanish language and which is part of the first author’s doctoral thesis. These linguistic features were evaluated separately or in combination with word-embeddings with a Convolutional Neural Network (CNN) and with sentence-embeddings with Support Vector Machines (SVM).
The rest of the paper is organised as follows. In Section 2, the datasets provided to the participants are described, whereas Section 3 describes the materials and methods used in our proposal. Then, Section 4 shows the results from each model and task along with an analysis of the results. Finally, Section 5 summarises the results according to the research questions and describes further improvement actions.
The datasets involved in these tasks were provided by the organisers of the TASS workshop
[
It is worth noting that hashtags and mentions were replaced by the tokens HASHTAG and @USER in order to prevent that automatic classifiers could overfit their results based on wrong assumptions. Another significant fact was that the label Neutral also referred to those tweets where no sentiment was assigned during the manual classification of the corpus. This is a Model Subtask 1.1 (es) Subtask 1.1 (pe) Subtask 1.1 (cr) Subtask 1.1 (uy) Subtask 1.1 (mx) Subtask 1.2 Subtask 2 novelty with respect to previous editions of the TASS Workshop, in which tweets had been classified as neutral and none separately.
In this section, we describe (1) the preprocessing techniques applied to the corpus according to our proposal (see Section 3.1), (2) the linguistic features extracted (see Section 3.2), and (3) the machine-learning models used for training the sentiment classifiers (see Section 3.3).
Our preprocessing stage involved (1) encoding each letter to its lowercase form, (2) fixing misspelling and typos by using the Aspell library1, (3) contracting white-space characters, such as spaces, tabs or new lines, and (4) removing expressive lengthening (the intentional elongation of letters in a word to emphasise it). The normalised version of each tweet is used as input for extracting those linguistic features that are dictionary based. However, we also kept the original version of the tweet to extract certain features regarding the usage of uppercase letters (which may be indicative of shouting or emphasis) or to identify the number of misspellings among other features.
For the extraction of linguistic features we used UMUTextStats [
The state of the art regarding SA makes use of word-embeddings and deep-learning methods.
Word-embeddings, compared with traditional methods such as those models based on n-grams
and one-hot-vectors, have two significant advantages. On the one hand, they represent words
as dense vectors rather than sparse vectors. The main idea beyond this approach is that dense
vectors allow clustering words with similar meanings. On the other, word-embeddings can be
initialised by applying unsupervised techniques from general purpose trained sources, such as
social networks, free encyclopedias or news sites, instead of being initialised with random values.
Moreover, the idea of word-embeddings could be extended from words to whole texts, in order
to represent sentences of paragraphs as dense vectors [
Our participation in the TASS-2020 workshop involved three runs. The first run (LF + WE) consisted in linguistic features trained with a Multilayer Perceptron in combination with wordembeddings trained with a Convolutional Neural Network (CNN), the second run (LF) entailed the linguistic features trained with Support Vector Machines, and the third run (LF + SE) involved combining the linguistic features with sentence embeddings.
For the first run we used the functional API of Keras [
For the second and the third runs, we used the Weka platform [
Our team participated in all the tasks proposed in the TASS-2020 workshop. All runs were evaluated using the macro-averaged versions of F-measure (F1), Precision (P), and Recall (R). First, Table 2 contains the results of our runs for Subtask 1.1 regarding monolingual classification. For the first run, with the combination of CNN and linguistic features, the European Spanish dataset (es) achieved the best result with a macro F1-measure of 0.503311. As precision was higher than recall, we can assume that our proposal fails regarding class detection, but when this detection is successful, the results are quite reliable. However, the rest of the Spanish varieties achieved significantly worse results between a macro F1-measure of 0.322492 for the Costa Rica dataset and 0.391288 for the Uruguay dataset. We can also observe that precision and recall were more similar between them. However, for runs 2 and 3, where the training was performed by combining the training and development datasets, the results varied and both runs achieved their best result with the Uruguay dataset (0.49832 for run 2 and 0.520168 for run 3). However, with the European Spanish dataset the results are lower than the ones achieved for the first run, owing to the lack of the development dataset during the training stage.
When our results are compared with the rest of the participants, it will be observed that they achieved more stable results and their macro-F1 measure is much more similar across the diferent datasets. For example, the best result, achieved by daniel.palomino.paucar, obtained a macro-F1 measure of 0.64667, which outperforms the results from our proposal.
Next, the results achieved by our proposal for Subtask 1.2 regarding multilingual classification are shown in Table 3. In this subtask, we merged the datasets provided for Subtask 1.1 in order to create a multilingual classifier. It will be observed in the results that the runs for this subtask were similar to the ones achieved in Subtask 1.1, achieving our best macro F1-measure with 0.357876 for run 1, but obtaining similar precision and recall rates. The best result achieved in this subtask among all the participants was 0.497966. However, as the number of participants for this subtask was low, this result must be taken with caution. It is worth noting that the rules of the tasks allowed participants to use external corpora or linguistic resources. However, our participation was limited to the datasets provided. In this sense, we could not provide a fair comparison with the rest of the participants until we analyse their approaches in detail.
Compared with purely statistical models such as the ones based on word-embeddings, linguistic features provide interpretability. It is possible, therefore, to obtain the most discriminatory linguistic features by calculating Information Gain (IG), which is a metric used by ensemble methods such as decision trees in order to determine when a new branch must be created. Figure 2 shows the 20 best features with major information for the combination of the training datasets of the diferent datasets provided for Subtask 1.1. We can observe that Sentiments (SEN) is the linguistic category which provides the largest number of linguistic features including positive, negative, anger, ofensive and sad. Out of these features, positive is, by far, the most discriminatory feature. Grammatical features (GRA) is another linguistic feature that provides several features, such as adverbs-negation and adjectives-qualifying. It is worth noting that these linguistic features are hard to obtain by applying models based on word-embeddings like those linguistic features regarding stylistic patterns (misspellings, linguistic errors, and swear), as well as other features used to add emphasis such as the number of exclamatory sentences.
Finally, the results from Task 2, emotion detection, are shown in Table 4. As can be observed, the usage of linguistic features and sentence embeddings (run 3) achieved the best result over the rest of our runs. It worth noting that run 3 only achieves slightly better results than the run 2 (LF), but with lower precision and higher recall. Our best macro F1-measure is 0.378694, which is reasonably successful, in view of the complexity of the task although the best score was achieved by jogonba2elirf with a macro F1-measure of 0.446582.
In this paper, the participation of the UMUTeam in the TASS’2020 Workshop on Sentiment Classification applying linguistic features has been described. For Subtask 1.1, our proposal achieved decent results only for one dataset: European Spanish with CNN and linguistic features, and for the Uruguay dataset with SMO and linguistic features with average word-embeddings. For Subtask 1.2, which consisted in the combination of the datasets from diferent Spanish varieties, we achieved more stable results irrespective of the model applied. However, the low number of participants hindered the achievement of better insights. Finally, for Task 2, which consisted in emotion detection, we achieved reasonably good results with a macro F1 of 0.372774 over six diferent classes.
As regards RQ1 and RQ2, we have observed that our proposal achieves good results when comparing the usage of LF separately with the rest of our runs, but the results are far from being the best results. In this sense, we need to perform a more detailed analysis of the rest of the participants in order to identify the weakness of our proposal. Regarding RQ3, we have observed that our proposal does not provide stable results for Subtask 1.1 when the diferent dialects are considered separately, but the results were more stable when training and testing contained tweets from the diferent dialects as the same time. This fact suggests that some of the linguistic features between each dialect and cultural background are complementary, but not enough to perform a reliable sentiment classification.
We are well satisfied with our participation for the first time in a TASS workshop and our
competition in challenging NLP tasks. We are, however, aware of the limitations of our proposal
and the long way to go. To our mind, the main drawback is that we focused only on European
Spanish during the design of the linguistic features. We will, therefore, focus on improvements
for other varieties enabling the adaptation of the system. Regarding the technological aspect,
we will include the hyper-parameter tuning in our pipeline, in order to choose the optimal
hyper-parameters for a learning algorithm. We will also try to incorporate other pre-trained
and contextualised word-embeddings such as ELMo [
This work has been supported by the Spanish National Research Agency (AEI) and the European Regional Development Fund (FEDER/ERDF) through projects KBS4FIA (TIN2016-76323-R) and LaTe4PSP (PID2019-107652RB-I00). In addition, José Antonio García-Díaz has been supported by Banco Santander and University of Murcia through the Doctorado industrial programme.