This work presents the participation of the Intelligent Systems Group (GSI) at Universidad Politecnica de Madrid (UPM) in the Emotion Analysis competition EmoEvalEs, part of IberLEF 2021 Conference. The addressed challenge proposes an emotion classi cation task of Spanish tweets, categorizing each message into seven emotions. We propose the design and development of a ne-tuned neural language model (XLM-RoBERTa) to tackle this challenge. We have obtained excellent results with this approach, obtaining the rst place in the competition with a macro-averaged F1 score of 71.70%. Additionally, we also explore the application of several ensemble methods built over the neural language model.
Recent advances in machine learning research are rapidly pushing the sentiment
analysis eld forward. Works that use neural architectures to improve previous
models are established, and current state-of-the-art models are heavily based on
these techniques [
This paper presents our participation in IberLEF 2021 [
In the e ort of addressing this task, we use the ne-tuned XLM-Twitter
(XLM-T) language model [
For replication reasons, we provide the source code used to generate the models and their respective submissions. It is available online at https://github. com/gsi-upm/emoevales-iberlef2021.
The rest of the paper is organized as follows. Section 2 provides the background of the methods used in this work. Next, the proposed approach and architecture for emotion classi cation are described in Section 3 and evaluated in Section 4. Finally, Section 5 states our conclusions and proposes future lines of work. 2
Deep learning approaches are common in sentiment analysis and have proved
helpful in emotion analysis [
The use of word embeddings to elicit sentiment and emotion represents a large
eld. In this work, we use previous models to incorporate them into our ensemble
model. One of those is the SIMilarity-based sentiment projectiON (SIMON)
model, which computes the representation of a particular word in a document
by considering its projection to a set of domain words [
In recent years, an approach that has proved successful in most Natural
Language Processing (NLP) tasks is using large pretrained language models based
on a transformer architecture. Transformers [
In this domain, XLM-RoBERTa (XLM-R) [
We have ne-tuned the Twitter-speci c pretrained language model in the
downstream task of emotion classi cation following parameter-e cient transfer
learning techniques [
We have implemented this architecture and run the training process using the
modules and the Trainer API from the HuggingFace Transformer library [
The classi cation head consists of a dense layer at the output of the language model, followed by a dropout layer with the default dropout probability of the language model and a nal projection layer with the number of labels. For the Additional Features model, we have added new inputs to the rst dense layer.
We have followed the training process as described in Transformers documentation. For the sake of reproducibility, the source code is available at https://github.com/gsi-upm/emoevales-iberlef2021, and the ne-tuned model is available at the HuggingFace model hub.
The hyper-parameters used are a batch size of 16 per GPU, max length (tokenizer) of 200, and training for 5 epochs. The rest of the parameters are the default in the Trainer API. The trainer API also saves the checkpoint of the best epoch that usually occurs at epoch 3. We run the training process for 1 hour on two NVIDIA Titan X Pascal GPUs.
Before tokenizing, we have slightly preprocessed the tweets with the Twitter
preprocessing module of the GSITK library [
To improve the nal prediction scores, we have developed an ensemble model
that combines, at the prediction level, di erent models trained on varied data.
In this sense, previous works [
SIMON [
model aggregates the component word vectors from a document, obtaining a xed document representation. Previous works have found that this method is reliable across di erent domains. In this work, we aggregate the word vectors using the average aggregation operation.
the TF-IDF as a simple text representation method. Our model instance considers both uni and bigrams.
N-grams. Similarly, as before, we use this feature to enhance the variety of the ensemble model. Moreover, as before, we consider uni and bigrams.
Additional features. The challenge dataset contains additional information that can be easily used as features. Concretely, we used the event and o ensive data elds. The event category speci es the general event from which the message has been extracted. In contrast, the o ensive category speci es whether the message contains o ensive language, which may aid in the task at hand. Please note that we do not train a learning model with these features solely. However, instead, we add them to other feature sets.
MeaningCloud. Since sentiment and emotion are intimately related we have incorporated a new feature, the sentiment estimation that MeaningCloud (https://www.meaningcloud.com/) o ers. MeaningCloud o ers a professional sentiment analysis service that can be accessed via a web API (https://www. meaningcloud.com/products/sentiment-analysis). We extract the sentiment estimations for all messages using this service. This information is included as an additional feature.
Considering the described feature types, we train di erent learning models that train on the mentioned features. We select a simple algorithm for the base learners, logistic regression, since all predictions are combined in an ensemble fashion. For feature combination, we concatenate the feature vectors. When using learning models to train for the ensemble, we have selected logistic regression and random forests. 4
4.1
This section describes the performance of the di erent approaches we have followed during the ne-tuning of the pretrained model. Table 1 shows the accuracy and weighted F1 scores of the di erent approaches on the development set, where the model ne-tuned in the multiclass classi cation problem has achieved the best results. Table 2 shows the same information for the test set, where the best model is the multiclass estimator again.
XLM-T Multi-label classi cation XLM-T Binary one-vs-all classi cation
Additional Features
XLM-T Multi-label classi cation XLM-T Binary one-vs-all classi cation
Additional Features
These results show the superior performance of the multiclass classi er over the combination of various binary classi ers, although better combination strategies could improve the results of the latter. Moreover, including additional features without any preprocessing and at the same level as the output produced by the pretrained language model decreases the classi er performance.
Figure 1 depicts the confusion matrix produced by the XLM-T multi-label classi er on the test set. We observe the evident unbalancing of the dataset, where almost half of the records belong to the others class. Moreover, this class is usually confounded with the joy class. Additionally, this matrix shows the di culty of distinguishing between emotions that share similar features, such as anger and disgust. Finally, the low number of records in some classes (anger, disgust, and surprise) is an additional challenge since the models tend to fail in those classes.
Fig. 1: Confusion matrix on test set produced by XLM-T multi-label classi er 4.2
As explained, we have designed an ensemble methodology to combine several base models and raise the nal classi cation performance. Our primary model is the one we have obtained through the use of the XLM-T transformer ne-tuned for the multiclass classi cation problem, and thus we include this model into the ensemble. Table 3 details the models used in the ensemble, along with the features they use to train. To evaluate the di erent models, the accuracy and weighted averaged F1 score have been used.
Word embedding combination (M G): averaged word vectors.
Uni and bi-gram representations.
TF-IDF features, considering both uni and bigrams.
SIMON features using an extracted word set. n-gram representations in combination with the dataset's additional features.
Ensemble that uses a logistic regression model to learn from base classi er predictions.
Ensemble that uses a random forest model to learn from base classi er predictions.
Ensemble LR combined with the dataset's additional features.
Ensemble RF combined with the dataset's additional features.
Ensemble LR + add. features, combined with the sentiment estimation obtained from MeaningCloud.
We have evaluated the models detailed in Table 3 on the development dataset [
Following, we can observe that adding additional features (event and o ensive categories) to the n-gram approach improves the regular n-gram features. This fact indicates that these additional features can be leveraged to improve classi cation performance.
Table 4 shows that combining all base models through an ensemble generally improves the classi cation performance when attending to the ensemble methods. Nonetheless, the metrics have not been improved over the XLM-T model, even though this model is included in the ensemble.
This situation changes when adding additional features and the MeaningCloud sentiment analysis to the ensemble. The ensemble using all features gets a lower accuracy, but the weighted F1 score is slightly higher in the development set, although this does not represent a relevant improvement. Please note that in this last case, the ensemble learner is trained with a combination of the predictions from the base classi ers, additional features, and MeaningCloud's sentiment analysis results.
When attending to the test set results, we have observed a di erent situation. The last ensemble, with additional features and sentiment analysis, does not improve the XLM-T nal performance. This paper has described our participation in the EmoEvalEs competition framed in the IberLef 2021 Conference. Our proposal relies on using large pretrained language models, outperforming previous methods with little e ort using the HuggingFace library, which provides a straightforward implementation of these pretrained language models. The pretrained model we have used is a RoBERTa transformer trained on a multilingual corpus of tweets, XLM-T. We have evaluated di erent strategies to approach the problem, nding that the ne-tuned model for a multiclass classi cation task obtains better results than the combination of various binary classi ers and the model with additional features. We have achieved rst place in the EmoEvalEs competition with this model, obtaining a macro-averaged F1 score of 71.70%.
This work also presents an ensemble method that combines several base classi ers with the XLM-T model to improve the nal performance by adding more knowledge to the system. Although we have found a slight improvement in the overall classi cation metrics in the development set, this enhancement has not continued in the test set. The obtained results suggest that combining such a transformer architecture with classical machine learning methods is a challenge, which must be done carefully.
We propose further lines of research to improve this work. Firstly, the e ective
combination of additional features that carry new information could enhance the
classi er's overall performance. Secondly, we suggest using a weighted validation
loss during the ne-tuning of the language model to deal with the unbalanced
dataset problem. Moreover, using a monolingual pretrained model for the speci c
language of the task could improve the obtained results. In this sense, using the
Spanish language model BETO [
The authors want to thank the help and support from the Catedra Cabify (Cabify Chair) at the ETSI Telecomunicacion of the Universidad Politecnica de Madrid. Moreover, the authors would also like to gratefully acknowledge MeaningCloud's support in facilitating this research work. Finally, the authors would like to acknowledge the support of NVIDIA Corporation with the donation of the Titan X Pascal GPU used in this research.