This paper presents our approach to the Sentiment Analysis Task at REST-MEX 2021. The goal of the task is to predict the polarity of opinions on Mexican tourist sites. Recent advances in transfer-learning with pre-trained models in English have produced state-of-the-art results in sentiment analysis. In this work, we apply two Bert-based approaches for review classi cation in ve classes. Our rst approach consists of ne-tuning Beto, a Bert-like model pre-trained in Spanish. Our second approach focuses on combining Bert embeddings with the feature vectors weighted with TF-IDF. The results obtained using the standalone BERT model ranked rst in the task.
Sentiment analysis is an area of research in Natural Language Processing (NLP)
whose goal is to extract a sentiment or emotion of a given opinion [
To encourage the research community in NLP to develop new research areas
in Spanish, the Iberian Language Evaluation Forum (IberLEF) organizes yearly
a comparative evaluation campaign [
This paper is organized as follows: Section 2 states the task we worked on and the data sets provided by the task organizers; Section 3 describes the two systems that we designed and implemented; Section 4 details our experiments and the obtained results; and Section 5 reports our conclusions. 2
The sentiment analysis task required the prediction of a class for each review provided in the evaluation set. The available classes were integers between 1 and 5. The reviews were taken from the website TripAdvisor and were written by a tourist who evaluated a landmark in Guanajuato, Mexico. All of them were in Spanish.
The participants were provided with two di erent data sets; one for training
and one for evaluation. The training set was made up of 5197 rows and 9 columns
described as follows:
1. Index: The index of each opinion.
2. Title: The title that the tourist himself gave to his opinion.
3. Opinion: The opinion expressed by the tourist.
4. Place: The tourist place that the tourist visited and to which the opinion is
directed.
5. Gender: The gender of the tourist.
6. Age: The age of the tourist at the time of issuing the opinion.
7. Country: The Country of origin of the tourist.
8. Date: The date the opinion was issued.
9. Label: The label that represents the polarity of the opinion: [
For our experiments, we only trained the models with the \Opinion" column.
The classes in the training set were not balanced. The distribution can be seen in Table 1
At 2020's edition of the task on Semantic Analysis at SEPLN, a Bert-like model
for sentiment analysis at three levels yielded the highest accuracy [
Another reason for working with Bert was that ne-tuning it for downstream
tasks, such as sentiment analysis, is computationally inexpensive [
Because the reviews in the data sets were written in Spanish, we decided to
use Beto as our baseline model. This Bert-like system was pre-trained using a
corpus in Spanish with a similar size to that of the corpus used to train
BertBase [
Section 3.1 describes the approach for ne-tuning Beto, which ranked rst in
this Task. Section 3.2 outlines a second approach we took in hopes of improving
our results. In this system, we added a Bag-of-words feature vectors weighted
with TF-IDF, to the contextual embeddings generated by Beto after ne-tuning
it. The motivation behind this was that TF-IDF captures global information
from all the entries in the data set [
It is important to mention that, before proceeding with the ne-tuning, we performed a data pre-processing step, which consisted of removing the quotation marks in the reviews.
The code for implementing the systems listed here can be found in our Github repository. 3.1
First, we passed the reviews through the Bert tokenizer (loaded with the weights from Beto). Once the tokens were generated, we fed them to Beto with a classi cation layer on top. This step executed the domain-speci c training (or netuning). The next step was to repeat this same process on the o cial evaluation set. Once we tokenized the reviews in this second set, we predicted the classes by passing these tokens through the previously ne-tuned model.
The hyperparameters used for the ne-tuning were the ones recommended
in the original Bert paper [
We started by extracting the contextual embeddings generated after ne-tuning Beto with the training set (we followed the same steps listed on Section 3.1). Then, we obtained a new set of features by rst tokenizing the original training set using Spacy's model \es dep news trf". Next, we calculated the TF-IDF weights for those tokens. Once we had the two set of features, we concatenated them into one set, which we then used for training a logistic regression algorithm. Next, we generated the corresponding contextual embeddings from Beto and TF-IDF bag-of-words features for the evaluation set. Finally, we predicted the classes with the previously trained logistic regression model.
For this system we utilized the same hyperparameters listed on 3.1. 4
For this competition, the ranking was determined by measuring the systems
with the mean absolute error (MAE). As can be seen in Equation 1, this metric
outputs a nal number which is calculated by summing the magnitudes (absolute
values) of the errors to obtain the \total error" and then dividing the total error
by n [
n M AE = n 1 X jeij i=1 (1)
In order to get an overview of how our systems performed, we tested various classi cation algorithms on a partition of the training set. For all the experiments in Table 2, we tokenized the reviews using the Spacy model \es dep news trf". The next step was to generate the TF-IDF weights. Then, we trained the di erent classi cation algorithms. Finally, we evaluated each model.
The results in Table 2 show the performance of the supervised algorithms, of the ne-tuned Beto, and of Beto with the added TF-IDF feature vectors. It is observed that ne-tuning Beto yields the best results in this task.
Table 3 presents the results provided by the organizers of the competition. These metrics were obtained on the evaluation set. 5
The approach described in 3.1 obtained the lowest mean absolute error in the Subtask 2 of this year's sentiment analysis shared task at IberLef. This suggests that transfer learning is a functional approach for sentiment analysis with ve classes. Even though we achieved the lowest mean absolute error among all the teams in the shared task, we consider that this number is still very high. This could be due to the di culty of doing classi cation among ve classes.
One limitation of our approach is that transfer learning depends on the corpora used to pre-train the model. This restricts the learning capabilities to certain topics. Also, Beto was pre-trained using a corpus with a size similar to that of Bert-Base. We hypothesize that using a pre-trained model with more parameters would greatly improve our results.
Another limitation of Beto is the computational power. Until now, these architectures can only deal with a maximum length of 512 tokens per sequence. This means that some reviews that exceed that token length are truncated before the encoding, leading to a loss in data, and, therefore, in learning.
Furthermore, we propose working on di erent approaches to multi-class classi cation. While our second system did not produce better results than the rst one, it did achieve the fth place among the participants. We hypothesize that combining di erent sets of features generated by di erent language models could improve the classi cation results when faced with ve classes.
Finally, by manually analyzing the data set, we observed that sometimes the classes had very little relation to the review. For example, the following review was labeled with class 1: \If you go as a couple this place is a must, it is special to climb and kiss on the third step, very romantic and emblematic". When reading the review, one can not infer that this would be given such a low class. This kind of annotation in the data set could be the reason behind the poor performance obtained in this task. This research was partially funded by CONACyT project CB A1-S-27780 and DGAPA-UNAM PAPIIT grants number TA100520 and TA400121.