In this paper, we describe the methods used to submit our results to the Rest-Mex Sentiment Analysis task of the Iberian Languages Evaluation Forum 2021. Our contribution is twofold. Firstly, we propose an unsupervised method for keyword extraction, in order to construct a list of prototypical words conveying a sentiment weight (pre-score). Secondly, we substantially improved a scoring system previously proposed by us. We emphasize here the match of the pre-scores of prototypical words with the labels of the texts where they appear. The classi cation task is done by a SVM applied to vector representations of text entities. These vectors are obtained as a partial mean of word representations, selecting the words with the highest absolute value of the score in each text entity.
Tourism has become a crucial source of revenue worldwide. From a socioeconomic point of view, tourism has become one of the largest and fastest growing industries in the world, extending activity online in the most recent decade [5]. In Mexico, this phenomenon is no exception, accounting for 8.7% of the national GDP, generating around 4.5 million direct jobs. However, with the COVID-19 pandemic, which began in Mexico in mid-March 2020, tourism was one of the most a ected sectors in this country [4].
In this context, the use of Arti cial Intelligence (AI), and in particular, Natural Language Processing (NLP), could be of great help to identify problems based on the analysis of the semantic aspects of tourists' opinions. In the case of tourism, a signi cant number of users express their views and opinions regarding the experience of traveling to a certain place through social media. These opinions are subjective information that represents the user's feelings, and the user's assessment associated with that experience. Online customer reviews of hotels and restaurants for tourism play a key role in decision making. Text reviews on travel websites can potentially in uence destination selection. Tourists use this information to satisfy their preferences. Similarly, managers of tourism services and public institutions dedicated to promoting tourism can use this information to improve customer service. In this way, tourism content shared through social networks has become a highly in uential source of information that may impact tourism in many ways. Thus, mining the opinions of tourists in search of the polarity of this opinion could in uence decision making throughout the value chain and support this industry.
In this paper, we describe the methods used to submit our results to the RestMex Sentiment Analysis task of the Iberian Languages Evaluation Forum 2021 [4]. For this competition edition, the sentiment analysis problem is de ned as follows: "Given an opinion about a Mexican tourist place, the goal is to determine the polarity, between 1 and 5, of the text." The sentiment analysis sub-task is a classi cation task where the participating system has to predict the polarity of an opinion issued by a tourist who traveled to the most representative places of Guanajuato, Mexico. This collection was obtained from tourists who shared their opinion on TripAdvisor between 2002 and 2020.
Sentiment analysis task in tourist texts has gained relevance in the last decade [5]. Despite the fact that most of the e orts have focused on English, there are some studies that have focused on Spanish language from Spain, and few address Spanish spoken in Mexico. These approaches are typically applied to collections taken from social networks such as tweets so that tourist texts have not been directly addressed [4]. In this sense, we can say that there are practically no linguistic resources that are suitable for processing the Mexican Spanish language, and that have been applied to the direct study of tourism opinion texts. Our work overcomes this gap and seeks to be a contribution in this direction. In order to train a classi er to predict the polarity of each entry, we rst extract keywords exclusive to each class of text entities. These keywords will ideally carry the sentiment weight in each class, and their presence may hint which class the text entity would belong to. Then, we de ne and implement a scoring system that assigns a value to each word in the corpus, accounting for the sentiment polarity of the word. Once these word scores are calculated, we de ne vector representations for text entities based on these scores and their word2vec embeddings. These vectors are the text features that will be feed to the classier. The last two steps of this proposed method is a follow up to the method presented in [20].
This paper is organized as follows: In Section 2 we describe the related work. In Section 3, we fully describe the keyword extraction method, the word scoring system and text representations. In Section 4, we describe the experimental setup and balancing strategies. In Section 5, we report and discuss the results of the experiments. Finally, we conclude in Section 6. 2
Sentiment analysis is concerned with the automatic extraction of sentimentrelated information from text. Traditionally, sentiment analysis has been concerned with opinion polarity (positive, negative, neutral), but in recent years there has been increasing interest in the a ective dimension (angry, happy, sad, etc) [13]. The main types of sentiment analysis algorithms belong to one of these three classes: Knowledge-based ; Machine Learning ; and Hybrid systems. Knowledge-based approaches perform sentiment analysis based on lexicons that typically incorporate sentiment word lists from many resources [9],[19]. To construct lexicons, one can use or compile Dictionaries (usually annotated by humans), or create lists of prototypical words that are further enriched with corpus data by seeking syntactic-semantic similarities.
Under the label of Machine Learning, we can nd two kinds of approaches. The rst can be viewed as a feature engineering problem, in which the objective is to nd a suitable set of a ect features in combination with an appropriate Machine Learning technique (e.g. Support Vector Machines, Classi cation Trees, Probabilistic models, etc.) [1], [2], [11], [5]. The second approach concerns the use of neural networks or deep learning architectures to learn sentiment-speci c word embeddings [18], [12].
Hybrid systems combine both approaches, and our work belongs to this category. Here, we can cite [6], whose method uses basic NLP tools, a sentiment lexicon enhanced with the assistance of SentiWordNet [9, 7], and fuzzy sets to estimate the semantic orientation polarity and its intensity for sentences. More recently, [24] proposed a hybrid sentiment classi cation method for Twitter by embedding a feature selection method. The authors used principal component analysis (PCA), latent semantic analysis (LSA), and random projection (RP) as feature-extraction methods. They presented a comparison of the accuracy of the classi cation process using Support Vector Machine, Nave Bayes, and Random Forest classi ers. They achieved performance rates on the order of 76%. Similar to our approach, [23] use word embeddings from word2vec to compute the Sentiment Orientation [21] of a Weibo (Tweet). In order to attain their goal, the objective of [23] was to construct a Sentiment Dictionary, based on a basic dictionary for which each word was previously annotated by humans with its polarity and intensity. The Sentiment Dictionary was constructed by extracting the 100 words that were most similar to every word in a Weibo, using the cosine similarity measure over the embeddings of both the words of the Weibo and the words of the annotated basic dictionary. Then, the authors proposed scoring methods for computing the Sentiment Orientation for a Weibo. Another example using word2vec is [3], who analyzed short texts from Bengali micro-blogging websites by using a tagged corpus of comments and sentiment scoring formulae based on empirically (trial and error) tuned parameters to achieve the highest performance.
In this paper, we propose a method, based on TextRank [14] and frequency counting, to construct a lexicon made of keywords in documents belonging to each label. These keywords will ideally carry the sentiment weight in each class and their presence may hint which class the text entity would belong to. Contrary to the previous version of our method [20], in which the lexicon was constructed manually, in this paper, we extract the keywords in an unsupervised manner. Other keyword extraction methods can be found in the literature, such as Rapid Automatic Keyword Extraction (RAKE) [17], Degree of Fractality [16], C-Value/NC-Value [10]. Also, as an improvement of the method reported in [20], in this paper, we propose to emphasize the contribution to the score of the words of each review having congruent polarity, and decreasing their score in incongruent cases; that is, \positive"/\negative" reviews should reinforce the contribution of \positive"/\negative" words and diminish this same contribution in \negative"/\positive" reviews. 3
In this section we provide a full description of the pre-processing of the corpus, the implementation of the word scoring system and the text representations that will be used as features for the classi er.
The word scoring system is an improvement of our scoring system described in [20]. 3.1
First, we de ne the notation we will be using. The corpus is made up of entries with the following form title1 opinion1 `1
... ... ...
titleN opinionN `N
We denote by W the vocabulary appearing in the union of the elds ftitleig and fopinionig. Each entity in the corpus has a label ` 2 [1; :::; 5] indicating the polarity of the opinion. The labels are hugely unbalanced as shown in Figure 1. Later, in Subsection 4.1, we will describe how we deal with the unbalanced classes and give more details about the text entities.
We remove stopwords, numbers and punctuation marks from the text entries. We consider each text entity i as the concatenation of titlei and opinioni. We de ne a normalized label ` 2 [ 1; 1] for each text entity, depending on the label ` of the entity, given by We rst compile a list of prototypical words W0, which should ideally carry certain sentiment weight, so that their presence in a text may help to predict the label of said text. Instead of considering a manually de ned list of words as in [20], we now use a non-supervised method to extract these prototypical words. Each of these words z 2 W0 will have a pre-score s~(z) 2 [ 1; 1]. We obtain the list W0 using a hybrid approach. On the one hand, we use TextRank [14] which assigns an index (w) to each word w in a document. This index accounts for the importance of this word in the document. We apply TextRank to the each of the 5 documents made up of all text entities in each of the 5 label classes; thus, we obtain 5 lists of words `, ` = 1; :::; 5. Each list consists of the 20 words with the highest index (wj) in each label. If a word appears in more than one list `, we keep the word only in the list where it has the highest index. These lists contain the most important words in each label, according to TextRank.
Also, we apply TextRank to the whole collection of text entities to extract the 20 most important words in the whole corpus, we denote this set of words by
G. It is worth noting that the lists ` may contain words that are important in the corpus as a whole; i.e. words in ` that also belong to G. Hence, we will take the following subset of each ` containing words that are only relevant to each speci c label We denote by L(w) = ` the index of the unique list ` to which the word w 2 W0TR belongs to.
Recall that, in each label, each word w 2 ` has a TextRank index given by (w). Let f : [ min; max] ! [0; 1] be the function applying the min-max normalization. We also could have used another keyword extraction method for this previous step, such as RAKE [17], Degree of Fractality [16] or C-Value/NC-Value [10]. However, TextRank showed the best performance in the classi cation task compared to the other methods.
On the other hand, we consider the following frequency-based pre-scoring of words. For each word w 2 W, we de ne Fi(w) as the frequency of w in the collection of text entities with label ` = i, where i = 1; :::; 5. We de ne t~(w) = 1 5
X ` Fi(w);
F (w) i=1 where F (w) is the frequency of w in the corpus. This function t~(w) assigns, to every word w, a weighted mean of the labels where it appears; the weights are the frequencies in each label. In order to dampen the e ect of the frequencies, we de ne We consider the 30 words in W with the largest jt(w)j, and we denote this list of words by W0t.
Finally, the list of prototypical words W0 is given by t(w) = 1
e1 F (w) t~(w)
W0 = W0t [ W0TR;
(
Now, we describe the word scoring system. This system assigns a value to each
word, accounting for the sentiment content of the word. A rst version of the
approach may be found in [20]. Here, we propose a substantial improvement by
emphasizing the match of the pre-scores of prototypical words with the labels
of the texts where they appear; in other words, we de ne new label-dependent
scores. To achieve this, we will consider the prototypical pre-scores of Equation
(
We denote by w the embedding of the word w. We consider a list of neighbours of words in W0, denoted by W1, de ned as
W1 = fz 2 W j 9w 2 W0; sim(z; w) > g The hyper-parameter 0 de nes a closeness threshold. In other words, it de nes how similar two word embeddings have to be in order to consider them as neighbours. We use = 0:5.
For each word z 2 W1, there exists another word (z) 2 W0 such that sim(z; (z)) = max sim(z; ): 2W0 We de ne a label-dependent score for each word w, denoted by s~`(w). Let s~`(w) = 0 whenever w 62 W0 [ W1.
For each word w 2 W1 in a text entity with label ` consider the term ` s~( (w)). We have two cases: 1. If ` s~( (w)) 0, then
s~`(w) = tanh ( 1x) ; where x = ` + sim(w; (w)) s~( (w))
where F (w) is the frequency of w in the corpus and F`(w) the frequency of this
word in text entities with label `.
In this subsection, we de ne the representations of words and text entities, using
the word scoring de ned in Equation (
wj1 ; :::; wjn js(wj1 )j > ::: > js(wjn )j:
k R(m) = 1 X r(wji ): k i=1 This is the representation of text entities we will use as features for the classi er. 4 4.1
In this subsection, we describe how we deal with the unbalanced classes. Our approach is twofold: rst, we modify the text entities by taking combinations of titles and opinions in order to create new instances of the rst three classes; and second, we use SMOTE [8] (Synthetic Minority Oversampling Technique) to completely balance the classes. SMOTE is an algorithm that creates new instances of unbalanced classes by taking points lying in the line segments between points of unbalanced classes, thus, preserving the convex envelope of the unbalanced class.
The counting of labels is shown in Figure 1. We denote by i1; :::; ip the indices of the entries with label ` = 1; 2, and we de ne new text instances as shown in Table 2. In Tables 2 and 3 the addition sign denotes concatenation of strings. We denote by j1; :::; jq the indices of the entries with label ` = 3; 4; 5, we de ne new text instances as shown in Table 3.
titlej1 + opinioni1 `i1
... ...
titlejq + opinionN `jq
Our set of text entities will be given by Tables 2 and 3. Figure 4 shows the label counting at this stage.
This set of instances is passed to SMOTE to obtain a completely balanced set of classes, each class now has 2689 instances. 4.2
We start with the corpus, as described by Table 1. Then, we train a word2vec model [15] on the pre-processed text, which is the list of tokens in each text entity. We denote this trained model M1 with vector size d = 100. We also consider the pre-trained word2vec model M2 [22] with vector size d = 300. For a word w, we denote by w the word embedding given by M1 and w0 the word embedding given by M2.
We calculate the sets W0 and W1, and the words scoring as described in subsections 3.2 and 3.3, using the model M1.
We apply the balancing strategy described in Subsection 4.1 to the corpus and obtain a set of K = 160 + 290 + 686 + 1595 + 2689 = 5420 labeled text entities as follows Fig. 4: Labels counting after taking the set of text entities given by Tables 2 and 3. The classes now have 160, 290, 686, 1595, 2689 instances respectively Now, using the model M2, we apply the text representations described in Subsection 3.4 to obtain the following set of features text1 `1
... ... textK `K R(m1) `1
::: :::
R(mK ) `K We split this data set into training and test (validation) sets using a 4:1 ratio. We feed the training set to a support vector machine with Gaussian kernel. 4.3
Once the models were trained, the test data, which were not labeled, were received. Thus, we describe now the experimental setup used to produce the test results. The test corpus consists of entries of the following form title1 opinion1
... ...
titleN0 opinionN0
We removed stopwords, numbers and punctuation marks from each text entry. We consider each text entity i as the concatenation of title and opinion as shown in Table 5. This will be test corpus. As the texts in the test data set brought a large amount of unknown words (not previously seen during training), we decided to resort to the model M2 to build the test vocabulary. We denote by WT the corpus resulting from the intersection of the M2 model vocabulary and the test corpus (Table 5). In order to obtain the word representations of the words in WT , we take a word w 2 WT , and consider two cases:
2. If w 62 W, the representation of w is given by r( (w)), where (w) = argmax sim (w0; z0) :
z2W Now, we have a word representation for every word in WT . Therefore, we can obtain the text entities for every instance of Table 5. These vector representations will be used as features of each instance, and they will be fed to the pre-trained classi er in order to obtain the test label predictions. 5
In this section, we report and discuss the results obtained after performing the experiments described in Section 4. The classi er performance, on the validation (training phase), and test subsets is shown in Table 6. In Figure 5 we show the confusion matrix of the classi cation task for the validation phase. The Mean Absolute Error (MAE) was the primary metric used to determine the overall ranking of participants. With respect to this metric, our results were ranked in the 7th place out of the 14 di erent runs.
Looking at Table 6, we can see that, compared to the baseline (Majority Class), our model performs 11% better with respect to MAE but 4.5% below with respect to Accuracy. This can be explained by the large imbalance in the data set, which would also explain the poor performance of the baseline in terms of recall. Compared to the best result of the competition, our performance is 35% Metric MAE Accuracy Recall
Validation (training phase) and 13.5% below in terms of MAE and Accuracy respectively. Our recall is also nearly 35% below the best result. In all metrics (except for the accuracy), our method performed better than average. This is shown in Figure 6. This gure was obtained by taking the distribution of the results of all participants, in each metric reported by the competition organizers. The gure shows violin plots depicting the distribution of the results, where the average, the best result and our performance are depicted on this plot.
Finally, it is worth noting the performance drop between the train (validation) and test metrics (see Table 6). We brie y discuss the possible reasons behind this drop in performance and some strategies to improve our method. Recall from Subsection 3.4 that features used to characterize the text entities are means of vector representations of words. As a consequence of the underlying vector addition involved, many vector representations of text end up being close to the origin, thus, making the classi cation task more di cult. In order to avoid this e ect, the initial vectors must be close to each other. Their closeness will determine a threshold which is key to better understand and solve this problem. Fig. 6: Performances of all the participating runs in each of the metrics of the competition. The black dot in each metric is the performance of our method. The red point is the best performance in each metric. The white point is the average of the performances. where jzj denotes the Euclidean norm.
One possible bound is the following: If r0 = minw2W jwj and 12 > r > 0, then for any two vectors u; v 2 W, such that sim(u; v) > = 2 rr22 , it holds 21 ju + vj > r, 0 On the other hand, there are some strategies to possibly improve the performance of our method when it faces unlabeled documents. As a rst approach, we can restrict our attention to speci c parts of speech, such as adjectives and adverbs, since these words might carry more sentiment weight than verbs or nouns. 6
In this paper, we presented the methods we used to address the Rest-Mex Sentiment Analysis task, of the Iberian Languages Evaluation Forum 2021 [4]. We rstly proposed an unsupervised method for keyword extraction using TextRank [14]. Then, we proposed a substantial improvement to our word scoring system [20]. This improvement consisted in emphasizing the contribution to the score of the words of each review having congruent polarity, and decreasing their score in incongruent cases; that is, \positive"/\negative" reviews reinforced the contribution of \positive"/\negative" words. The polarity of the words was calculated based on their similarity to prototypical words obtained from the training corpus in a unsupervised manner.
The classi cation task is done by a SVM applied to vector representations of text entities, obtained as a partial mean of word embeddings. The results obtained were ranked in the 7th place out of the 14 di erent runs. They outperformed the majority class baseline in MAE and Recall, and performed slightly better than the average nal test results. Better performance may be obtained by improving the text representations, and possibly focusing on speci c parts of speech, such as adjectives and adverbs. This is because these classes of words might carry more sentiment weight than verbs or nouns.
Research was partially funded by CONACYT Project A1-S-24213 of Basic
Science and CONACYT grant 28268.
8. Chawla, N.V., Bowyer, K.W., Hall, L.O., Kegelmeyer, W.P.: Smote: synthetic
minority over-sampling technique. Journal of arti cial intelligence research 16, 321{
357 (2002)
9. Esuli, A., Sebastiani, F.: Sentiwordnet: a high-coverage lexical resource for opinion
mining. Evaluation 17(