In recent years, tourist texts have taken on great importance in artificial intelligence investigations. This is due to the advantages that can be obtained from analyzing this type of text. One of them is to analyze the sentiment of tourists who leave, writing through digital platforms such as TripAdvisor. In this way, it is possible to automatically determine the user's experience, determine if their comment is positive or negative and through this information, find possible improvements that can be made to improve the experience of other tourists over time.
This task falls in the area of natural language processing, specifically within sentiment analysis. This task determines if the author of a text expresses himself positively or negatively about a product or service received. There are variations in the task where it is also about determining if the opinion is neutral; it can even go further and determine a numerical scale between 0 and N where 0 would be the most negative and N the most positive [8].
In this way, the sentiment analysis task can be seen as an automatic classification task where the instances are texts, and the class is the text polarity.
Typically, various textual representations are used for this task, such as ngrams, dictionaries, and embeddings, among others, used to feed classifiers, train them, and test them to observe their performance. However, there are scenarios where it is more critical than few computational resources are used both in time and memory due to limitations of specific tasks, for example, implementing solutions for IoT devices [5].
In this work, we propose to study the scope and effectiveness of features based on describing the text to be analyzed. For their simplicity, we will call these features Naive Features.
To test these types of features, the database that was released for the Rest-Mex 2021 evaluation forum will be used [2]. For this edition, a corpus of texts from tourists who visited Guanajuato in Mexico and its attractions was released. In this way, the effectiveness of these features can be tested in texts in Spanish since one of their advantages is that they are independent of the language.
The rest of the document is organized as follows; In section 2, the methodology followed in this work is described. In section 3, the results and their analysis are presented. Finally, section 4 presents the conclusions of this work.
This section presents the database with which it experimented and the proposed methodology to represent the tourist texts.
The analysis of sentiments task in tourism texts, which this year proposed within the Rest-Mex 2021 evaluation forum, predicts a class for each review provided in the evaluation set. The available classes are integers in the range [1,5]. Reviews were taken from the TripAdvisor website and were written by a tourist who evaluated some of the emblematic places in Guanajuato, Mexico. It is essential to mention that the whole set was in Spanish, being the first data set with these available features for evaluation.
The forum organizers released two different data sets ; one for training and one for evaluation. The training set consists of 5197 opinions with 9 pieces of information described below:
-Index: the index of each opinion.
-Title: The title that the tourist himself gave to his opinion.
-Opinion: The opinion expressed by the tourist. -Place: The tourist place that the tourist visited and to which the opinion is directed.
-Gender: the gender of the tourist.
-Age: The age of the tourist at the time of issuing the opinion.
-Country: The country of origin of the tourist.
-Date: the date the opinion was issued.
-Label: The label representing the polarity of the opinion: [1,2,3,4,5].
The training set classes are unbalanced. as Table 1 shows. Finally, the test data set contained 2216 rows and the same information as the training set, except the class information.
To attack the sentiments analysis task in tourist data, it is proposed to use simple features that can capture important information to determine the polarity of an opinion in such a way that it is quick to calculate and represent. Especially to offer an option for restricted applications in time or memory (such as IoT solutions) and that cannot use approaches that, although they have outstanding effectiveness results, can be slow or use much computational power, in addition to having the advantage of being language-independent features.
Given a text in the data set, its representation will be given by the following features proposed: The information available for each opinion will also be added as:
-F13: The gender of the person who gave the opinion -F14: The age of the person who gave the opinion -F15: The country of the person who gave the opinion For feature F13 that refers to the opinion author's gender, the value will be 0 if it is a man, 1 if it is a woman, and 2 if the gender of the person is not known. For feature F15, the country will be coded as 0 if the person is from Mexico or 1 if not.
Each option is transformed into a vector representation of dimension 15, which is easy and fast to calculate and independent of the language, which means that data sets in different languages can be evaluated.
The 10 fold cross-validation approach was used to classify the data set [9]. For each partition, the following classifiers were applied:
-Support Vector Machines (SVM) [6] k-Nearest Neighbor (KNN) with k ∈ {1, 3, 5, 7} [3] -Decision Tree (DT) [4] -Random Forest (RF) [7] -Naive Bayes (NB) [1] Accuracy, F-measure, and MAE were used as evaluation measures since it is the measure that the organizers take as official.
In this Section, the results obtained for the training partition are presented. Afterward, the chosen model is presented to be evaluated in the training partition together with its obtained result.
Table 2 shows the results for each classification algorithm for the representation described in the 2 section. This table shows the columns of accuracy (Acc), maco F-measure (F), the F-measure of each class (F CN where N represents each of the 5 classes), and mae (MAE).
In the results, it can be seen that SVM obtains the best results for accuracy and for class 5, which is the majority class; however, it obtains 0 for all other classes. This means that although it performs well for the measure mae, it is only classifying one class. On the other hand, KNN obtains the best results for macro measurement F. Class 1 obtains its best result with KNN-1, class 2 with
Algorithm Acc F F C1 F C2 F C3 F C4 F C5 MAE SVM 51,74 0,13 0 0 0 0 0,68 0,71 KNN-1 38,21 0,21 0,04 0,05 0,15 0,29 0,51 0,87 KNN-3 38,59 0,21 0,02 0,04 0,19 0,25 0,54 0,91 KNN-5 41,96 0,21 0 0,01 0,18 0,3 0,55 0,77 KNN-7 44,17 0,21 0,02 0,01 0,16 0,28 0,59 0,74 DT 38,57 0,2 0,01 0,04 0,15 0,3 0,52 0,87 RF 48,64 0,19 0 0,02 0,07 0,21 0,65 0,7 NB 47,46 0,19 0,02 0,08 0,01 0,21 0,64 0,79 Table 3 shows the best features under the information gain measure. Only the features that obtained a value greater than 0,1 appear in this table. These results give evidence that the best feature to solve this task is the number of words in the opinion, the second is the number of stop words, while the number of digits, the average of the length of the words, the different words not counting stop words and the length of the longest word complement the list.
Figure 1 shows the decision tree when only the word count feature is used. Ten opinions from class 1 (Bad label) and ten from class 5 (Good label) were used to build this decision tree. These opinions were chosen randomly. For this sample, it is possible to see that negatively valued opinions tend to have more words, which gives evidence that when the tourist is not satisfied, he uses more words to exorcise it, and could be the reason why this feature is important in this task.
It is clear to see that although a good mae result can be considered for all the algorithms, the F-measure results are shallow, which is a consequence of the data imbalance. This makes choosing a classification model not easy.
To choose the best model from those presented in Table 2 it is proposed to implement a weighting scheme to determine which of all the algorithms presents the best balance between the different results (accuracy, F-measure of each class, and mae).
It is proposed to generate a linear combination to measure the quality of each of the results obtained as presented in the equation 1.
Where C i represents the importance of each variable in the equation. F j represents the F-measure results for the class j. Acc, F, and MAE represent the valor of accuracy, macro F-measure and mae, respectively.
To choose the value of each constant C i , the following weights are proposed:
-C 1 : Since accuracy is not an important measure because the collection is unbalanced, it will only be given a weight of 1. -C 2 : It seeks to obtain a high F-measure macro result so that it will be given a weight of 10. -C {3,4,5,6,7} : The higher the number of opinions in a class, the easier it is to classify, which means that classes with little data are more complicated. In this way, it is sought to reward the well-classified elements of minority classes by putting as a weight 100 − D(i) where D(i) is the percentage of the class i. -C 8 . Since MAE is the measure taken into account to order the results by the organizers, it will be given the greatest weight, which must be negative since the ideal is to get as close to zero in this measure. Thus the value of this constant will be -100.
Table 4 shows the results of the equation 1. It is possible to see that the algorithm that presents the best value of Q is KNN-7, which did not present a high individual value of some measure; however, it is the one that obtains the best balance. On the other hand, SVM that obtained good results for accuracy, F-measure for class 5, and mae is the one that obtains the worst value of Q.
For this reason, in order to be evaluated in the Rest-Mex 2021 evaluation forum, it was decided to send the model generated by KNN-7 to the organizers.
For the official results, the model proposed obtained the following results:
-Accuracy: 45,71 -Macro F-measure: 0,17 -Mae: 0,76 With these results, the approach proposed obtained 10th place of 15 teams. Also, it is obtained better F-measure results than the baseline, and it was capable of classifying instances in three of the five classes.
In this work, a study was presented to measure the performance of naive features to attack the sentiment analysis problem for Mexican tourist texts.
This solution consisted of representing each tourist opinion in 15 simple features that can be extracted very quickly. This simplicity makes this solution ideal for some applications with an extreme limit of space and memory, for example, in IoT devices, and thus they can use some of these features to obtain an acceptable performance in a shorter response time.
When evaluating this solution in the Rest-Mex 2021 corpus, 0,76 of MAE was obtained, where the best result obtained in the competition was 0,47. Considering that the maximum possible error is 4 (when the result can be 5, and the prediction is 1, for example), 0,29 represents 7,25 % of the possible error, which is an acceptable loss considering the simplicity solution.
Evidence is given that the number of words in the opinion gives much information about polarity. Also, the length of the words is an essential source that a classifier can use. Other important features for this task are those that have to do with the stop words. Finally, for this task and in this database, demographic features such as gender, age, and place of origin of the author of the opinion do not seem to provide relevant information for the classification.
As work in the future, it is proposed to apply this solution to a multilingual collection and to be able to exploit its best feature, which is that it is a languageindependent solution.