This paper presents a simple approach to extract naive features to represent and classify tourists' opinions in Mexican places to participate in the Rest-Mex 2021 evaluation forum. The proposed approach consists of extracting 15 simple features. Then, various classi cation algorithms were used to evaluate the quality of these features, such as SVM, KNN, Decision Tree, Random Forest, and Naive Bayes. A weighting scheme was also proposed to obtain the best combination between algorithms and features, where it turned out that the best algorithm for this set of features was KNN with seven neighbors. Of these features, the best turned out to be what had to do with the length of words and characters and the number of stop words. With this approach, 0.76 of MAE was obtained, obtaining 10th place out of 15 teams, which considering the simplicity of this solution, makes it an acceptable result.
In recent years, tourist texts have taken on great importance in arti cial intelligence investigations. This is due to the advantages that can be obtained from analyzing this type of text. One of them is to analyze the sentiment of tourists who leave, writing through digital platforms such as TripAdvisor. In this way, it is possible to automatically determine the user's experience, determine if their comment is positive or negative and through this information, nd possible improvements that can be made to improve the experience of other tourists over time.
This task falls in the area of natural language processing, speci cally within sentiment analysis. This task determines if the author of a text expresses himself positively or negatively about a product or service received. There are variations in the task where it is also about determining if the opinion is neutral; it can even go further and determine a numerical scale between 0 and N where 0 would be the most negative and N the most positive [8].
In this way, the sentiment analysis task can be seen as an automatic classication task where the instances are texts, and the class is the text polarity.
Typically, various textual representations are used for this task, such as ngrams, dictionaries, and embeddings, among others, used to feed classi ers, train them, and test them to observe their performance. However, there are scenarios where it is more critical than few computational resources are used both in time and memory due to limitations of speci c tasks, for example, implementing solutions for IoT devices [5].
In this work, we propose to study the scope and e ectiveness of features based on describing the text to be analyzed. For their simplicity, we will call these features Naive Features.
To test these types of features, the database that was released for the
RestMex 2021 evaluation forum will be used [
The rest of the document is organized as follows; In section 2, the methodology followed in this work is described. In section 3, the results and their analysis are presented. Finally, section 4 presents the conclusions of this work. 2
This section presents the database with which it experimented and the proposed methodology to represent the tourist texts. 2.1
The analysis of sentiments task in tourism texts, which this year proposed within
the Rest-Mex 2021 evaluation forum, predicts a class for each review provided in
the evaluation set. The available classes are integers in the range [
The forum organizers released two di erent data sets ; one for training and
one for evaluation. The training set consists of 5197 opinions with 9 pieces of
information described below:
{ Index: the index of each opinion.
{ Title: The title that the tourist himself gave to his opinion.
{ Opinion: The opinion expressed by the tourist.
{ Place: The tourist place that the tourist visited and to which the opinion is
directed.
{ Gender: the gender of the tourist.
{ Age: The age of the tourist at the time of issuing the opinion.
{ Country: The country of origin of the tourist.
{ Date: the date the opinion was issued.
{ Label: The label representing the polarity of the opinion: [
The training set classes are unbalanced. as Table 1 shows.
Finally, the test data set contained 2216 rows and the same information as the training set, except the class information. 2.2
To attack the sentiments analysis task in tourist data, it is proposed to use simple features that can capture important information to determine the polarity of an opinion in such a way that it is quick to calculate and represent. Especially to o er an option for restricted applications in time or memory (such as IoT solutions) and that cannot use approaches that, although they have outstanding e ectiveness results, can be slow or use much computational power, in addition to having the advantage of being language-independent features.
Given a text in the data set, its representation will be given by the following features proposed: { F1: Number of capital letters in the opinion { F2: The length of the longest word in the opinion { F3: The average words length in the opinion { F4: Number of words in the opinion { F5: Number of characters in the opinion { F6: The ratio between the number of di erent words and total words in the opinion { F7: The number of digits in the opinion { F8: The ratio of the number of stop words and total words in the opinion { F9: Number of punctuation marks in the opinion { F10: Number of stop words in the opinion { F11: Number of characters in the opinion without stop words { F12: The ratio between the number of di erent words and total words in the opinion without stop words
The information available for each opinion will also be added as: { F13: The gender of the person who gave the opinion { F14: The age of the person who gave the opinion { F15: The country of the person who gave the opinion
For feature F13 that refers to the opinion author's gender, the value will be 0 if it is a man, 1 if it is a woman, and 2 if the gender of the person is not known. For feature F15, the country will be coded as 0 if the person is from Mexico or 1 if not.
Each option is transformed into a vector representation of dimension 15, which is easy and fast to calculate and independent of the language, which means that data sets in di erent languages can be evaluated.
The 10 fold cross-validation approach was used to classify the data set [9].
For each partition, the following classi ers were applied:
{ Support Vector Machines (SVM) [6]
{ k-Nearest Neighbor (KNN) with k 2 f1; 3; 5; 7g [
Accuracy, F-measure, and MAE were used as evaluation measures since it is the measure that the organizers take as o cial. 3
In this Section, the results obtained for the training partition are presented. Afterward, the chosen model is presented to be evaluated in the training partition together with its obtained result. 3.1
NB, class 3 with KNN-3, and class 4 with DT. The best result of mae is obtained with RF; however, this algorithm cannot capture any instance of class 1.
Table 3 shows the best features under the information gain measure. Only the features that obtained a value greater than 0,1 appear in this table. These results give evidence that the best feature to solve this task is the number of words in the opinion, the second is the number of stop words, while the number of digits, the average of the length of the words, the di erent words not counting stop words and the length of the longest word complement the list.
Figure 1 shows the decision tree when only the word count feature is used. Ten opinions from class 1 (Bad label) and ten from class 5 (Good label) were used to build this decision tree. These opinions were chosen randomly. For this sample, it is possible to see that negatively valued opinions tend to have more words, which gives evidence that when the tourist is not satis ed, he uses more words to exorcise it, and could be the reason why this feature is important in this task.
It is clear to see that although a good mae result can be considered for all the algorithms, the F-measure results are shallow, which is a consequence of the data imbalance. This makes choosing a classi cation model not easy. 3.2
To choose the best model from those presented in Table 2 it is proposed to implement a weighting scheme to determine which of all the algorithms presents Fig. 1. Decision tree for the number of words feature on 20 random opinions Algorithm SVM KNN-1 KNN-3 KNN-5 KNN-7 DT RF NB
Q 14,63 19,63 15,20 30,74 36,73 17,11 34,22 26,17 the best balance between the di erent results (accuracy, F-measure of each class, and mae).
It is proposed to generate a linear combination to measure the quality of each of the results obtained as presented in the equation 1.
Q = C1 Acc+C2 F +C3 F1 +C4 F2 +C5 F3 +C6 F4 +C7 F5 +C8 M AE (1)
Where Ci represents the importance of each variable in the equation. Fj represents the F-measure results for the class j. Acc, F, and MAE represent the valor of accuracy, macro F-measure and mae, respectively.
To choose the value of each constant Ci, the following weights are proposed: { C1: Since accuracy is not an important measure because the collection is unbalanced, it will only be given a weight of 1. { C2: It seeks to obtain a high F-measure macro result so that it will be given a weight of 10. { Cf3;4;5;6;7g: The higher the number of opinions in a class, the easier it is to classify, which means that classes with little data are more complicated. In this way, it is sought to reward the well-classi ed elements of minority classes by putting as a weight 100 D(i) where D(i) is the percentage of the class i. { C8. Since MAE is the measure taken into account to order the results by the organizers, it will be given the greatest weight, which must be negative since the ideal is to get as close to zero in this measure. Thus the value of this constant will be -100.
Table 4 shows the results of the equation 1. It is possible to see that the algorithm that presents the best value of Q is KNN-7, which did not present a high individual value of some measure; however, it is the one that obtains the best balance. On the other hand, SVM that obtained good results for accuracy, F-measure for class 5, and mae is the one that obtains the worst value of Q.
For this reason, in order to be evaluated in the Rest-Mex 2021 evaluation forum, it was decided to send the model generated by KNN-7 to the organizers. 3.3
O
cial results { Accuracy: 45,71 { Macro F-measure: 0,17 For the o cial results, the model proposed obtained the following results:
With these results, the approach proposed obtained 10th place of 15 teams. Also, it is obtained better F-measure results than the baseline, and it was capable of classifying instances in three of the ve classes. 4
In this work, a study was presented to measure the performance of naive features to attack the sentiment analysis problem for Mexican tourist texts.
This solution consisted of representing each tourist opinion in 15 simple features that can be extracted very quickly. This simplicity makes this solution ideal for some applications with an extreme limit of space and memory, for example, in IoT devices, and thus they can use some of these features to obtain an acceptable performance in a shorter response time.
When evaluating this solution in the Rest-Mex 2021 corpus, 0,76 of MAE was obtained, where the best result obtained in the competition was 0,47. Considering that the maximum possible error is 4 (when the result can be 5, and the prediction is 1, for example), 0,29 represents 7,25 % of the possible error, which is an acceptable loss considering the simplicity solution.
Evidence is given that the number of words in the opinion gives much information about polarity. Also, the length of the words is an essential source that a classi er can use. Other important features for this task are those that have to do with the stop words. Finally, for this task and in this database, demographic features such as gender, age, and place of origin of the author of the opinion do not seem to provide relevant information for the classi cation.
As work in the future, it is proposed to apply this solution to a multilingual collection and to be able to exploit its best feature, which is that it is a languageindependent solution. 4. Charbuty, B., Abdulazeez, A.: Classi cation based on decision tree algorithm for machine learning. Journal of Applied Science and Technology Trends 2(01), 20{28 (2021) 5. Dinculeana, D., Cheng, X.: Vulnerabilities and limitations of mqtt protocol used between iot devices. Applied Sciences 9(5), 848 (2019) 6. Huang, Y., Zhao, L.: Review on landslide susceptibility mapping using support vector machines. Catena 165, 520{529 (2018) 7. Mishra, S., Tadesse, Y., Dash, A., Jena, L., Ranjan, P.: Thyroid disorder analysis using random forest classi er. In: Intelligent and cloud computing, pp. 385{390.
Springer (2021) 8. Mukherjee, S.: Sentiment analysis. In: ML. NET Revealed, pp. 113{127. Springer (2021) 9. Rohani, A., Taki, M., Abdollahpour, M.: A novel soft computing model (gaussian process regression with k-fold cross validation) for daily and monthly solar radiation forecasting (part: I). Renewable Energy 115, 411{422 (2018)