This article presents a proposal for the treatment of an unbalanced tourist database with emphasis on minority classes for its classification, in this case, one based on BERT, called BETO. This methodology originally forms part of the thesis project of the authors, with the objective of balancing data with a tourist focus and being able to measure the impact that it has on the classification of texts.
The tourism sector contributes around 8% of the Gross Domestic Product (GDP) [
Since 2021, the Rest-Mex team has served as an evaluation forum that seeks to specialize in
the analysis of texts from the tourism sector to solve diferent tasks in Mexican Spanish. In the
2023 edition, there are two tasks where the opinions of tourists, which are the object of analysis,
obtained from the TripAdvisor site [
This article describes the participation of the Dataverse team in the sentiment analysis task. First, a Dataset description is shown, followed by Proposed methodology for class (a) Distribution by “Country” (b) Distribution by “Type” (c) Distribution by “Polarity” balance and their classification. Subsequently, the Results obtained are shown, ending with the Conclusions and future work.
There is a total of 251,702 tourist reviews obtained from TripAdvisor tagged as follows:
• Polarity: It is an integer between [
Mexico, Cuba, or Colombia. • Type: Represents the type of destination being reviewed and can be: Hotel, Restaurant, or Attractive.
The distribution of the data by class and by label type is depicted in Figure 1. As can be seen in Fig. 1b the Type reviews are not very unbalanced, for country reviews, Fig. 1a, there is a slight imbalance that can be dealt with. On the other hand, for polarity, Fig. 1c, there is a very notorious imbalance, as seen in Fig. more than half of the reviews are from class 5, that is, very positive, in contrast to the data from class 1, very negative. and 2 negative, which present less than 5% of the reviews, and these represent the minority classes [11].
For the preprocessing part of the data, it was all lowercase; special characters, multiple spaces, numbers, stop words and words of length equal to or less than 3 were removed.
It is important to deal with minority classes, to address class imbalance to improve the performance of our model by avoiding possible bias, especially when there are highly imbalanced classes, because if the data set is biased towards one class, the model trained with the same data will be biased towards the same class[12].
There are several methods that can be used to address class imbalance. They are subsampling and oversampling. On the one hand, there is subsampling, which consists of reducing the data obtained by randomly taking reviews from the majority classes. On the other hand, there is oversampling, which is the increase in data, generally on the representation of minority classes where new instances can be generated or duplicate existing data of those classes randomly. In this last case, it has been found that it is not very eficient since new information is not introduced, it does not address the fundamental problem of the lack of information and real variability in this type of classes. However, generating new instances can alleviate this problem, but overfitting should be avoided[13].
In the case of subsampling, it is proposed to randomly obtain a certain number of reviews than the total of each one that coincides with the number of reviews of the “neutral” class, in this case reduce class 4 and 5. In the case of oversampling It is proposed to obtain the representative words of all classes with mutual information, which helps us to measure the dependence or association between two random variables. In this context, to measure the association between words within classes[14]. It can be defined as follows: (, ) (, ) = 2 ( () ()) (1) where (, ) is the joint probability that word and class appear together. () and () are the marginal probabilities that they will appear in reviews. Mutual information measures the deviation of the joint probability of and from what would be expected if they were independent. A higher value of mutual information indicates a stronger association between the word and the class.
Once you have these words, ordered from the most representative to the least, synthetic opinions are generated, which consists of replacing one of the representative words that will generate a new review as a synonym from a Web dictionary or a FastText embedding.
The synonyms are obtained from the Word Reference virtual dictionary, and the embeddings from the Spanish version of FastText trained by Common Crawl, which is an organization that has been doing web scraping since 2008 and makes its data public, and with the free encyclopedia Wikipedia [15].
Having already the possible words that can be chosen for the generation of new instances, it should be considered that by having representative words of each class, if only those values were taken, there could be a risk of bias and being over adjusted both in the classes. In general, as well as in the context of the data that is available, this could be observed using techniques that help us to lower the dimension of the entire data set and see the behavior of the new data. To mitigate the above, it is proposed to include the hyperparameters temperature and probability.
The generation function requires the following hyperparameters: • The database of representative words • The class in which to generate new data • The maximum number of words that can be in the new data • What other classes are taken into account for the generation • Temperature • Probability • Synonym source type
First, the number of words that the text will have is randomly chosen, there is a minimum of 4 and a maximum of the total number of words that can be included, given as a hyperparameter.
The probability refers to whether or not one of the representative words is considered as a synonym for the generation of the new text. The procedure consists of first giving an integer and then choosing a random number between 1 and that given number. Having that value, it is verified if it matches the number given at the beginning. For example, if you give the number 2, there is a 50% probability that the given word will be considered, on the other hand, if the number 1 is given, there is a 100% probability of including it in the new data.
Subsequently, it is decided if it is with a dictionary or FastText, in any of both, a list of similar words based on is obtained and a word is chosen according to a certain temperature, which will be the random number between 0 and the minimum between a given number and the total number of synonyms that are taken into account. For example, if the number 1 is given, the minimum between the two is taken, and the random one between 0 and 1 is more likely that a word is similar to the representative one, in this case it can be the first or second value of the list of synonyms, that is, the most similar ones, on the other hand, if a large value is given, it gives more possibility of taking words that are further away from the representative word that is found. On the other hand, in order not only to have words from one class, a limited random number of words that can be included are taken from the representative words of the respective classes chosen to enter the model. To finish this part, all the values generated are taken to form the text strings where the length of each one varies and the type of words that are also present.
The classification model used is BERT (Bidirectional Encoder Representations from Transformers) is a language model based on neural networks that has revolutionized natural language processing (NLP). BERT is a pre-trained model that uses the Transformer architecture, which is an attention-based neural network. Unlike previous language models that were trained in a unidirectional fashion (left to right or right to left), BERT is bidirectional, which means that it can capture the context of words in both directions[16, 17].
BERT training is done in a task known as “pretraining language modeling”. During this stage, BERT is trained to predict hidden words in masked sentences and to predict the relationship between pairs of sentences. This massive training allows you to learn contextualized representations of words, capturing information from the context that surrounds them. in the Spanish version that is called BETO.
Training details: • 3 epochs • Adam Optimizer using a learning rate of 5− 5 • Batch size of 32 elements • Unfreezed BETO weights
In this edition of Rest-Mex, the Dataverse team evaluated models with diferent metrics, however, minority classes are given more weight. The team placed seventh in the competition with a Sentiment Track Score of 0.7173586609. The results of the metrics obtained are the following:
The result of the classification of the test reviews that was sent, the dictionary option was taken into account, with temperature 100 and probability 1, they were balanced with respect to polarity and countries. From the results of the metrics obtained, as can be seen in the Table1, it can be seen that in most of the metrics, with the exception of the MAE in the median, both the average and median values of all the participants were exceeded. Something similar happens in the F1 scores of the minority classes as seen in the Table 2.
It can be concluded that the oversampling and subsampling techniques are very useful together with the BERT model, it produced good results, but particularly not the best ones. However, oversampling shows that it can be an efective strategy to improve the performance of models in unbalanced data sets. Nevertheless, it is important to note that oversampling must be applied carefully, as excessive generation of synthetic examples can lead to overfitting and degradation of model performance. Tests were needed with the combinations of temperature and probability, which could have better results in this competition.
The authors thank the Mexican Academy of Tourism Research (AMIT) for their support of the project ”Creation of a labeled database related to tourist destinations for training artificial intelligence models for classifying relevant topics" through the call ”I Research Projects 2022", which originated this work. [11] M. Á. Álvarez-Carmona, Á. Díaz-Pacheco, R. Aranda, A. Y. Rodríguez-González, L. BustioMartínez, V. Muñiz-Sánchez, A. P. Pastor-López, F. Sánchez-Vega, Overview of rest-mex at iberlef 2023: Research on sentiment analysis task for mexican tourist texts, Procesamiento del Lenguaje Natural 71 (2023). [12] G. E. Batista, R. C. Prati, M. C. Monard, A study of the behavior of several methods for balancing machine learning training data, ACM SIGKDD explorations newsletter 6 (2004) 20–29. [13] Y. Ma, H. He, Imbalanced learning: foundations, algorithms, and applications (2013). [14] K. Church, P. Hanks, Word association norms, mutual information, and lexicography,
Computational linguistics 16 (1990) 22–29. [15] E. Grave, P. Bojanowski, P. Gupta, A. Joulin, T. Mikolov, Learning word vectors for 157 languages, arXiv preprint arXiv:1802.06893 (2018). [16] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805 (2018). [17] N. Sabharwal, A. Agrawal, N. Sabharwal, A. Agrawal, Bert algorithms explained, Hands-on Question Answering Systems with BERT: Applications in Neural Networks and Natural Language Processing (2021) 65–95.