Extracting useful information for sentiment analysis and classification problems from a big amount of user-generated feedback, such as restaurant reviews, is a crucial task of natural language processing, which is not only for customer satisfaction where it can give personalized services, but can also influence the further development of a company. In this paper, we present a work done on collecting restaurant reviews data as a sentiment analysis dataset for the Uzbek language, a member of the Turkic family which is heavily afected by the low-resource constraint, and provide some further analysis of the novel dataset by evaluation using diferent techniques, from logistic regression based models, to support vector machines, and even deep learning models, such as recurrent neural networks, as well as convolutional neural networks. The paper includes detailed information on how the data was collected, how it was pre-processed for better quality optimization, as well as experimental setups for the evaluation process. The overall evaluation results indicate that by performing pre-processing steps, such as stemming for agglutinative languages, the system yields better results, eventually achieving 91% accuracy result in the best performing model.
The power of Natural Language Processing (NLP) techniques relies on amounts of labelled data
in many applications. Sentiment analysis is the process of analyzing and labelling the opinion
which is posted by consumers. Consumers usually post their feedback about places/foods to
famous applications such as Google Maps 1, Yelp2, etc). They often encourage consumers to
actively participate in reviews, and massive user-generated restaurant reviews allow consumers
to fully express their needs while helping merchants provide real-time and personalized service
[
3https://en.wikipedia.org/wiki/Uzbek_language 4https://fasttext.cc/docs/en/crawl-vectors.html 5https://github.com/SanatbekMatlatipov/restauranat-sentiment/tree/main
In recent years, several works were done in the NLP field for Uzbek, including sentiment analysis datasets [10, 11], created by collecting and analyzing Google Play app reviews, with two types of data: A medium-size manually annotated dataset and a larger-size dataset automatically translated from English. [12] obtained bilingual dictionaries for six Turkic languages and applied them to cross-lingually align word embeddings, backed by a bilingual dictionary induction evaluation task. They showed that obtained aligned word embeddings from a low-resource language can benefit from resource-rich closely-related languages. Another similar paper [13] investigated the efect of emoji-based features in opinion classification of Uzbek texts. A semantic evaluation dataset was presented with semantic similarity and relatedness scores in word pairs as well as its analysis for Uzbek in a recent work [14]. There is a very recent growing trend in NLP that makes use of AI-based techniques, which can be seen in the work on Uzbek with neural transformers-architecture based language model trained of raw Uzbek corpus [ 15].
In a global outlook to the field of sentiment analysis, there is a work [ 16] that used various methods of sentiment analysis techniques, such as machine learning and deep learning, in their work with an idea to take into account the diferences in opinions and thoughts that exist on popular social platforms such as Twitter, Reddit, Tumblr and Facebook.
In this paper, we proposed a machine learning and deep learning-based sentiment analysis framework for the restaurant domain dataset (Figure 1). The framework includes data collection using web-crawler, pre-processing(cleaning, stopwords, lexicon-free stemming), constructing TF-IDF weight matrix, performing ML and DL for sentiment analysis;
We start by looking at a high number of the dataset available for crawling in the Uzbek language. However, the usual approaches such as Twitter or movie reviews are not the case for Uzbek. Therefore, we decided to collect restaurant reviews as local people mostly loved giving feedback which is restaurants. we think it makes sense as Uzbek cuisines are one of the most popular throughout the Commonwealth of Independent States (CIS, CA countries). In most CA cities, for instance, it’s easy to find busy restaurants specializing in Uzbek cuisine 6. We crawled all local restaurants in Tashkent from Google Maps. Firstly, we selected a list of more than 140 URLs which has at least 3 reviews and we retrieved all info shown in Figure 2. While crawling, we considered Google’s anti-spam and anti-DDOS policies as there are certain limitations on harvesting data. The source code is available on the repository.
The collection of texts with star ratings in the crawled dataset was noisy and required manual correction. The comments containing only emojis, names or any other irrelevant content, such as username mentions, URLs or specific app names were removed. Those written in languages diferent from Uzbek (mostly in Russian and some in English) were translated using the oficial Google translate API. Although people in Uzbekistan use the oficial Latin alphabet, the use of the old Cyrillic alphabet is equally popular, especially among adults. The comments that were written in Cyrillic were converted to Latin using the Uzbek machine transliteration tool [17]. Then, we applied stop words to remove low-level information words from our comments to focus on important information. The technique is based on [8] paper where it is a proposed algorithm of automatic detection of single word stop words collection using TFIDF(Term frequency inverse document frequency). After that, each word is processed to lexicon-free stemming tool [7] algorithm for decreasing the word capacity because of prefixes and sufixes. The basic idea is using the combinatorial approach of eligible endings candidates. Following table 1 shows processed data which is ready for TFIDF-vectorizer.
We selected a set of words to visualize the word count. Figure 3 shows that people tend to give more positive feedback than negative on the domain of restaurants.
6BBc Travel: https://www.bbc.com/travel/article/20191117-is-uzbek-cuisine-actually-to-die-for
Review Birinchi Milliy taomlardan biri - keng assortimentli taomlar! Gastro-turistlar uchun juda jozibali joy - bu yerda barcha turdagi milliy taomlar mavjud.Yagona salbiy tomoni shundaki, bunday yirik muassasa uchun to’xtash joyi kichik. Narxlar nisbatan arzon! Turistlar uchun juda arzon!
After processing Bir/ milliy/ taom/ keng/ assortiment/ taom/ gastro/ turist/ juda/ joziba/ joy/ tur/ milliy/ taom/ mavjud/ salbiy/ tomon/ yirik/ muassa/ to’xta/ joy/ kichik/ narx/ arzon/ turist/ arzon
The collected novel dataset has been split into training and testing subsets for evaluation with 8 x 2 ratio respectively. After the data cleaning process, we have the original dataset as follows, where ⃗ represents feature vectors and ⃗ represents annotated labels: (⃗, ),
= 1, 2, 3, ..., ⃗ = (1, 2, ..., ) and is equal to the number of reviews and length of the feature vector, respectively.
Then we calculate TFIDF scores for each feature vector ⃗ which vectorises words by taking into account the frequency of a word in a given review and the frequency between reviews. The final result of all ⃗s is defined as a sparse matrix.
⃗ = () ()
The \Logistic regression model is (|⃗) = ℎ(⃗) = 1/(1 + exp(− )) {︃ℎ(⃗),
if = +1() 1 − ℎ(⃗), if = − 1() Logistic regression[18] model is a classification algorithm, known for its exponential and loglinear functions. It works with discrete values and maps the function of any real value into 0 and 1. For sentiment analysis, the hypothesis shows, reviews are either positive or negative by using the (4). The Support Vector Machine(SVM) model has the following response function: ℎ(⃗) = (⃗) SVM algorithm is known for its fast and dependable classification which resolves two-group classification problems. The classification is conducted for finding a hyperplane between two classes’ positive and negative reviews in the model: After all, we implemented LR and SVM models utilizing the Scikit-Learn [19] machine learning library in Python with default configuration parameters. For the LR models, we implemented a variant based on word n-grams (unigrams and bigrams), and one with character n-grams (with ranging from 1 to 4). We also tested a model combining said word and character n-gram features.
Keras [20] is used on top of TensorFlow [21].The FastText pre-trained word embeddings of size 300 [22] for the Uzbek language are applied. For the CNN model, we used a multi-channel CNN with 256 filters and three parallel channels with kernel sizes of 2,3 and 5, and drop out of 0.3. The output of the hidden layer is the concatenation of the max-pooling of the three channels. For RNN, we use a bidirectional network of 100 GRUs. The output of the hidden layer is the concatenation of the average and max-pooling of the hidden states. For the combination of deep learning models, we stacked the CNN on top of the GRU. In the three cases, the final output is obtained through a sigmoid activation function [23] applied to the previous layer. In all cases, Adam optimization algorithm [24], an extension of stochastic gradient descent, was chosen for training, with standard parameters: learning rate = 0.0001 and exponential decay rates 1 = 0.9 and 2 = 0.999. Binary cross-entropy was used as a loss function. The same steps, but slightly diferent parameters were used in a work that presents guidance to use CNN for sentiment classification [ 25]. Inspired by their example that perfectly illustrates the steps of performing deep learning based sentiment classification using CNN, the visualisation of our steps can be seen in Figure 4. (3) (4) (5)
Confusion[26] matrices are used in the task to determine the gap between predicted and true values which is shown in Table 2. Precision, Recall and F1-score are used as evaluation metrics for model performance. The calculation of Precision and Recall is shown below: =
+ =
+ The F1-Score is used, which takes into account both accuracy and recall, and the F1-Score is calculated as follows: 1 = 2 * * + (6) (7)
This section presents a detailed description of the results obtained by the evaluation process using both machine learning and deep learning techniques applied to the collected novel sentiment analysis dataset of restaurant reviews.
The overall experiment results of the above-mentioned evaluation were performed, and the results can be seen in Table 3.
The Logistic Regression(LR) based on word n-grams obtained a binary classification accuracy of 90% on the dataset, while the one based on character n-grams, with its better handling of misspelt words, improved it to 91%(which is the winner of this comparison). Support Vector machines based on Linear kernel mode have shown 88% accuracy overall. Recurrent Neural network models without and with fastText embedding show the same accuracy (88%). Convectional Neural Network showed slightly less performance(89.23%) than LR. However, this is the reason for lacking data for neural-network models, as it requires big data for better performance.
Nowadays, unstructured data are becoming more and more in the restaurant domain which requires performing high accuracy sentiment analysis. Especially, this is the case for lowresource languages. Based on the review data of Google Maps(Tashkent location) which is obtained by web-crawling, the paper has shown several ML& DL methods. It was observed that the LR algorithm outperforms the others which makes sense as our dataset is relatively small. The research also mentioned some theoretical and practical implications. We believe, in terms of gaining massive user reviews on the domain can provide consumers make their decision in the best manner such as lower cost and faster speed. However, we also wanted to point out some limitations in this research paper. The dataset we gathered has an unbalanced number of positive and negative reviews, which can cause deviations in the result. Moreover, we used the review rating in the annotation process which sometimes, in reality, consumers may give a high rating score, but polarity context may be related to negative, and vice versa.
In this paper, we have shown a novel dataset in the restaurant domain for the Uzbek language, with 8210 reviews, annotated with positive or negative labels, which is crawled from Google Maps using URLs of all locations in the capital city Tashkent, and was labelled as their corresponding star score. Then, we applied full pre-processing steps to the dataset which contributed to increasing the accuracy of our baseline models. Further analysis of the collected dataset was shown with evaluations using both machine learning and deep learning techniques. The best accuracy result (91%) on the dataset was obtained using a logistic regression model with word and character n-grams.
In the foreseen future, we are planning to extend the work by collecting more data, which can efectively analyze the restaurant reviews in a practical level. Also, the work is underway to remove the evaluation bias of the training experiments by using cross-validation methods in data splitting.
This work partially has received funding from ERDF/MICINN-AEI (SCANNER-UDC, PID2020113230RB-C21), and from Centro de Investigación de Galicia ”CITIC”, funded by Xunta de Galicia and the European Union (ERDF - Galicia 2014-2020 Program), by grant ED431G 2019/01. Elmurod Kuriyozov was funded for his PhD by El-Yurt-Umidi Foundation under the Cabinet of Ministers of the Republic of Uzbekistan. [4] L. Zhang, S. Wang, B. Liu, Deep learning for sentiment analysis: A survey, Wiley Interdiscip. Rev. Data Min. Knowl. Discov. 8 (2018). URL: https://doi.org/10.1002/widm.1253. doi:10.1002/widm.1253. [5] M. Artetxe, I. Aldabe, R. Agerri, O. Perez-de Viñaspre, A. Soroa, Does corpus quality really matter for low-resource languages?, 2022. URL: https://arxiv.org/abs/2203.08111. doi:10.48550/ARXIV.2203.08111. [6] G. Matlatipov, Z. Vetulani, Representation of Uzbek morphology in prolog, in: Aspects of Natural Language Processing. Lecture Notes in Computer Science, volume 5070, Springer, 2009. [7] S. Matlatipov, U. Tukeyev, M. Aripov, Towards the uzbek language endings as a language resource, in: M. Hernes, K. Wojtkiewicz, E. Szczerbicki (Eds.), Advances in Computational Collective Intelligence, Springer International Publishing, Cham, 2020, pp. 729–740. [8] K. Madatov, S. Bekchanov, J. Vičič, Automatic detection of stop words for texts in the uzbek language, 2022. [9] U. Tukeyev, A. Turganbayeva, B. Abduali, D. Rakhimova, D. Amirova, A. Karibayeva, Lexicon-free stemming for kazakh language information retrieval, in: 2018 IEEE 12th International Conference on Application of Information and Communication Technologies (AICT), 2018, pp. 1–4. doi:10.1109/ICAICT.2018.8747021. [10] I. Rabbimov, S. Kobilov, I. Mporas, Opinion classification via word and emoji embedding models with lstm, in: International Conference on Speech and Computer, Springer, 2021, pp. 589–601. [11] E. Kuriyozov, S. Matlatipov, Building a new sentiment analysis dataset for uzbek language and creating baseline models, in: Multidisciplinary Digital Publishing Institute Proceedings, volume 21, 2019, p. 37. [12] E. Kuriyozov, Y. Doval, C. Gómez-Rodríguez, Cross-lingual word embeddings for Turkic languages, in: Proceedings of the 12th Language Resources and Evaluation Conference, European Language Resources Association, Marseille, France, 2020, pp. 4054–4062. URL: https://aclanthology.org/2020.lrec-1.499. [13] I. Rabbimov, I. Mporas, V. Simaki, S. Kobilov, Investigating the efect of emoji in opinion classification of uzbek movie review comments, in: A. Karpov, R. Potapova (Eds.), Speech and Computer, Springer International Publishing, Cham, 2020, pp. 435–445. [14] U. Salaev, E. Kuriyozov, C. Gómez-Rodríguez, Simreluz: Similarity and relatedness scores as a semantic evaluation dataset for uzbek language, arXiv preprint arXiv:2205.06072 (2022). [15] B. Mansurov, A. Mansurov, Uzbert: pretraining a bert model for uzbek, arXiv preprint arXiv:2108.09814 (2021). [16] Y. Chandra, A. Jana, Sentiment analysis using machine learning and deep learning, in: 2020 7th International Conference on Computing for Sustainable Global Development (INDIACom), 2020, pp. 1–4. doi:10.23919/INDIACom49435.2020.9083703. [17] U. Salaev, E. Kuriyozov, C. Gómez-Rodríguez, A machine transliteration tool between uzbek alphabets, arXiv preprint arXiv:2205.09578 (2022). [18] E. Christodoulou, J. Ma, G. S. Collins, E. W. Steyerberg, J. Y. Verbakel, B. Van Calster, A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models, Journal of Clinical Epidemiology 110 (2019) 12–22.