The development of the role of the OTA (Online Travel Agent) site has become one of the E-WOM (Electronic Word of Mouth) media in addition to its main function as a platform for ticket reservations to encourage stakeholders in the hotel industry to utilize E-WOM for business continuity. One of the guest houses in Malang realized the importance of E-WOM because 90 percent of the booking process originated from the OTA website. However, the process of processing customer reviews only focuses on physical reviews, namely Guest Reviews. Meanwhile, information from online sources can have a more significant impact on E-WOM. One of the techniques of text mining is sentiment analysis which can be used to process and group text reviews. Sentiment analysis can be done to determine the sentiment of opinions on customer reviews to determine customer satisfaction with guest house services that aim to produce a positive E-WOM. Sentiment analysis is carried out at the aspect level using aspects of location, room, food, price, and service. The text of the review used in Indonesian originates from the sites Agoda.com, Expedia, Pegi-Pegi, Booking.Com, TripAdvisor and has a timeline from 2012 to 2019. This research yields findings in the form of customer satisfaction analysis of the five aspects where food aspects have urgency to be addressed and corrected immediately. Evaluation of the classification results also proves the effectiveness of the SVM method from Naïve Bayes
The rapid growth of technology has encouraged the development of the role of the OTA (Online Travel Agent)
site as one of the E-WOM (Electronic Word Of Mouth) media in addition to its main function as a platform for
ticket reservations.
One of the guest houses in Malang realized the importance of E-WOM in its business continuity because 90 percent of the booking process originated from the OTA website. Currently, the guest house has been listed on several OTA sites, namely TripAdvisor, Booking.com, Expedia, Agoda and Pegi-Pegi. However, the process of processing customer reviews only focuses on physical reviews, namely Guest Reviews. Meanwhile, information from online sources can have a more significant impact on E-WOM. The process of processing customer reviews becomes ineffective because it only focuses on one source, while the evaluation of guest house management services needs to be on target. In addition to these problems, the wide range of hotel attributes makes it difficult for stakeholders to determine aspects that have urgency to be addressed immediately.
A method that can be used to process and group text reviews is a sentiment analysis. Sentiment analysis or
opinion mining is a computational study of people's opinions, sentiments, and emotions through entities and
attributes that are expressed in text form
Machine Learning has an average accuracy that is much better above 80% and among the machine learning
classifier results Support Vector Machine has the best accuracy compared to other classifiers.
This study will discuss the use of sentiment analysis will be carried out at the selected aspect level to group
reviews into 5 aspects and determine the sentiment of customer reviews by applying stages in text mining using
machine learning classification methods. These five aspects were chosen based on research by
Sentiment analysis or opinion mining is an umbrella of branches of study such as opinion extraction, sentiment
mining, subjectivity analysis, affect analysis, emotion analysis, mining review, etc. Sentiment analysis is a field
of study that analyzes opinions, praise sentiments, one's emotions towards entities such as products, services,
organizations, events, problems, and attributes of entities
The SVM algorithm aims to find Maximum Marginal Hyperplane (MMH) using support vectors and margins.
MMH is the best hyperplane with the largest margin distance used to separate data maximally and accurately for
each class. Margin can be defined as the shortest distance of a hyperplane to one side of the margin is the same as
the hyperplane distance to the other side of the margin, provided that both margins are in a parallel position with
the hyperplane
If there is a dataset in the form (X1, y1), (X2, y2), (X3, y3), ... , (Xi, yi) where Xi is tuple training and yi is the class label with = 1 .... ∈ and ∈ {−1,1}. Every yi can choose one of two values either +1 or -1 SVM will form a classifier as shown in the Equation (1) as follows:
(!) = {("##,,%%!!"" )&'' (1) In SVM a hyperplane will be described in the following equation:
. + = 0 (2)
Based on equations (2), W is scalar Weight, n is attribute, b is scalar value or called bias, and X is training data set or training tuples. 106 2.3
Naive Bayes is an algorithm put forward by a British scientist Thomas Bayes which is a classification algorithm
using probability and statistical methods. This algorithm predicts opportunities based on past experience to be used
in the future so it is known as the Bayes Theorem
Based on Equation (4), Nc is number of class c in all documents, N is number of all documents. The n-word probability is determined using the laplacian smoothing technique:
( | ) = ((),))|)| (5)
In Equation (5), count (tn, c) is number of terms tn found in all training data with category c, count (c) is number of terms in all training data with category c, V is number of all templates in the training data 2.4
TF-IDF
TF-IDF Term Weighting is a weighting that is often used and is a combination of Term Frequency and Inverse
Document Frequency. TF-IDF consists of frequency terms and inverse documents obtained from dividing the total
number of documents to the number of documents that have these terms
!() = ! ∗ log /! (6) Based on Equation (6), !() is the weight of the term t in the document d, ! is frequency of occurrence of 8 is inverse document frequency value of the term t. the term t in document d, and log 9:! 2.5
Confusion Matrix contains information about the performance of a classification system that is evaluated using the data or metrics contained in the Confusion matrix. Confusion Matrix analyzes how well the classification has been done on the actual class and the predicted class.
Positive
Negative
Source:
= ;<) >;<<)) ;;==) >= Precision is the proportion of correctly identified labeling (Completeness), the formula for finding Precision: = ;<;)<;< (8) Recall is the proportion of information that can be found from a label, the search formula for Recall: = ;<;)<>=
Precision and Recall can be used to get the proportion of other measurements namely F1-Score. F1-Score is the harmonic mean of the calculation of Precision and Recall, the formula to find F1-Score:
The research phase begins by identifying the problem through an interview process with one of the stakeholders to determine the aspects needed in the Sentiment Analysis. Next, the data collection stage is carried out on the OTA (Online Travel Agent) website using webscraping tools to extract customer review data that will be used in the Sentiment Analysis process. Then the data design is done which includes the process of categorizing data and labeling data manually. The next stage is preprocessing text which aims to prepare data before entering the term weighting stage by using the NLTK and Literature modules in Python. Some of the stages in the preprocessing text are formalization and translation, cleansing, tokenizing, case folding, stemming, and stopword removal. Furthermore, the term weighting phase or term weighting will produce a review data that has word weight using the Sckit-learn module in Python. The sentiment classification stage requires two data: review data which already has labels and word weights. This stage is divided into two parts, where the first stage aims to choose the best model that will be used in the second stage, namely the classification of sentiments for each aspect. Model selection is done by applying Stratified k-Fold Cross Validation and classification is done by classifier machine learning. Finally, the analysis and evaluation of the classification results are carried out using F1-Score, Precision, and Recall metrics and the Accuracy calculation. Evaluation is done by utilizing the Scikit-learn module in the Python Programming Language. See Figure 2 for details.
This study uses a webscraping method with webcraper.io tools to extract variables containing information for
sentiment analysis. The data used is the text of customer reviews on the TripAdvisor, Booking.com, Expedia,
Agoda and Pegi-Pegi sites. Customer reviews that are used in Indonesian and have a period of 2012-2019. Total
data collected was 1,561 consisting of 435 texts from the Agoda.com site, 37 texts from the Expedia site, 27 texts
from the Pegi-Pegi site, 235 texts from the Booking.Com site, and 827 texts from the TripAdvisor site. The
example of customer review can be seen in Figure 3. After that, the data that has been collected will be labeled
manually based on the five aspects that have been selected namely location, room, food, price, and service as well
as the polarity of sentiment aspects that are positive and negative. Data labeling is based on guidelines in
Negative hotel does not meet customer expectations regarding the services they have hotel cannot set prices according to the value obtained hotel cannot fulfill matters related to hotel room conditions hotel cannot meet the customer's expectations regarding the food and drinks served hotel cannot meet the customer's expectations regarding the condition of the location of the hotel and its surroundings Text Preprocessing is a step taken to prepare data before being analyzed in the sentiment classification process. The implementation of Text Preprocessing uses text variables of customer reviews on Kertanegara Premium Guest House through the NLTK and Sastrawi modules in Python. Formalization is the stage of changing words into standard forms in accordance with KBBI and Translation is the stage for translating words from foreign languages into Indonesian. Both stages are carried out manually by adjusting the KBBI Standard Form. Cleansing aims to eliminate elements that are not needed in the sentiment analysis process. These elements consist of punctuation, numbers, html tags. Tokenizing aims to separate the text or sentence review into pieces of words. Case Folding aims to change the words produced in the tokenizing process into lowercase characters. Stemming aims to eliminate word affixes in the customer review text so that the basic words are obtained. Stopword Removal aims to eliminate words that have little effect on sentiment analysis or words contained in a stopword list. Furthermore, the data enters into the Term Weighting phase which aims to provide weights that measure how important the value of a term is to the document. This study uses TF-IDF (Term Frequency-Inverse Document Frequency) which will display the word weight which increases according to the appearance of the word and IDF calculation produces a weight corresponding to the level of uniqueness of the word in the word index. This stage utilizes the Scikitlearn module in Python. Table 3 shows a representation customer reviews before and after the text preprocessing process. Sentiment classification uses review text that has labels and weights. There are 2 stages in the Classification phase, namely the stage of model selection and the stage of classification model implementation. The model selection phase aims to choose the model that will produce the best accuracy. This stage uses K-Fold Cross Validation to limit the problem of overfitting data in compiling data into training and testing, there is a single variance or overlap in the distribution of training data and testing data so that the model does not lose significant data for the modeling and testing process. The k value used in this stage is 6 so the dataset is divided into 6 parts. Next, the model implementation stage uses SVM and Naïve Bayes classification algorithms. All stages in classification utilize the Scikit-learn module at Python.
The highest number of reviews is in the aspect of rooms with a value of 1070 as shown in Table 4. This shows that the customer is very concerned about the condition and quality of rooms provided by the Guest House. Room cleanliness, amenities, bathroom conditions are things that are often discussed by customers during the stay. Comparison of positive and negative sentiment can be seen in Figure 4 and 5, respectively. The food aspect has the biggest ratio between positive and negative reviews. This shows that this aspect has urgency to be addressed immediately. The results of a trend analysis of the five aspects also show that food aspects have not been addressed optimally with increasing graphs in negative sentiment in 2019. Meanwhile, the other four aspects experienced an increase in the number of positive reviews in 2019. In 2013 and 2017 the highest average for negative sentiment. This shows that stakeholders have reformed and improved the management of the guest house so that the graph of positive sentiment increases rapidly in 2019 except in the aspect of food. The variety and taste of food that does not meet the standards of a guest house is often complained by customers.
The study also produced a comparison between the two classification algorithms, SVM and Naïve Bayes, which uses TF-IDF term weighting. The classification algorithm testing utilizes the classification report function in the Python Sckit-learn. Table 5 shows that overall SVM has a better Accuracy value than Naïve Bayes on all five aspects. This proves that SVM has better effectiveness than Naïve Bayes. The value of Precision, Recall and F1-Score almost reached 70 percent in all aspects except the food aspect. This can be caused by the process of labeling data review manually or an unbalanced dataset comparison.
This study discusses the use of sentiment analysis at the aspect level on customer reviews as a guest house that aims to assist stakeholders in improving and improving management evaluations. Sentiment analysis conducted on the five aspects resulted in the finding that the food aspect had an urgency to be immediately addressed by the stakeholders. Evaluation of sentiment classification results in the SVM class of Accuracy, Precision, Recall, and F1-Score better than Naive Bayes which proves that SVM has better effectiveness. From this study it can be concluded that the guest house customers are satisfied with the management and accommodation of the guest house, but it is good for the guest house to continue to monitor and improve its services to all aspects, especially food aspects to produce a positive E-WOM.