Bakso President Malang highly appreciate the opinions of customers regarding the products and services they provide to increase customer satisfaction. However, Bakso President Malang does not have customer opinion data, nor does the technology in processing and analyzing data that can produce information about customer perspectives on customer satisfaction aspects. To get the customer perspective of Bakso President Malang on customer satisfaction aspects can be done through Sentiment Analysis at the aspect level. In this study, sentiment analysis is performed on customer review using algorithms namely, Multinomial Naïve Bayes Classifier and Linear Support Vector Machine. This study uses 2,252 customer review data from 2012 to 2019. Customer review data is obtained through Web Scraping techniques on the TripAdvisor and Google Review sites. Experimental results shows that Linear Support Vector Machine has better accuracy performance compared to Multinomial Naïve Bayes Classifier.
Restaurants owner runs their business not only consider about strategies that prioritize products, but they should implement strategies that could appraise what is needed by customer. This should be done in order to create a good relationship between customers and restaurants. Therefore, restaurant owners need to pay attention to customer review regarding the products and services that they have been provided. Bakso President Malang already has a lot of customer reviews on sites like TripAdvisor and Google Reviews. The problem faced by Malang President's Meatballs is that it has not been able to extract information from the many customer reviews that are expected to improve service and product development. The existence of customer reviews will also have an impact on competitive advantages and the brand image that will be experienced by Bakso President Malang.
Based on the problems faced by Bakso President, one alternative that can be offered is through an analysis of
data related to customer opinions obtained from websites that provide customer reviews. Customer review data
are then to be processed using sentiment analysis to get customer perspective related to several aspects of customer
satisfaction. Customer satisfaction is obtained from the food quality, price, ambiance, and service quality that
provided by the restaurant
Previous research conducted by
Based on the background that has been described and some research that has been done before, the authors are interested in conducting sentiment analysis study. The sentiment is gathered from TripAdvisor and Google Review sites related to customer satisfaction aspects of the Bakso President Malang. Sentiment analysis is carried out at the aspect level using two classifier, the NBC and SVM. In this study NBC and SVM is chosen as the classifiers because SVM works on high dimension data and NB classifier is a group of basic probabilistic classifiers dependent on a typical assumption that all features are free of one another, given the class variable, and it is regularly utilized as the pattern in text classification. Each of the classifier is combined with TF-IDF as Term Weighting. This research will find out which methods have better performance to be implemented in sentiment analysis. The paper is distributed in 6 different sections. Section 2 discuss about the related studies. Section 3 describe about the methodology of the proposed work, dataset and techniques used. Section 4 discuss about the experiment results that performed in this paper. Section 5 describe the conclusion and future scope of the paper.
According to Urru and
Sentiment analysis or commonly referred as opinion mining is a field of study that analyzes an opinion, sentiment,
or someone's judgment regarding an entity such as a product, service, organization, individual, problem, event,
and topic. Unlike factual information, opinions and sentiments have characteristics that indicate that it is
subjective. An examination of a group of opinions from many parties is needed because it is a subjective view that
comes from more than one person so a summary is needed to represent an opinion
The method used for text classification in sentiment analysis is Naïve Bayes Classifier. Multinomial Naive Bayes
Classifier is a classification method of Bayes algorithm that can be used in the classification of text or documents
) = ( ) ( 1| ) ( 2| ) ( 3| ) … ( | )
tn is a word document d to-n.
( ) = ( | ) = ( , )+1 ( )+| | (1) (2) (3)
count(tn,c) is a number of terms found in all training data with c category count(c) is a number of terms in all training data with c category | | is a number of terms in all training data 2.4
Han et al. (2012) defined Support Vector Machine is a classification method for finding the best separator to create maximum separation for all classes or Maximum Marginal Hyperplane (MMH). Margin could be defined as the shortest distance from a hyperplane to one side of margin is same as distance of a hyperplane to the other side of margin, note that both margins are in a parallel position with the hyperplane. As shown in Figure 1. Then it can be concluded that the greater the margin can be said to have higher accuracy.
SVM uses training dataset in the form (Xi yi), where Xi is a tuple and yi is class label with I = 1...N, Xi ∈Rd and yi ∈{−1, 1}. The purpose of SVM is to be able to form a classifier like Equation 4 below. to form a hyperplane explained in the following equation,
2.5
W b X = Vector weight{W1, W2, W3,…,Wn}, n is the number of attributes = Scalar, or commonly referred as bias = training tuples
≥0, = +1 ( ) = {<0, = −1 . +
= 0 Confusion matrix is a method that can calculate the accuracy of data mining. The results of the Confusion matrix are accuracy, precision, and recall values. Accuracy is the level of closeness between the predicted value and the actual value. Meanwhile, Precision is the level of accuracy between the information requested by users with the answers provided by the system. In addition to accuracy and precision, there is Recall which is the success rate of the system in rediscovering information. (5) (6)
Precision = + Recall = + F1-Score = 2 ∗ ∗ + (7) (8) (9)
At this stage, observations were made at the Malang Bakso President restaurant and direct interviews with the Bakso President Malang manager to find out the business processes. This step also explores the problems that exist in the Bakso President related to extracting information in customer reviews. 3.2
This step is searching for the right theory and research that has been studied previously that have similar problems domain to search for solutions carefully. References are obtained from the internet through books, journals, and credible sites about Text Mining, Sentiment Analysis, Naive Bayes Classifier, and Food Service Dimension, and
3.3
Data collection was conducted on reviews given by customers of Malang Bakso President Malang restaurant objects. Data sourced from the TripAdvisor site and Google Review using Web Scraping techniques. The tool used is Selenium Python to retrieve data in Google Review and Chrome extension, webscraper.io, for the
TripAdvisor website. The amount of data successfully scraped from TripAdvisor was 943 review data and from Google Review 1,209 review data. In total there are 2,252 comments on Bakso President by the customers gathered from 2012 to 2019. 3.4
Aspect categorization is useful for knowing what aspects are present in each customer review. The comments organized into related aspects. After that, each of the comment proceeded with labeling sentiment data whether positive or negative. The sentiment labeling is done manually. To minimize the subjectivity, the labeled sentiment is checked by the owner to validate the sentiment labeling precision. 3.5
Text Preprocessing is a phase that is carried out before doing Term Weighting and sentiment classification. Preprocessing stages will be carried out using standard text processing. The steps are Formalization and Translation, Remove Punctuation, Remove Number, Tokenization, Case Folding, Stop-word, Stemming, and
Formalization and Translation: Formalization is the process of changing words that were not standard into standard words in accordance with the Kamus Besar Bahasa Indonesia (KBBI). This process is done manually from scraping and then comparing it to the Indonesian dictionary. Translation aims to translate words from foreign languages and regional languages into words that are in accordance with the Kamus Besar Bahasa Indonesia (KBBI).
Remove Punctuation: Remove punctuation is the process of removing punctuation in data. This process uses the string library module in Python.
Remove Number: Remove number is the process of clearing data from numeric characters. This process uses the string library module in Python.
Tokenization and Case Folding: Tokenization is the stage to separate sentences into parts of words called tokens. The implementation of the toke
Stopwords: Stopword is the step of eliminating words that are included in stopword categories such as ‘di’, ‘pada’, ‘dan’, ‘yang’, ‘hanya’, ‘tidak’ and other words that are included in the stopwords list library.
Stemming: Stemming is a process of reducing a word to its word stem that affixes to suffixes and prefixes or to the roots of words.
Cleansing: Cleansing is a process to remove whitespace or empty tokens. The data used in this process is the result of the previous stemming process. 3.6
Term weighting is done by applying Term Frequency-Inverse Document Frequency (TF-IDF). Weighting is done by utilizing a module in the Python Programming Language with the Scikit-learn module to weight the TF-IDF.
3.7
In the sentiment classification stage, two documents are needed, namely the preprocessing document that has been given a sentiment class label and Term Weighting from the previous stage. The sentiment class label is divided into positive and negative. This sentiment classification is carried out on every aspect namely Food, Service, Price, and Atmosphere aspects. Model selection is done in order to get the highest accuracy and the best data model for each data. The selection of the model in this study uses Stratified Cross Validation, which is to divide the dataset randomly with a balanced amount and prevent multiples or different amounts of data. The dataset is divided into 2 folds, so the ratio of training data and test data is 50:50. After that the test data and training data will be stored into four variables namely X_train, Y_train, X_test and Y_test. The variables X_train and X_test are variables that contain term weighting data which are divided into training data and test data. While the Y_train and Y_test variables are variables that store label data. The next step is the sentiment classification used by the MultinomialNB module to implement the Multinomial Naïve Bayes and LinearSVC module to implement Support Vector Machine algorithm as a classifier. 3.8
After obtaining the results of sentiment classification, the next phase is testing with 4 parameters, namely Accuracy, Precision, Recall, and F1-Score. This process uses the library from Scikit-learn by using the classification_report module. In this test also used a confusion matrix to see the comparison of the predicted amount of data. 3.9
The final step in this research is to draw conclusions based on the formulation of the problems that have been obtained and focus on the results of sentiment analysis and provide suggestions for further research. The food aspect defines the quality of food and drinks from Bakso President, both in terms of taste, presentation, texture, portions, and also menu variations. The positive sentiment class on food aspect shows customer reviews which give positive responses in the form of comments stating customer satisfaction with food and beverage quality from Bakso President Malang. While in the negative sentiment class on this aspect of food shows negative responses such as disappointment to the quality of food and drinks from Bakso President Malang. The total number of customer reviews for food aspects is 1,927 reviews, with each number for positive sentiment classes 1,702 reviews and negative sentiment classes is 225 reviews.
Service Aspect shows customer reviews that describe customer satisfaction with service quality from Bakso President Malang. The service in question is employee performance such as employee attitudes and behavior in treating customers. In this aspect not only shows how customers respond to Malang President's Meatballs employees, but also about the condition of the cleanliness of food equipment, the ordering process, to the serving of food. The positive sentiment class on the service aspect shows customer reviews that give positive responses in the form of comments that indicate customer satisfaction with service quality from Bakso President Malang. Whereas in the negative sentiment class on this aspect of food shows negative responses such as disappointment to the quality of service from Bakso President Malang. The number of customer review data on service aspects with negative sentiment was 383 reviews. Whereas for customer review data on service aspects with positive sentiment was 114 reviews.
The price aspect shows the customer's response regarding the price set by the Bakso President Malang. In a positive sentiment class, customer reviews contain opinions that illustrate that the price set by the Bakso President Malang is in accordance with or what has been obtained by customers. In addition, customer reviews that have positive sentiment on this aspect also stated that the price set by Bakso President was cheap or still affordable. As for customer reviews that have negative sentiment is when the customer's response to the price set by Bakso President Malang is not in accordance with what is obtained by the customer and is expensive. The number of customer review data on the price aspect with negative sentiment of 170 reviews. As for the customer review data on the price aspect with positive sentiment is 217 reviews.
This aspect of the atmosphere explains the atmosphere or environment of the President's Meatballs that are felt by customers. Customer reviews in this aspect talk about Malang Bakso President's location conditions. In a review
0.33 0.89 0.78 0.40 0.68 0.88 0.30 0.85 1.00 0.89 0.80 0.90 1.00 0.89 1.00 0.89 0.07 0.98 0.97 0.07 0.88 0.68 0.04 0.98 0.05 1.00 0.11 1.00 0.05 1.00 0.08 1.00 0.12 0.93 0.86 0.12 0.77 0.77 0.06 0.91 0.10 0.94 0.19 0.94 0.10 0.94 0.15 0.94 of customers who have positive sentiments stated that the Bakso President Malang environment is unique because it is next to the railroad tracks so that customers can feel a pleasant atmosphere. While in customer reviews who have negative sentiments stated that the Bakso President environment which is next to the railroad tracks is not good so it is dusty and dangerous. The total number of customer reviews for Atmosphere aspects is 1,083 reviews, with each number for positive sentiment classes 915 reviews and negative sentiment classes 168 reviews. 4.2
This section describes the performance report and discussion of the sentiment analysis classification algorithms on the customer reviews of Bakso President Malang. Table 4 shows testing results using Multinomial Naïve Bayes in each aspect. Table 5 shows testing results using Support Vector Machine in each aspect.
Sentiment analysis with Linear Support Vector Machines method showed better performances than Multinomial Naïve Bayes. The accuracy of every aspect using Linear SVM shows the same value, which is 89%. While the use of the MNB method shows different accuracy values for each aspect. The highest accuracy value in MNB is in the aspect of food that is 88% which has a slight difference compared to Linear SVM. One of the factors that influence a classification to get good accuracy results, can be seen from how man y combinations of training data and test data generated in the selection of models. The more combination of training data, the better the classification results will be produced because if more learning models are used in classifying training data, the better the classification of testing data in recognizing existing patterns (Han et al., 2012). Thus, if the distribution of training data and testing data uses more than 2-fold with a comparison of training data of more than 50%, it is possible to get better accuracy results.
From the discussion that has been presented based on the research that has been done, some conclusions are obtained. Web scraping techniques can be used to obtain customer review data needed to conduct sentiment analysis on the aspect level of the TripAdvisor and Google Review sites. The total data obtained from the implementation of Web Scraping in the form of customer review data on Bakso President Malang is 2,152 reviews with a time span from 2012 to 2019. The results of the application of the research conducted using the Naïve Bayes Classifier and Support Vector Machine method and Term Frequency-Inverse Document Frequency as Term Weighting can be used as an option in solving problems in the classification analysis of aspects of sentiment level for assessing customer satisfaction of Bakso President Malang. Algorithms that shows better performances from accuracy results is Support Vector Machine.
Future work is expected to use automated methods that are not manual in labeling data and extracting aspects, such as CRF (Conditional Random Field), LDA (Latent Dirichlet Allocation) combined with feature extraction like POS-Tagging, unigram, or bigram. The purpose of using those method so the labeling of 1 opinion is not subjective.