Nowadays, there are a large number of users who post complaints about a certain service on the Internet. Because users have various values and views, even if they receive the same service, they may complain in diferent ways. However, it is quite dificult to respond to various user demands for service in real time and there are almost no direct solutions when users feel dissatisfied with a certain service. Therefore, in this paper, we propose an advice recommender system by analyzing complaint data from Fuman Kaitori Center. First, the system generates query keywords according to various user complaints about a certain service by calculating the score of each query. Then suitable web pages containing advice are recommended from the results of the query. This advice could address users' dissatisfaction and respond to their various demands in a comprehensive way. Also, we verify the usability of proposed system by using a questionnaire survey evaluation.
• Information System → Information Retrieval; • Query Processing → Query Suggestion.
In recent years, many users post negative reviews about a certain
service online. However, it is quite dificult to respond to various
user demands for service in real time as the service is provided by
the company. In addition, there are almost no direct solutions when
users feel dissatisfied with a certain service. Therefore, this paper
is focused on user complaints related to services and proposed a
system to search for advice that could address users’ dissatisfaction
by generating query keywords from complaint reviews[
The remainder of this paper is structured as follows. Section 2 presents a brief summary of related work. Section 3 introduces the dataset we use for research and explains the proposed system. Section 4 discusses the experimental results and the evaluation of the proposed system. Finally, Section 6 concludes this paper and discusses future work.
The FKC dataset has been used for several studies in recent years.
Mitsuzawa et al. [
Hasegawa et al. [
Sakai et al. [
Hashimoto et al. [
Yoshida et al. [
Song et al. [
Hayashi et al. [
In this paper, we propose a system by analyzing complaint data from Fuman Kaitori Center for recommendation of advice in order to address users’ dissatisfaction about a certain service. Figure2 shows the system flow of our proposed method. First, we extract the company name and complaint topic words by calculating the importance of the nouns in the negative reviews. Second, we obtain candidate search keywords of these extracted words. Then, the system extracts the advice topic word by calculating the relevancy of the candidate keywords to the FKC dataset and score them using morphological and polarity analyses. Third, we create the query by combining the company name, complaint topic word, and advice topic word according to their various complaints about a certain service. Finally, suitable web pages containing advice that could address a user’s complaint are recommended from the results of the query. 3.2
In this study, we analyze a dataset of complaints from the Fuman Kaitori Center, which is provided by Insight Tech Inc. from the National Institute of Informatics. In this paper, we refer to the Fuman Kaitori Center’s dataset as the FKC dataset. The Fuman Kaitori Center is a website on which users can post their complaints about topics such as products, services, education, work, and relationships. Moreover, users get points when they post complaints that they can exchange for coupons for online shopping websites. This dataset contains about 5 million negative reviews that were posted from 18 March 2015 to 12 March 2017 by around 100,000 users. Each negative review contains the information shown in Table 1. In FKC dataset, each category contains several subcategories, and each subcategory contains several companies.In this paper, because we focus on user service complaints, our proposed system uses the data fields for “company" and “text" .
Our proposed system extracts company names from FKC dataset directly from the company field of each record. Next, we extract the complaint topic word by analyzing negative reviews. In this paper, we only use the negative reviews that are labeled with the company name. To extract complaint topic words, we first extract all companies’ negative reviews for one subcategory and extract all nouns from the negative reviews. Next, we calculate the importance of each noun using the following equation.
t f t f |A| × Íd ∈D t fd (1)
Here, t f is defined as the number of occurrences of a particular noun in the complaints for a certain company, |A|is defined as the number of all nouns in the complaints for a certain company, and Íd ∈D t fd is defined as the number of occurrences of certain noun for the complaints for all companies. Finally, we extract all nouns whose importance values are above the determined threshold value and define them as that company’s complaint topic words. 3.4
To extract the advice topic word, we first obtain candidate search keywords of the company name and complaint topic word. Because the FKC dataset is full of negative reviews, we hypothesize that candidate keyword that are less relevant to the FKC dataset will make better advice topic words. To verify this hypothesis, we calculated the relevance of these candidate keywords for each company and each complaint topic word and define as “company relevancy” and “complaint topic relevancy”. It is calculated using the following equation.
Here, Rcd is defined as the number of occurrences of certain candidate keyword in complaints for the company in the FKC dataset and Rc is defined as the number of negative reviews of that company. Rtd is defined as the number of occurrences of the candidate keyword with the complaint topic word in the negative reviews of the FKC dataset and Rt is defined as the number of negative reviews with that complaint topic word.
After that, to exclude some negative words as well as verbs and adjectives which do not help users acquire advice, we weight candidate keywords using morphological and polarity analyses, as shown in Table 2.
Finally, we calculate the final score of the candidate keywords by combining the arithmetic mean of the company and complaint topic relevancies with the weight as the following equation.
Score = relevancies 2 × W eiдht
After calculating the final score of each candidate keyword, we determine the threshold value for each company. Candidate keywords those scores are under the threshold value become the advice topic words. (2) (3) (4) 3.5
In this study, each company name and complaint topic word are matched with several advice topic words. To search for suitable websites, We use an OR-based search method to acquire advice websites. Our proposed system generates the query based on one company name, one complaint topic word, and one advice topic word. 3.6
Our proposed system recommends suitable web pages containing advice from the results of the query which is based on users’ complaints. Figure3 shows the user interface of our proposed system. First, the system generates several queries by analyzing user’s negative review. Next, user can choose and browse the web page based on their needs by a web search using the ofered queries. The system recommends the advice information that could address user’s dissatisfaction expressed in the negative review. In this study, we conducted an experiment to extract the complaint and advice topic words in order to verify the feasibility of proposed system. For this experiment, we analyzed the subcategory of “IT web services" of the FKC dataset, which is under the category “industry.” We analyzed 1,000 negative reviews for each of three companies.
First, we extracted the complaint topic words and determined diferent threshold values for each of the three companies. For company A, we extracted 186 complaint topic words above the threshold value of 0.00080. For company B, we extracted 144 complaint topic words above the threshold value of 0.00076. For company C, we extracted 86 complaint topic words above the threshold value 0.00080. Table 3 shows examples of the complaint topic words for each company. These examples show that each complaint topic word implies the object of diferent users’ dissatisfaction.
Next,we extracted advice topic words from the candidate keywords that had a score less than 0.0043, 0.0020, and 0.0033 for companies A, B, and C, respectively. Some examples of these words are shown in Table 4. As TABLE 4 shows, the proposed method is suficient for ranking candidate keywords. In this paper, we conducted a questionnaire-based survey to evaluate the usability and efectiveness of the proposed method.The questionnaire-based survey contained following 3 questions. For Q1, we extracted all nouns from the negative review for respondents to choose from. For Q2, we provided 15 queries for each negative review to choose from. 5 of the queries’ candidate keywords were made by ourselves. For Q3, we evaluated the satisfaction of the result of advice recommendation.
Q1:Please choose one word which you think could represent the dissatisfaction of the following reviews.
Q2:please choose the query that you think the contents returned by a search using this query keyword could address the complaints found in the negative review.(multiple choices are allowed) Q3:Please make a web search with the queries you have chosen. Do you feel satisfied with the contents of the advice? 4.3
We collected the answers of 10 respondents, and the results are shown in Table 5 and Table 6. We defined those nouns and queries were chosen by over 5 answers as true positive.
The result of Q1 showed that if we search by using those nouns are with the highest value of importance for one time only, 60% of appropriate complaint topic word can be extracted from the 0.70 0.18 0.28 0.67 0.50 0.57 0.60 0.75 0.67
The result of Q2 showed that if we search advice by using the query which is with the lowest score for one time only, 70% of appropriate queries can be ofered to make web search in order to address the complaints expresses in the negative reviews. This result demonstrated that scoring candidate keyword is efective. However, we found out that the longer the candidate keyword was, the lower the score will be when making the candidate keywords . In the future, we plan to develop a method to ensure if the candidate keyword is related to the complaint topic word to better exclude noise in the results.
For Q3, the result showed that 75% of the answer felt satisfied with the contents of advice they searched with the queries they’ve chosen. From this result, it is observed that by using the proposed system could address users’ dissatisfaction and the recommendation of advice respond to the demands of diferent users. Moreover, it implied by using proposed system can help users to release their burden when searching for advice comparing to traditional search engine. 5
In this paper, we proposed a recommender system by analyzing complaint data to recommend suitable advice. We extracted query keywords from various user complaints about a certain service by calculating the score of each query. Then suitable web pages containing advice are recommended from the results of the query. In addition, we evaluated the efectiveness and usability of the proposed system through a questionnaire survey, and the results shows that the generated query keywords would be useful for collecting advice. In addition, the recommendation of advice returned by query keywords could address users’ dissatisfaction with a service and respond to diferent user demands in a comprehensive way.
In the future, we plan to evaluate the satisfaction of each query and analyze the result. Furthermore, we will consider new methods to obtain candidate query keywords which users are hard to associate to enhance the usability of the proposed system.
In this paper, we used FKC Data Set provided for research purposes by National Institute of Informatics in cooperation with Insight Tech Inc.