In this paper we present a hotel filter recommendation method designed to address the cognitive load users face in an overchoice scenario. As online products and services are continuously diversifying, user needs are also becoming increasingly sophisticated. However, with more items to choose from, grasping the entire choice set and differentiating among all matching options becomes increasingly difficult, leading to sub-optimal outcomes. Conventional hotel reservation platforms provide with a limited set of additional filters, but these can not accommodate all intricate user needs. Employing natural language processing and machine learning techniques, we provide a simple framework that identifies meaningful filters from customer reviews. We define criteria and scoring methods to acquire relevant and interesting filters that may help customers refine their needs or even identify hidden, previously unknown ones. Our simulated user experiments show that our proposal is capable of identifying intricate and useful filters, leading to increased customer satisfaction. CCS Concepts: • Computing methodologies → Machine learning; • Information systems → Content ranking; Recommender systems; Rank aggregation; Similarity measures. Additional Key Words and Phrases: overchoice, clustering, filter recommendation
Online services have become not only ubiquitous, but indispensable in almost every aspect of our life. Nearly every imaginable product or service is available through e-commerce transactions, including online shopping, restaurant or hotel reservation to matchmaking. In a conventional hotel reservation service the customer is provided with an interface that facilitates search using some of the most crucial criteria, typically objective queries that are meant to reduce the choice set to a manageable size (e.g., number of visitors, length of stay, location, etc.).
The emergence of e-commerce systems or online reservation services brought forward the advantage of having an
increased selection at the convenience of only a few clicks away. Both classic economics and psychology emphasize the
benefits of a larger number of choices [
Furthermore, with continuous product diversification, user queries are also becoming even more refined, contributing
to customer satisfaction and self-satisfaction being increasingly difficult to achieve [
In our work we provide a simple framework for automatically acquiring filters related to the hotels that match the customer’s initial query, by identifying useful mentions from customer reviews. We especially focus on customer experiences that are potentially both relevant and interesting for other customers, while also having the capability of reducing the choice set in an intuitive and natural way.
The main contributions of this paper are the following: (1) In order to address the overchoice problem, we present a simple clustering based approach to identify useful iflters in a dynamic manner, from customer reviews. (2) We define key concepts and strategies in scoring and ranking filters that are meaningful and natural for the customer. (3) We present simple but eficient methods to implement filter scoring and ranking. (4) We validate our proposal through a series of user experiments. We found that subjective, experience based filters that express quality judgements were especially useful for potential users to narrow down the search space.
The paper is organized as follows: in Section 2 we discuss the related work, followed by the definition of key concents of our approach in Section 3, data description in Sections 4 and details of our proposal in Section 5. We describe our experiments in Section 6, followed by discussions with future directions in Section 7 and the concluding remarks in Section 8. 2
Automatic facet generation is a closely related field to our task. Faceted search augments traditional search by presenting a
set of attributes or filters that are grouped into facets, allowing customers to narrow down the search results [
As another method to reduce information overload, customer review summarization is also related to our field
[
Customer reviews have also been the target of sentiment analysis [
Also, customer reviews can be employed to generate recommendations [
Published work on query suggestion and recommendation has been prominently focused on the web domain
[
Related to query recommendation is the field of query rewriting, the task which aims to reformulate customer
queries into well-formed ones, in order to improve customer experience [
Interestingness or uniqueness discovery, key concepts in our work, is another related field, with special focus on
news articles [
On a note, the field of anomaly detection [
Our goal is to automatically identify filters that are characterized by: (1) being appealing to the customer; (2) having the potential of addressing the overchoice problem by reducing the choice set in an intuitive and natural way. We define a iflter “appealing” as having the quality of being both relevant and unique. Also, we define a filter set “appealing” as being diversified, without too much emphasis on a single topic or aspect. Furthermore, to perform choice set reduction in an intuitive way, we introduce size control policies.
Relevance, uniqueness, diversity and size control are key concepts of our proposal. We employ size control policies and diversity rules as hard constraints to identify possible filters, while using relevance and uniqueness scores to determine the final filter ranking. 3.1
Filters are required to hold enough decision power in order to be viable expressions of user intent. Pre-defined static iflters of conventional hotel reservation platforms are good examples of high relevance (e.g., breakfast included, late check-out). We attempt to assign relevance scores to all possible filters. While relevance is highly subjective, we can 1“an observation that deviates so significantly from other observations as to arouse suspicion that it was generated by a diferent mechanism” argue that all else being equal, certain filters satisfy a larger audience than others (e.g., close to the city center versus bright pink curtains). Detailed information about relevance scoring can be found in Section 5.2.1. 3.2
Filters are also required to be representative of the choice set that matches the customer’s original query, capturing characteristics that are unique within the search results. The motivation behind uniqueness is to identify options that are especially appealing within the hotels that already match the user query (e.g., next to the city aquarium), with the added potential to ofer choices previously unknown by the customer (e.g., private hot-spring). Section 5.2.2 ofers detailed description on uniqueness scoring. 3.3
The importance of diversity and serendipidity is well recognized in the context of recommender systems [
By definition, filters are designed to address overchoice and reduce the choice set, i.e., the number of matching hotels. We argue that the degree of the size reduction is also important. Providing highly appealing, but too generic or too specific filters might result in a too drastic or too shallow choice set reduction, leading to customer dissatisfaction. Instead, our strategy is to identify and provide only the filters that are guaranteed to result in a “just right” window of matching hotels, compared to the original number of matching hotels. Naturally, this implies that our filters are based on availability, i.e., filters that obey size control rules are guaranteed to reduce the choice set.
Intuitively, in practice this should provide a natural way in reducing the choice set, balancing between relevance and uniqueness. However, more often than not, relevance and uniqueness work against each other. Highly relevant iflters are often not very unique (e.g., free continental breakfast), while highly unique filters may not be relevant to a large audience (e.g., stay at a buddhist temple). When the choice set is large, arguably it is more natural to select from more generic, thus high relevance, low uniqueness filters, with the preference shifting towards high uniqueness, low relevance filters with a decreasing choice set. With a large choice set, size control policies rule out filters that are not frequent enough, thus disregarding long-tail, but unique filters, with higher relevance ones gaining more prominence. With a decreasing choice set, long-tail, unique filters should gain more exposure at the expense of more generic, relevant iflters. More information about size control policies can be found in Section 5.1.2. 4
As our data source we use over 20 million sentences extracted from hotel reviews, collected from one of the largest hotel booking sites in Japan2. The hotel review corpus contains the customer review texts and the location data associated to each hotel.
Original predicate-argument structure very delicious food, really delicious food, all food is delicious, food is of course delicious, delicious food as advertised, more than delicious food hotel is close to the station, really close to the station, close to the station as mentioned, closest to the station, pretty close to the station extremely clean rooms, very clean rooms, rooms clean as always, rooms are of course clean, thoroughly cleaned rooms, rooms cleaned to the last detail 4.1
An underlying assumption of our method is that a user friendly filter extracted from customer reviews can be represented
by a simple predicate-argument structure. To this end, we extracted over 20 million predicate-argument structures
from our corpus by using JUMAN++ (v2.0.0-rc3), a Japanese morphological analyzer [
Some of the resulting predicate-argument structures were unrelated to hotels or had negative polarities, unsuitable
for our filter policies. As a result, we employed a filtering method based on the automatic classification results of
two BERT-based classifiers fine-tuned with an annotated corpus 4 [
To represent filters, we pre-trained a BERT [
To improve on BERT’s embeddings, we employed the sentence embedding framework described in [
4https://github.com/megagonlabs/jrte-corpus
representation described in [
We developed machine learning based methods to identify and rank filters. Given a set of hotels that match an initial set of original user queries, we automatically extract the non-negative core predicate-argument structures described in Section 4.1 from the customer reviews of the matching hotels. These core predicate-argument structures will act as potential filters. First, using the sentence embedding representations of the filters, we apply a 2 staged hierarchical clustering method to group them into semantically similar clusters. In this step we employ policies to identify clusters that follow size control restrictions. Next, we score each cluster for relevance and uniqueness to determine the final filter class ranking. In this step we employ diversity policies. Also in this step we label the top ranked filter clusters. Below is a detailed description of each step. 5.1
5.1.1 Stage 1: main topic identification. In the first stage of clustering we attempt to group filters into main latent topics, e.g., food, location, hot spring, etc. The purpose is to serve diversity by identifying such latent topics, with the assumption being that filters from diferent clusters after stage 1 will roughly have diferent topics 5.
In order to identify the main topics, we employ Ward’s agglomerative clustering method [
We recognize that a carefully curated knowledge-driven approach may have the advantage in accurately associating pre-defined topics to filters. However, besides cost issues, our data-driven approach has the advantage of recognizing a potentially infinite number of intrinsic topics that would be dificult to manually acquire.
Also note that the purpose of the first stage is solely to associate filters with latent topics, thus this step can be performed beforehand, independently of user queries. 5.1.2 Stage 2: filter identification with size control. In the second clustering stage we identify filter clusters from each main topic from Section 5.1.1 whose size obey size control rules. The size of a filter cluster is defined as the total number of hotels that the members of the cluster are linked to. Size control is governed by two parameters, _ and _ that represent the lower bound percentage and upper bound percentage, respectively, in respect to the size of the original choice set.
To achieve this, for each topic output by the main topic identification step described in Section 5.1, we parse the hierarchical subtree of each topic by incrementally moving up in the cluster hierarchy. During this process we retain clusters that obey size control rules and stop where the linkage drops below a certain similarity threshold, empirically set to 0.7. Empirically, we set _ and _ to 30% and 70%, respectively. 5.2
We score and rank filter clusters retrieved in the 2-stage clustering step based on their relevance and uniqueness. After ranking, we apply diversity rules and label the top filter clusters as described below.
5Note that we do not attempt to label the resulting topics. Instead, we only attempt to identify filter groups that belong to the same latent topic. 5.2.1 Relevance score. In Section 3.1 we stressed the importance of discovering filters that are crucial enough in the decision making process. Determining relevance is a non-trivial task, since people’s preferences are obviously not uniform. A filter that may be highly relevant for one customer, may be less relevant for another one (e.g., rich choice of baby formula or free pair ticket to the city aquarium), depending not only on personal preferences, but also on situations or even purpose of visit.
Since this is a cold start scenario, personalized relevance estimators based on customer action logs are not feasible. Instead, we define the relevance of a filter independently from the original user query, as the average of multiple subjective relevance scores.
We pre-computed filter relevance scores using a simple k-nearest neighbor classifier. For each filter we took the top = 5 similar filters from our training data, and computed their average relevance, weighted by the similarity score, as shown in the below formula, where denotes the target filters, denote filters of the training data, relevancegold denotes gold relevance scores of the training data.
Í relevance( ) = =1 cossiÍm(, ) × relevancegold ( ) (1)
=1 cossim(, )
As similarity score we employed cosine similarity, computed on the sentence embeddings described in Section 4.2. We normalized the relevance score by scaling it to between 0 and 1.
As training data we randomly selected 8000 filters and asked 5 crowd workers to label their degree of relevance from
5 to 16. The most relevant was labeled with 5, the least relevant being 1. Ungrammatical or semantically unsound filters
were labeled with 0. We calculated pairwise inter-annotator agreement using Weighted Cohen’s kappa [
For our classifier we used the filters that were judged as grammatically correct by at least 4 out of the 5 workers. For the grammatically correct filters we averaged the individual worker scores. We preferred to use truncated mean (e.g., ignoring the lowest and highest scores of the 5 workers) in order to counter for highly subjective relevance scores (e.g., rich choice of baby formula). Table 2 shows an excerpt of the filters and their respective averaged relevance scores.
We evaluated our relevance classifier on a held-out data of 1000 samples by calculating the precision on increasing error ranges. We achieved a precision of 60.40% when the error range between the estimated relevance and reference relevance was less or equal than 0.1 points, and 85.50% precision at 0.2 points error range, as shown in Figure 1. 6We manually selected the crowd workers based on their demographic information (i.e., gender, age range) to ensure diversity.
We compute relevance scores for filter clusters as the weighted average relevance of its member filters as shown in the below formula, where denotes filter clusters, relevance( ) denotes filter relevance and freq( ) denotes the frequency of filter in the target choice set. 5.2.2 Uniqueness score. We define the uniqueness of a filter as the property of being important within a selected group of hotels. We employed term frequency–inverse document frequency (tf-idf) as uniqueness of each filter, where denotes the filter, denotes the reviews of a specific hotel.
uniqueness(, ) = tf (, ) × idf ( ) (3)
Intuitively, a unique characteristic of a subset is more dominant in the subset than within the entire population. The
sparse nature of the filters makes it unfeasible to handle them individually, thus for the purpose of computing tf -idf, we
employed Ward’s agglomerative clustering method [
Similarly to relevance scores, we computed filter cluster uniqueness score as the weighted average uniqueness of its member filters as shown in the below formula, where denotes filter clusters, uniqueness(, ) denotes filter uniqueness for filter in hotel review set , freq( ) denotes the frequency of filter in the target choice set. uniqueness( , ) = Í ∈ freq( ) × uniqueness(, ) Í ∈ freq( ) 5.2.3 Filter ranking and employing diversity rules. Filter cluster ranking is determined by multiplying the filter cluster’s relevance and uniqueness scores, weighted by their respective weights (i.e., and for relevance and uniqueness, relevance( ) = Í ∈ freq( ) × relevance( ) Í ∈ freq( ) (2) (4) respectively), as shown in the below formula. During preliminary empirical evaluations, we found that a reasonable value for and were 1 and 2, respectively.
rank( ) = ( + relevance( )) × ( + uniqueness( ))
To produce the final ranking, only the top iflter clusters are retained for each main topic described in Section 5.1. We set to 1, e.g., we retain only the top filter from each topic 7. (5) 5.2.4 Labeling filters. Finally we label the top filter clusters that will be presented for the customer. We perform cluster labeling by choosing the most representative member, i.e., the member closest to the cluster centroid. We utilize cosine similarity to determine the representative member that will act as the label of the filter cluster. 6
We conducted a number of user experiments to evaluate: (1) the top overall output and (2) the top individual filter outputs of our proposed method, described in Section 5. Particularly, we compared the filters output by our proposed method against manually acquired filters. Also, we assessed the efect of uniqueness, relevance and diversity policies. To this end, we performed pairwise comparison against the following baseline models: • human: a manually compiled filter list described below in Section 6.1. • relevant: proposed without the uniqueness score, i.e., filter ranking is determined only by relevance. • unique: proposed without relevance score, i.e., filter ranking is determined only by uniqueness. • non-diverse: proposed without diversity policies, i.e., output is not restricted to the top iflters for each main topic.
To manually acquire filters in a simulated overchoice scenario, we randomly selected 10 <original request, location> query tuples with the following conditions: • the resulting hotel hit count is at least 30 in our hotel review corpus; • the total number of corresponding reviews8is at least 3000 in our hotel review corpus.
Table 3 shows the query tuples used in this process. From the resulting reviews we randomly selected 1000 reviews for each query tuple. Next, using the selected reviews, we asked 3 crowd workers to extract all simple short phrases which in their opinion contain meaningful information in further filtering the choice set. Such phrases were manually grouped by each worker into clusters that share the same meaning. Finally, all clusters were aggregated by a fourth crowd worker, registering the number of contributing workers and the number of hotels each cluster links to. As the ifnal output, we considered filters that had a number of majority contributors (at least 2 out of 3), ranked in descending order by the number of corresponding hotels. Table 4 shows an example of manually acquired filters. 7One important note is that before applying diversity policies, first we remove filters that are semantically too similar to the original user queries. We perform this by using cosine similarity on their sentence embeddings described in Section 4.2. Empirically we set this similarity threshold to 0.8. 8We performed exact text match in retrieving hotels reviews that mention an original request. in Section 6.1. For each query, we considered the top = 5 filters from each method’s output.
We crowdsourced the pairwise evaluation, asking 300 workers using Yahoo!Japan’s crowdsourcing service9 to choose the filter list they find more suitable in further narrowing down the choice set. In randomized order, we showed the iflter lists of the two methods (named lists A and B, respectively), asking the workers to choose exactly one of four choices: • list A is more useful than list B • list B is more useful than list A • list A and list B are both equally useful • neither of the lists are useful
We also asked the workers to motivate their choice for each filter list pair. After basic data quality check (i.e., removing workers that (1) did not provide any explanation for their choices, (2) always choose the same option, or (3) working time was too short), we retained the results of 224 workers. Table 5 shows the overall list comparison results for each
Against the manually generated human filters, proposed was considered to be the significantly better 10overall choice with 5 out of the 10 evaluation query tuples. In 2 out of 10 cases the human output was considered to be significantly superior to the output of proposed. Overall, proposed was found to have a significant advantage over human with over
Analysing the workers’ comments, we observed that the overall output of proposed was overwhelmingly preferred over human when the filters ofered very specific choices or experiences (e.g., the parking lot is large, thus easy to park the car, delicious food with local ingredients). At the same time, human was preferred by workers who value filters which proposed considered as relevant, but not unique enough to rank high (e.g., large room, clean hotel). It is also worth mentioning that when human outperformed proposed, the number of votes counted for either of the methods was actually smaller than in average, both lists being equally preferred or unpreferred by a large number of workers. We can also note that proposed was voted as the better overall choice by an overwhelming majority with a number of query tuples (e.g., Close to the sea @Shizuoka). The reason for this vote diference is that proposed managed to identify filters that are highly specific to the initial query tuple, and at the same time are also quite appealing to potential customers (e.g., the splendid alphonsino was very delicious), while human failed to identify such filters with a high enough frequency.
Against non-diverse as well, proposed exhibited a significant vote advantage (45.4 points), validating the efect of the diversity policies. However, non-diverse did perform better with some query tuples that are related to locations especially recognized or famous in relation with a specific main topic, which was captured and over-represented by the non-diverse method (e.g., nature topic in Akita).
Proposed also outperformed unique and relevant by over 6.3 and 29.2 points vote count diference, respectively, validating that both relevance and uniqueness contribute significantly to proposed. It is worth mentioning that relevant behaved very similarly to human against proposed, suggesting that the workers employed in acquiring the manual filters may have had preference towards more relevant, rather than unique filters.
10We checked for significance using binomial test of significance with set to 0.05.
In the second set of experiments we performed pairwise comparison of the individual filters output by proposed against the outputs of the target methods11. Here we attempt to counter the tendency some workers may have had in rejecting certain filter lists during list based evaluation described in Section 6.2, for the reason of containing unappealing filters. We used the same filters as with list based evaluation, merging and shufling the filters into a single list. In case of duplicates, a single occurrence was retained. We crowdsourced the evaluation asking 300 workers using Yahoo!Japan’s crowdsourcing service. As opposed to filter list evaluation, here workers were asked to award individual filters, by selecting from 1 up to at most 5 filters they find appealing in further reducing the choice set. We also asked the workers to motivate their choice for each selection set.
After basic data quality check (i.e., removing workers that (1) did not conform with the rule regarding the number of selected filters, or (2) working time was too short), we retained the results of 197 workers. For each query, we counted the total number of votes each method received. Filters that were duplicates counted for both methods. The results for each query tuple are summed up in Table 6.
Against the manually acquired human method, proposed had a significantly larger vote share with 5 out of the 10 evaluation queries, while being outperformed in only one case. We also found that proposed received more votes for filters that express experiences or quality judgements, positive opinions regarding a specific service or the hotel in general (e.g., the open-air hot spring was excellent, the free breakfast was delicious), while the human filters had the tendency to have more factual filters or presence/absence indicators (e.g., open-air hot spring was available, free breakfast). Similarly to overall filter list evaluation, human received numerous votes for filters that are highly relevant, but were ruled out by size constraints by proposed (e.g., clean rooms being too frequent, washing machine available being too rare).
Also similarly to filter list evaluation, proposed outperformed both relevant and unique. However, it must be noted that both relevant and unique received numerous votes with filters that are very relevant, but less unique (e.g., clean rooms, excellent service, free wifi ) or unique, but arguably not relevant enough (e.g., karaoke machine is available, dog run attached to the hotel). These results suggest that while relevance and uniqueness both contribute to proposed, their importance is highly subjective.
11Here we skip pairwise comparison against non-diverse, since the target of this experiment are filters, as opposed to filter sets. 7
Firstly, as observed during both list based and individual filter evaluations, workers preferred highly specific filters as opposed to more generic ones (e.g. the splendid alphonsino was very delicious versus food was delicious, the parking lot is large, thus easy to park the car versus parking lot available). This validates our assumption that the majority of potential customers are interested in very specific details in attempting to reach a decision.
Secondly, during the individual filter evaluation, we observed the worker’s tendency in preferring filters that were formulated as an experience or quality judgement, rather than as a fact or presence/absence indicator, when both options were available. (e.g., hot-bath was great versus hot bath is available; food was delicious versus food available at hotel). This tendency was weak with topics in which experience itself may not be too relevant (e.g., the experience expressing easy to park was not overwhelmingly preferred over the factual parking lot available, arguably because the fact that parking is actually available is the crucial piece of information, rather than the ease of parking), but it was very prominent with topics such as food, location or other service related ones, where previous user’s experiences and reviews are more valuable than the simple availability of that specific option.
We validated this assumption with a very simple experiment. We manually selected 30 (fact, experience) filter pairs and for each filter pair we asked 20 crowd-workers to choose the filter list they find more suitable in further narrowing down the choice set. We randomized the order of the two filters (named A and B, respectively), asking the workers to choose exactly one of four choices: • filter A is more useful than filter B • filter B is more useful than filter A • filter A and filter B are both equally useful • neither of the filters are useful
After basic data quality check (i.e., (1) always choose the same option, or (2) working time was too short), we retained the results of 19 workers. In 28 out of 30 pairs the experience based filter received the higher share of votes, although both fact and experience based ones received many votes. Most of these pairs had the topic of location, food, hot spring or some other type of hotel service. In case of a single pair the diference was only minimally in favor of the experience based filter, namely parking as topic. One pair was voted as being equally helpful, having received only a few votes for either fact or experience based filters, in the topic of hotel amenities (amenities are available versus very basic amenity).
This result suggests that the balance between fact and experience based filters is both subjective and possibly topic dependent. While customer reviews mainly ofer intricate experience-like details, fact based filters still remain valuable. As current filters provided by conventional hotel reservation systems are largely fact based ones, undoubtedly intricate experience based filters extracted from customer reviews could add significant value. In deploying such a customer review based filter recommender, strategies need to be implemented to combine various types of filters from multiple sources of information. In the future we are planning to investigate how various sources of information can complement each other in providing meaningful filters. 8
In this paper we proposed a simple clustering based approach to address the overchoice problem in the hotel industry domain. We introduced size control and diversity policies, together with scoring verticals, in order to identify and score iflters that could reduce the search space in a natural and intuitive way. We validated our proposal through a series of user experiments where we also showed that the filters identified by our method were more useful than the manually acquired ones.