The primary goal of tourism management platforms, e.g. booking.com, is to provide the best matches to travelers. User-generated reviews are often leveraged in various ways to influence user's decision-making process. One straightforward approach is ranking historical reviews based on user's preferences by checking the “helpfulness” votes. A major issue with this approach is that many reviews do not receive helpfulness votes, leading to a presentation bias. In this work, we incorporate multiple review features to rank reviews based on user profiles. Reviews are encoded using a state-of-the-art transformer encoder model (e.g., SBERT), and cosine similarity is computed between user profiles and reviews. The ranking performance is assessed with MRR@10 (Mean Reciprocal Rank) and Precision@10. Our results demonstrate that beyond helpfulness votes, leveraging additional features (e.g., accommodation type, review title, positive aspects of reviews, etc.) significantly improves performance. The implementation of our method is available on the GitHub 1
The rapid growth and success of e-commerce have sparked extensive research aimed at improving
customer engagement [
In the traditional approach, reviews are ranked based on either recency or helpfulness. Helpful
reviews are those that receive positive feedback from other users who purchased the same product,
while recent reviews are sorted by the time they were posted [
In fact, sorting reviews based on “helpfulness” votes is subject to the Matthew efect which states that a
review positioned at the top of the list tends to stay there because users primarily interact with and
vote on the top reviews when making purchase decisions. In contrast, a review placed at the bottom
remains unnoticed, as users rarely scroll down to see it[
To address this issue, many e-commerce platforms have introduced various methods to filter reviews
and enhance customer satisfaction. Hsieh et. al. [
In the RecTour 2024 Challenge organized by Booking.com [
Review ranking is considered as one of the efective methods to enhance the user engagement in the
e-commerce business. Numerous studies have been conducted on personalized review ranking [
Integration of diverse user feedback has proven highly efective in understanding user preferences
and item characteristics for recommendation systems [
Recent advancements in natural language processing have yielded sophisticated text encoding models.
BERT (Bidirectional Encoder Representations from Transformers) [
Consider a user, identified by _ , and an accommodation, identified by _ . Each accommodation is associated with multiple reviews, each with a unique _ . For a specific user and accommodation, the objective is to retrieve the top 10 most relevant reviews. The method involves three components: 1) user and accommodation profile creation, 2) feature extraction, and 3) similarity search and ranking.
The first step of the review ranking process is the user and accommodation profile construction. As a baseline approach, we rank the review based on the helpfulness vote. To achieve this, for each user and accommodation, we create a list of pairs with review IDs and “helpfulness” vote counts. Then we sort them in descending order and the top-voted review ids are kept. Similar to the “helpfulness” vote, we follow the same strategy when using the review scores for review ranking.
As the next step, we hypothesize that accommodation review profiles are influenced by attributes such as _ , _ , and _ while user profiles are linked to _ , _ , _ , _ , _ _ , and _ _ _ . To build comprehensive user and accommodation profiles, we concatenate these features. We experiment with several combinations of features for both user and accommodation profiles. The detailed combination of diferent features is discussed in section 4.2.
We apply pre-trained sentence-BERT for feature extraction from the user and accommodation profile.
Sentence-BERT is a variant of the pre-trained BERT [
To compute the similarity score between each user and the list of reviews for accommodation, we use the cosine similarity measure. For a given user and a review , the similarity score can be computed as follows.
sim(, ) = ⋅ ‖‖‖ ‖ where and are the vector representations of the user and review. The detailed workflow of the method is shown in Figure 1 (1)
In this work, we use the dataset provided from the RecTour 2024 Challenge organized by Booking.com
[
We perform several experiments using diferent combinations of features. In all cases, for feature
extraction, we use the smaller and faster version of SBERT [
Experiment 1 and 2: In this case, for each user and accommodation, we sort the reviews based on “review_helpful_votes”, and “review_score” respectively. Next, we retain the top 10 reviews for each User ID and accommodation ID pair.
Experiment 3: In experiment 3, user profiles are constructed by concatenating the diverse set of review features including “guest_type”, “guest_country”, “room_nights”, “month”, “accommodation_type”, “accommodation_country”, “accommodation_score”, “accommodation_star_rating”, “ location_is_beach ”, “location_is_ski”, and “location_is_city_center” respectively. Similarly, accommodation profiles are created using various review features such as “review_title”, “review_positive”, “review_negative”, “review_score”, and “review_helpful_votes”. Each accommodation has a list of reviews. Using a sentence transformer, both the user profile and all the reviews from an accommodation are embedded. Then, the similarity score between each user profile and all the reviews is calculated. Finally, based on these similarity scores, the top 10 corresponding review IDs are retrieved.
Experiment 4 and 5: Similar strategy is applied for experiment 4 and 5. However, in experiment 4, accommodation profiles are kept the same as in experiment 3, but user profiles are shrunk to only “guest_type” and “guest_country”. On the other hand, experiment 5 keeps the user profile the same but the accommodation profile is added with more features, e.g. “accommodation_type”, “accommodation_country”, “accommodation_score”, “accommodation_star_rating”, “ location_is_beach ”, “location_is_ski”, and “location_is_city_center” respectively.
We evaluate the performance of our ranking system using MRR (Mean Reciprocal Rank) and Precision, specifically MRR@10 and Precision@10, which indicate the MRR and Precision of the top 10 ranked reviews.
MRR (Mean Reciprocal Rank)@K MRR@K measures the quality of a ranking system by focusing on the position of the first relevant item from the top K retrieved items. For a user u, the reciprocal rank is: where rank is the rank position of the first relevant item.
The Mean Reciprocal Rank across all is:
Reciprocal Rank =
1 rank MRR@K = 1 ∑| | 1
| | =1 rank
MRR@K is useful for evaluating how quickly the first relevant result appears. If no relevant item is found within the top K, the reciprocal rank for that user is 0. A higher MRR indicates better performance in placing relevant items early in the ranking. When using MRR@10, we focus specifically on the top 10 positions to evaluate performance within this subset.
Total Number of relevant items in the top K | | where K is the number of items in the ranked list.
A higher MRR@K score shows that the first relevant result appears early in the ranking, while a higher Precision@k score indicates a higher proportion of relevant items in the top K results. MRR@K is best for situations where finding the first relevant item quickly is important, whereas Precision@K is better for evaluating systems that need to return multiple relevant items.
The performance of our method is presented in Table 1. It is obvious from the results that when we use only helpfulness vote or review score, both the MRR@10 and the Precision@10 are low compared to the addition of features. In experiment 5, MRR@10 and Precision@10 are 0.0787 and 0.2605, respectively, while in experiment 1, they are 0.0735 and 0.2511, which indicates that MRR and Precision are 0.52% and 0.94% higher in experiment 5 compared to experiment 1. Experiment 1 and 2 give similar performance for both metrics, which indicates that both helpfulness vote and reviews score have similar impact on review ranking. Experiment 3 shows slightly better performance than Experiment 4, suggesting that the addition of features captures users’ comprehensive behavior. This highlights the importance of incorporating textual reviews when modeling user behavior and item characteristics.
Therefore, we can conclude that user-generated textual reviews contain user and item-representative features and leveraging them to review ranking improves the ranking quality compared to traditional helpfulness vote or review score-based ranking methods.
In this work, we present a review ranking method for accommodation. We incorporate several forms of review features to create user and accommodation profiles. Sentence Transformer is applied to extract the features from the review profiles, and finally, the cosine similarity score between the user profile and each review is measured to rank the reviews. The performance of the method is evaluated with MRR (Mean Reciprocal Rank) and Precision@K. Our results reveal that integration of linguistic features to create user and accommodation profiles outperforms the vanilla helpfulness vote-based review ranking method. In the future, we aim to leverage state-of-the-art LLMs to rank the reviews.
This project is partially sponsored by the Natural Science and Engineering Research Council of Canada (grant 2020-04760). 7. Appendix guest_country room_nights month accommodation_id accommodation_type accommodation_score accommodation_country accommodation_star_rating location_is_beach location_is_ski location_is_city_center Field name review_id review_title review_positive review_negative review_score review_helpful_votes user_id guest_type