The ACM RecSys RecTour 2024 Challenge focuses on ranking reviews, which is an important aspect that influences users' decision-making. The goal of this challenge is to match given accommodations and users to their respective review IDs. The concept is that when a new user interacts with the booking system, we can analyze the accommodation they are viewing along with available user features (e.g., couple, country, etc.). This enables us to display reviews in an order that considers the review content with respect to the user and accommodation characteristics. To do so, Booking.com provides a unique training dataset containing 1.6 million reviews based on real anonymized bookings. Thirty teams signed up for the challenge. The top-performing teams are invited to present short papers describing their solution approaches.
Our objective is to create a model that predicts the helpfulness of every review tailored to individual users. In essence, we aim to construct a personalized helpfulness function, denoted as ( | ), which evaluates the relevance of review to user given their context . This function assigns a score indicating the degree to which review is beneficial for user . These scores enable us to rank reviews, ensuring that those with the highest values are deemed most helpful within context .
Using the number of helpful votes as the target signal inherits multiple issues. First, it introduces a presentation bias towards the previous review ranking algorithm (usually sorted by votes). Additionally, the signal of votes is sparse as most of the reviews are not presented and therefore not voted. Moreover, there might be a cold-start problem where new reviews don’t have as many votes as older reviews which might be less relevant over time. Finally, in many cases, only the final number of votes is stored and therefore it’s not feasible to use this signal for developing personalized review ranking models.
Thus, we introduce a more feasible and novel approach for modeling personalized helpfulness measure. We model the personalized helpfulness of a review as the likelihood that it is written by its reviewer given the reviewer’s context. Notably, we define such that given a user context , it estimates the likelihood that review was written by the user. Formally, we optimize Workshop on Recommenders in Tourism (RecTour 2024), October 18th, 2024, co-located with the 18th ACM Conference such that given that review was written by a user with context , it holds: 1 if = ( | ) = { 0 if ≠ (1)
In this challenge the task is to match given accommodations and users to their respective review IDs. Therefore, we provide three sets of data as follows:
Users – hold information regarding users and accommodation features. Review – hold information regarding reviews. Matches – a true label between given _ , _ and _ (only positive examples).
The dataset we publish contains authentic user-generated reviews from 50,000 accommodations.
This includes information on the user reservation, the review and the accommodation. The
dataset consists of 2,031,914 anonymized reviews, along with guest and accommodation context.
It is based on real data, and available via the following repo1. Table 1 describes the dataset fields.
Further information regarding the creation of the dataset is presented in [
Key dates of the challenge are listed in Table 2.
The submission file should include 12 columns: Accommodation ID, User ID, and the top 10 review IDs ranked according to your model’s predictions on the test set.
An example of a ranked review for Accommodation ID 1 and User ID 1. The algorithm predicted the following ranking of reviews (158, 32, … 97). Consequently, the submission file will display it as demonstrated by Table 3.
The top 10 teams will be invited to submit short papers (up to 4 pages + references). The papers will include the team and the authors names, an abstract, sections describing the method and results, a link to their code repository and a reference to the Booking.com challenge in the following format: Amit Livne, and Eran Fainman. 2024. Booking.com RecSys RecTour 2024 Challenge. http://www.bookingchallenge.com , In Workshop on Recommenders in Tourism (RecTour 2024), October 18th, 2024, co-located with the 18th ACM Conference on Recommender Systems, Bari, Italy.
Selected papers are expected to present their work in the workshop. The submitted papers will be evaluated based on their clarity, novelty, and results presentation.
Please contact rectour2024challenge@booking.com for any questions.
1https://huggingface.co/datasets/Booking-com/accommodation-reviews
The goal of the challenge is to predict (and recommend) the review ID of each Accommodation and User IDs pair. The quality of the predictions is evaluated based on the MRR@10. The online results are available via the leaderboard2
2https://huggingface.co/spaces/Booking-com/rectour24-review-ranking-leaderboard-test
60 participants have signed up for the challenge. After two months of a contest, 10 of them applied a final submission. Top 4 performing teams are listed in Table 4.
The best performing team achieved MRR@10 of 0.1662. The teams have submitted short papers and code repositories with a detailed description of their solution methodology.