=Paper=
{{Paper
|id=Vol-2855/challenge_short_1
|storemode=property
|title=Booking.com WSDM WebTour 2021 Challenge
|pdfUrl=https://ceur-ws.org/Vol-2855/challenge_short_1.pdf
|volume=Vol-2855
|authors=Dmitri Goldenberg,Kostia Kofman,Pavel Levin,Sarai Mizrachi,Maayan Kafry,Guy Nadav
|dblpUrl=https://dblp.org/rec/conf/wsdm/GoldenbergKLMKN21
}}
==Booking.com WSDM WebTour 2021 Challenge==
https://ceur-ws.org/Vol-2855/challenge_short_1.pdf
ACM WSDM WebTour 2021, March 12th, 2021 Jerusalem, Israel 21
Booking.com WSDM WebTour 2021 Challenge
Dmitri Goldenberg Kostia Kofman Pavel Levin
dima.goldenberg@booking.com kostia.kofman@booking.com pavel.levin@booking.com
Booking.com, Tel Aviv, Israel Booking.com, Tel Aviv, Israel Booking.com, Amsterdam,
Netherlands
Sarai Mizrachi Maayan Kafry Guy Nadav
sarai.mizrachi@booking.com maayan.kafry@booking.com guy.nadav@booking.com
Booking.com, Tel Aviv, Israel Booking.com, Tel Aviv, Israel Booking.com, Tel Aviv, Israel
Figure 1: Multi-Destinations trip recommendation bar on Booking.com website
ABSTRACT a strategy for making the best recommendation for their next desti-
The ACM WSDM WebTour 2021 Challenge focuses on a multi- nation [2]. Booking.com releases this unique dataset to encourage
destinations trip planning problem, which is a popular scenario in the research on sequential recommendation problems [4]. The chal-
the travel domain. The goal of the challenge is to make the best lenge is part of the WebTour 2021 ACM WSDM workshop [3] on
recommendation of an additional trip destination. To encourage web tourism that will be held at the 14th ACM international 2021
research on this field, Booking.com provided a unique dataset based WSDM Conference.
on millions of real anonymized bookings.
More than 800 participants have signed up for the contest. Best Table 1: Dataset description
performing team achieved Accuracy @ 4 of 0.5939, using a blend
of Transformers, GRUs, and feed-forward multi-layer perceptron. Column Description
Additional leading teams implemented advanced state-of-the-art
user_id User ID
solutions to tackle the problem.
checkin Reservation check-in date
checkout Reservation check-out date
CCS CONCEPTS An anonymized ID of affiliate channel
• Information systems → Personalization; Recommender sys- affiliate_id where the booker came from (e.g. direct,
tems. 3rd party referrals, paid search engine, etc.)
device_class desktop/mobile
KEYWORDS Country from which the reservation was
booker_country
Personalization, Travel, Recommender Systems, Dataset, Challenge made (anonymized)
hotel_country Country of the hotel (anonymized)
city_id city_id of the hotel’s city (anonymized)
1 PROBLEM DESCRIPTION trip ID (a group of multi-destinations
utrip_id
Booking.com’s mission is to make it easier for everyone to experi- bookings within the same trip)
ence the world. By investing in the technology that helps take the
friction out of travel, Booking.com seamlessly connects millions of
travelers with memorable experiences, a range of transport options, 2 DATASET
and incredible places to stay.
The dataset consists of over a million anonymized hotel reserva-
Many of the travelers go on trips that include more than one
tions, based on real data, is available on the challenge website1
destination. For instance, a user from the US could fly to Amsterdam
and described in table 1. Each reservation is a part of a customer’s
for 5 nights, then spend 2 nights in Brussels, 3 in Paris, and 1 in
trip (identified by utrip_id), which includes at least four consec-
Amsterdam again before heading back home. In this scenario, the
utive reservations. There are 0 or more days between check-out
users are offered personalized recommendations [1] for extending
and check-in dates of two consecutive reservations. The evaluation
their trip immediately when they make their booking, as shown in
dataset is constructed similarly. However, the city_id of the final
figure 1.
reservation of each trip is concealed and requires a prediction.
The goal of this challenge is to use a dataset based on millions
of real anonymized accommodation reservations to come up with 1 https://www.bookingchallenge.com/
Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
ACM WSDM WebTour 2021, March 12th, 2021 Jerusalem, Israel 22
Dmitri Goldenberg, Kostia Kofman, Pavel Levin, Sarai Mizrachi, Maayan Kafry, and Guy Nadav
Table 2: Challenge key dates Table 4: Top 10 performing teams
When? What? Team Accuracy @ 4
December 1st, 2020 Challenge starts 1 NVIDIA RAPIDS.AI 0.5939
2 Synerise AI 0.5780
Test set release -
January 6th, 2021 3 TEAM DASOU 0.5741
teams registration deadline
4 mbaigorria 0.5566
Challenge closes - 5 aprec 0.5557
January 28th, 2021
results submission deadline 6 hakubishin3 & u++ & yu-y4 0.5399
February 4th, 2021 Announcement on the winners 7 testing 0.5332
8 Alexander Makeev 0.5310
February 18th, 2021 Paper submission deadline 9 YiNet 0.5112
February 25th, 2021 Paper notifications 10 Marlesson - MARS-Gym 0.4958
March 4th, 2021 Camera ready submissions
March 12th, 2021 Workshop day by using Top-4 Accuracy metric (4 representing the four suggestion
slots at Booking.com website). When the true city is one of the top
4 suggestions (regardless of the order), it is considered correct.
3 CHALLENGE TIMELINE
Key dates of the challenge are listed in table 2. 6 PRIZES
To encourage research contributions, the top three performing
4 SUBMISSION GUIDELINES teams will receive Booking.com Travel Credits. The best paper
The teams are expected to submit their top four cities predictions team will receive an additional prize. Paper submission and virtual
per each trip on the test set until January 28th 2021. The submission participation at the workshop are mandatory in order to be eligible
will be done in a csv file named submission.csv in the following for a prize.
format described in table 3:
7 RESULTS
Table 3: Submission format 820 participants have signed up for the challenge. After two months
of a contest, 97 of them performed a final submission, grouped in
utrip_id city_id_1 city_id_2 city_id_3 city_id_4 40 competing teams. Top 10 performing teams are listed in table 4.
The best performing team achieved Accuracy @ 4 of 0.5939, imple-
1000031_1 8655 8652 4323 4332 menting a blend of 3 different neural network architectures, using
Transformers, GRUs, and feed-forward MLP. Other solutions relied
utrip_id represents each unique trip in the test, and the rest of on Efficient Manifold Density Estimator, LSTM networks, Attention
the columns represent the city_id of the top four predicted cities. mechanisms, Lambdarank, and further state-of-the-art methods.
The top 10 teams will be invited to submit short papers (up to 4 The teams have submitted short papers and code repositories with
pages + references in ACM sigconf format2 ). The papers will include a detailed description of their solution methodology.
the team and the authors names, an abstract, a text describing the
method and the achieved score, and a link to their code repository REFERENCES
[1] Dmitri Goldenberg, Kostia Kofman, Javier Albert, Sarai Mizrachi, Adam Horowitz,
and refer to the Booking.com challenge in the following format: and Irene Teinemaa. 2021. Personalization in Practice: Methods and Applications.
Dmitri Goldenberg, Kostia Kofman, Pavel Levin, Sarai In Proceedings of the 14th International Conference on Web Search and Data Mining.
[2] Julia Kiseleva, Melanie JI Mueller, Lucas Bernardi, Chad Davis, Ivan Kovacek,
Mizrachi, Maayan Kafry, and Guy Nadav. 2021. Booking.com Mats Stafseng Einarsen, Jaap Kamps, Alexander Tuzhilin, and Djoerd Hiemstra.
WebTour 2021 Challenge. http://www.bookingchallenge.com 2015. Where to go on your next trip? Optimizing travel destinations based on
, In ACM WSDM Workshop on Web Tourism (WSDM WebTour’21), user preferences. In Proceedings of the 38th International ACM SIGIR Conference on
Research and Development in Information Retrieval. 1097–1100.
March 12th, 2021, Jerusalem, Israel. [3] Tsvi Kuflik, Catalin Mihai Barbu, Amra Delić, Dmitri Goldenberg, Julia Neidhardt,
Selected papers are expected to present their work in the work- Ludocik Coba, and Markus Zanker. 2021. WebTour 2021 Workshop on Web and
Tourism. In Proceedings of the 14th International Conference on Web Search and
shop (in a virtual format). The submitted papers will be evaluated Data Mining.
based on their clarity, novelty, and results presentation. Please con- [4] Sarai Mizrachi and Pavel Levin. 2019. Combining Context Features in Sequence-
tact wsdmchallenge@booking.com for any questions. Aware Recommender Systems.. In RecSys (Late-Breaking Results). 11–15.
5 EVALUATION CRITERIA
The goal of the challenge is to predict (and recommend) the final
city (city_id) of each trip (utrip_id). The quality of the predictions
is evaluated based on the top four recommended cities for each trip
2 https://www.acm.org/publications/proceedings-template
Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).