ACM WSDM WebTour 2021, March 12th, 2021 Jerusalem, Israel 21 Booking.com WSDM WebTour 2021 Challenge Dmitri Goldenberg Kostia Kofman Pavel Levin dima.goldenberg@booking.com kostia.kofman@booking.com pavel.levin@booking.com Booking.com, Tel Aviv, Israel Booking.com, Tel Aviv, Israel Booking.com, Amsterdam, Netherlands Sarai Mizrachi Maayan Kafry Guy Nadav sarai.mizrachi@booking.com maayan.kafry@booking.com guy.nadav@booking.com Booking.com, Tel Aviv, Israel Booking.com, Tel Aviv, Israel Booking.com, Tel Aviv, Israel Figure 1: Multi-Destinations trip recommendation bar on Booking.com website ABSTRACT a strategy for making the best recommendation for their next desti- The ACM WSDM WebTour 2021 Challenge focuses on a multi- nation [2]. Booking.com releases this unique dataset to encourage destinations trip planning problem, which is a popular scenario in the research on sequential recommendation problems [4]. The chal- the travel domain. The goal of the challenge is to make the best lenge is part of the WebTour 2021 ACM WSDM workshop [3] on recommendation of an additional trip destination. To encourage web tourism that will be held at the 14th ACM international 2021 research on this field, Booking.com provided a unique dataset based WSDM Conference. on millions of real anonymized bookings. More than 800 participants have signed up for the contest. Best Table 1: Dataset description performing team achieved Accuracy @ 4 of 0.5939, using a blend of Transformers, GRUs, and feed-forward multi-layer perceptron. Column Description Additional leading teams implemented advanced state-of-the-art user_id User ID solutions to tackle the problem. checkin Reservation check-in date checkout Reservation check-out date CCS CONCEPTS An anonymized ID of affiliate channel • Information systems → Personalization; Recommender sys- affiliate_id where the booker came from (e.g. direct, tems. 3rd party referrals, paid search engine, etc.) device_class desktop/mobile KEYWORDS Country from which the reservation was booker_country Personalization, Travel, Recommender Systems, Dataset, Challenge made (anonymized) hotel_country Country of the hotel (anonymized) city_id city_id of the hotel’s city (anonymized) 1 PROBLEM DESCRIPTION trip ID (a group of multi-destinations utrip_id Booking.com’s mission is to make it easier for everyone to experi- bookings within the same trip) ence the world. By investing in the technology that helps take the friction out of travel, Booking.com seamlessly connects millions of travelers with memorable experiences, a range of transport options, 2 DATASET and incredible places to stay. The dataset consists of over a million anonymized hotel reserva- Many of the travelers go on trips that include more than one tions, based on real data, is available on the challenge website1 destination. For instance, a user from the US could fly to Amsterdam and described in table 1. Each reservation is a part of a customer’s for 5 nights, then spend 2 nights in Brussels, 3 in Paris, and 1 in trip (identified by utrip_id), which includes at least four consec- Amsterdam again before heading back home. In this scenario, the utive reservations. There are 0 or more days between check-out users are offered personalized recommendations [1] for extending and check-in dates of two consecutive reservations. The evaluation their trip immediately when they make their booking, as shown in dataset is constructed similarly. However, the city_id of the final figure 1. reservation of each trip is concealed and requires a prediction. The goal of this challenge is to use a dataset based on millions of real anonymized accommodation reservations to come up with 1 https://www.bookingchallenge.com/ Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). ACM WSDM WebTour 2021, March 12th, 2021 Jerusalem, Israel 22 Dmitri Goldenberg, Kostia Kofman, Pavel Levin, Sarai Mizrachi, Maayan Kafry, and Guy Nadav Table 2: Challenge key dates Table 4: Top 10 performing teams When? What? Team Accuracy @ 4 December 1st, 2020 Challenge starts 1 NVIDIA RAPIDS.AI 0.5939 2 Synerise AI 0.5780 Test set release - January 6th, 2021 3 TEAM DASOU 0.5741 teams registration deadline 4 mbaigorria 0.5566 Challenge closes - 5 aprec 0.5557 January 28th, 2021 results submission deadline 6 hakubishin3 & u++ & yu-y4 0.5399 February 4th, 2021 Announcement on the winners 7 testing 0.5332 8 Alexander Makeev 0.5310 February 18th, 2021 Paper submission deadline 9 YiNet 0.5112 February 25th, 2021 Paper notifications 10 Marlesson - MARS-Gym 0.4958 March 4th, 2021 Camera ready submissions March 12th, 2021 Workshop day by using Top-4 Accuracy metric (4 representing the four suggestion slots at Booking.com website). When the true city is one of the top 4 suggestions (regardless of the order), it is considered correct. 3 CHALLENGE TIMELINE Key dates of the challenge are listed in table 2. 6 PRIZES To encourage research contributions, the top three performing 4 SUBMISSION GUIDELINES teams will receive Booking.com Travel Credits. The best paper The teams are expected to submit their top four cities predictions team will receive an additional prize. Paper submission and virtual per each trip on the test set until January 28th 2021. The submission participation at the workshop are mandatory in order to be eligible will be done in a csv file named submission.csv in the following for a prize. format described in table 3: 7 RESULTS Table 3: Submission format 820 participants have signed up for the challenge. After two months of a contest, 97 of them performed a final submission, grouped in utrip_id city_id_1 city_id_2 city_id_3 city_id_4 40 competing teams. Top 10 performing teams are listed in table 4. The best performing team achieved Accuracy @ 4 of 0.5939, imple- 1000031_1 8655 8652 4323 4332 menting a blend of 3 different neural network architectures, using Transformers, GRUs, and feed-forward MLP. Other solutions relied utrip_id represents each unique trip in the test, and the rest of on Efficient Manifold Density Estimator, LSTM networks, Attention the columns represent the city_id of the top four predicted cities. mechanisms, Lambdarank, and further state-of-the-art methods. The top 10 teams will be invited to submit short papers (up to 4 The teams have submitted short papers and code repositories with pages + references in ACM sigconf format2 ). The papers will include a detailed description of their solution methodology. the team and the authors names, an abstract, a text describing the method and the achieved score, and a link to their code repository REFERENCES [1] Dmitri Goldenberg, Kostia Kofman, Javier Albert, Sarai Mizrachi, Adam Horowitz, and refer to the Booking.com challenge in the following format: and Irene Teinemaa. 2021. Personalization in Practice: Methods and Applications. Dmitri Goldenberg, Kostia Kofman, Pavel Levin, Sarai In Proceedings of the 14th International Conference on Web Search and Data Mining. [2] Julia Kiseleva, Melanie JI Mueller, Lucas Bernardi, Chad Davis, Ivan Kovacek, Mizrachi, Maayan Kafry, and Guy Nadav. 2021. Booking.com Mats Stafseng Einarsen, Jaap Kamps, Alexander Tuzhilin, and Djoerd Hiemstra. WebTour 2021 Challenge. http://www.bookingchallenge.com 2015. Where to go on your next trip? Optimizing travel destinations based on , In ACM WSDM Workshop on Web Tourism (WSDM WebTour’21), user preferences. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval. 1097–1100. March 12th, 2021, Jerusalem, Israel. [3] Tsvi Kuflik, Catalin Mihai Barbu, Amra Delić, Dmitri Goldenberg, Julia Neidhardt, Selected papers are expected to present their work in the work- Ludocik Coba, and Markus Zanker. 2021. WebTour 2021 Workshop on Web and Tourism. In Proceedings of the 14th International Conference on Web Search and shop (in a virtual format). The submitted papers will be evaluated Data Mining. based on their clarity, novelty, and results presentation. Please con- [4] Sarai Mizrachi and Pavel Levin. 2019. Combining Context Features in Sequence- tact wsdmchallenge@booking.com for any questions. Aware Recommender Systems.. In RecSys (Late-Breaking Results). 11–15. 5 EVALUATION CRITERIA The goal of the challenge is to predict (and recommend) the final city (city_id) of each trip (utrip_id). The quality of the predictions is evaluated based on the top four recommended cities for each trip 2 https://www.acm.org/publications/proceedings-template Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).