This paper describes the 6th place approach to Booking.com WSDM WebTour 2021 Challenge, which is a challenge with a task of predicting travellers' next destination. We, in the team "hakubishin3 & u++ & yu-y4", trained four types of Long short-term memory (LSTM) models, and achieved the final score: 0.5399 by weighted averaging of these predictions. There are some diferences in these models in feature engineering, multi-task learning, and data augmentation. Our experiments showed that the diversity of the models boosted the final result. Our codes are available at https://github. com/hakubishin3/booking-challenge-2021 and https://github.com/ upura/booking-challenge-2021.
• Information systems → Information systems applications; Recommender systems;
Booking.com is one of the world largest online travel agencies. Its
mission is to make it easier for everyone to experience the world,
and it seeks the way of using information technology that helps
reduce the time and efort regarding travel [
In 2021, Booking.com published dataset and organized a challenge with a task of predicting travellers’ next destination. In this challenge, participants consider a scenario in which the users of Booking.com make a reservation and immediately suggest options for extending their trips.
The rest of this paper is organized as follows. In section 2, the overview of the challenge is described. Before our own approach, section 3 describes the previous works by the organizer that we followed. From section 4 to 6, we present our solution step by step. Each section shows the architecture of the neural network models, the input features, and the training methods. In section 7, we report the experimental results of our proposed method. The final section provides a conclusion of this paper. ∗These authors contributed equally to this work. The goal of the challenge is to develop a strategy for making the best recommendation of cities as a travellers’ next destination. The quality of the recommendations are evaluated by using Precision@4 metric. In other words, it is considered correct when the true city is one of the top four suggestions for each trip. 2.2
The training dataset consists of over a million of anonymized hotel reservations, with the following columns: • user_id: User ID • check-in: Reservation check-in date • checkout: Reservation check-out date • afiliate_id : An anonymized ID of afiliate channels where the booker came from (e.g. direct, some third party referrals, paid search engine, etc.) • device_class: desktop/mobile • booker_country: Country from which the reservation was made (anonymized) • hotel_country: Country of the hotel (anonymized) • city_id: city_id of the hotel’s city (anonymized) • utrip_id: Unique identification of user’s trip (a group of multidestinations bookings within the same trip)
The evaluation dataset is constructed similarly, however the city_id and hotel_country of the final reservation of each trip are concealed. We are required to predict the city_id.
The distribution of the city_id appearing in the dataset is longtailed. Though there are about 40,000 candidates, we can achieve the score of 0.036 by just suggesting the top four most frequent cities in the dataset. 3
The organizer’s previous works [
Some beneficial ideas are also explained in the previous works.
First, the experiment showed that adding features other than the
series of city_id greatly improved the performance[
We prepared two types of model architecture. Both architectures are described in this section. 4.1
As RNNs, we adopted Long short-term memory (LSTM) [
We also used a LSTM model with another type of structure, shown
in Figure 2. This is an extension of the model with the concept of
multi-task learning (MTL). MTL is a training paradigm in which
machine learning models are trained with the dataset from multiple
tasks simultaneously [
In this challenge, we built an architecture that predicts not only the city_id but also the hotel_country at the same time. Since there is an inclusion relationship between hotel_country and city_id, we thought that the prediction of hotel_country would also contribute to the quality of the prediction of city_id. To predict hotel_country is an easier task than city_id. When a hotel_country is given, the candidates of city_id are limited. This can help our model to predict a correct city_id. 5
We generated some features from the original dataset, and removed some of the original dataset. Features can be divided into three types, categorical, numerical, and graphical variables. Each type of feature is briefly summarized in this section. 5.1
There are seven diferent categorical features in the dataset, and six of them are used in the baseline as described in section 3. The only removed one is utrip_id, a unique identificator.
We added some categorical features. The followings are categorical features we used.
• month_checkin: Month of check-in. • past_city_id: Previous city_id.
• past_hotel_country: Previous hotel_country. 5.2
We extracted some numerical features. The days_stay is a feature that uses future information, we should be careful. In this challenge, check-in and checkout date of the targets are given. Therefore, days_stay which should not be available practically can be calculated, which is a useful feature since it indicates the characteristics of the targets.
• days_stay: The number of days staying in a current hotel. • days_move: The number of days from a previous hotel checkout date to a current hotel check-in date. • num_checkin: The number of check-in within utrip_id. • num_visit_drop_duplicates: The number of unique city_id within utrip_id. • num_visit: The number of city_id within utrip_id. • num_visit_same_city: The number of duplicated city_id within utrip_id. • num_stay_consecutively: The number of consecutive stay in the same city_id.
Each sequence of trips are just a fragment of it. We believe that
graph related features are important because it can lead to
reconstruct geographical information. We believe that the graph
related features are important because they can lead to reconstruct
geographical information. We used Word2Vec [
Figure 3 is a scatter plot of city_id vectors calculated by Word2Vec
compressed by UMAP [
CrossEntropyLoss is commonly used in multi-classification tasks.
However, the classes of city_id that are required to be predicted
in this challenge are imbalanced with long-tailed distribution. In
order to suppress the bias of the loss caused by the class imbalance,
we adopted FocalLoss [
We chose Stratified K-Folds so that the distribution of the length of trips in each fold is equal. This is because there is a diference in the length of trips shown in Figure 4, which may afect the quality of the prediction. The number of folds was set to five. We trained 15 epochs in each fold, and used a model with the best validation score for predictions.
We believe that each sequence of trips can be flipped. In applying this data augmentation, booker_country must be removed in order to keep consistency of the dataset.
In this challenge, we can use the information of the evaluation dataset. The final city of each trip in the evaluation dataset is masked to be used for evaluation. Therefore, we created an additional dataset from the evaluation dataset by regarding the second city_id from the final reservation as the target of each trip and added it to the training data.
Table 1 shows the settings of each model. Each model was composed of the combination of the proposed architecture, features, and the training methods.
MTL architecture was used in LSTM 2 and 3. Graph features and loss function were diferent in LSTM 4. Data augmentation techniques were applied in LSTM 3 and 4.
Table 2 shows the results of our experiments for each model. The validation scores were calculated by averaging the predictions of all folds.
We can see that our proposed models outperformed the baseline. From the comparison between LSTM 1 to 3, it was observed that MTL worked in our experiment, and data expansion by flipping didn’t work. LSTM 4 gave us the best result.
A weighted averaging was used for the ensembling. In this challenge, we decided to use it because of the small opportunity of submissions and the high computational cost. Table 2 shows that the diversity of the models leads to higher scores. It is interesting
The four numbers in Weights mean the ratio of LSTM 1 to 4 respectively. to note that even though LSTM 4 performed well on its own, the best result was obtained when the ratio was reduced to 0.4. 8
This paper described our approach to Booking.com WSDM WebTour 2021 Challenge. We trained four types of LSTM model, and weighted averaging of these predictions increased the accuracy. There is a diversity regarding feature engineering, multi-task learning, and data augmentation. In the end, we won the 6th place on the final leaderboard of this challenge.
We thank the organizers of the Booking.com WSDM WebTour 2021 Challenge for the opportunity to participate in this interesting challenge.