The ACM WSDM WebTour 2021 Challenge organized by Booking.com focuses on applying Session-Aware recommender systems in the travel domain. Given a sequence of travel bookings in a user trip, we look to recommend the user's next destination. To handle the large dimensionality of the output's space, we propose a many-to-many RNN model, predicting the next destination chosen by the user at every sequence step as opposed to only the final one. We show how this is a computationally eficient alternative to doing data augmentation in a many-to-one RNN, where we consider every subsequence of a session starting from the first element. Our solution achieved 4th place in the final leaderboard, with an accuracy@4 of 0.5566.
• Information systems → Recommender systems; • Computing methodologies → Neural networks.
Session-Aware recommender systems model the sequential decision
process of a user in the context of a session, also considering past
user actions or attributes [
The ACM WSDM WebTour 2021 Challenge organized by
Booking.com [
Hidasi et al. [
This manuscript proposes an approach to handling a high dimensional output space in RNN based Session-Aware recommender systems. Our main contributions to this problem include extending the many-to-one configuration typically-used RNN-based recommender systems to the many-to-many configuration. Instead of predicting the next booking by a user given a sequence of bookings, we predict the next booking at every time step. We show how this is a computationally eficient alternative to doing data augmentation in a many-to-one RNN, where we consider every subsequence of a session starting from the first element. Our empirical results show how this model can outperform the many-to-one RNNs by a significant margin. We also show how this extension biases the learning problem towards shorter trips and propose a correction by weighting the model’s loss function.
We organize the rest of the paper as follows: In Section 2, we ifrst give an overview of the challenge, the dataset provided by Booking.com, and the evaluation metric to be optimized. We then describe the many-to-many RNN architecture we used. We focus on explaining our design choices’ motivations and the approach we followed for training and inference (in Section 3). Lastly, in Section 4 we discuss both the implementation details we used to iterate quickly and our experiments’ results. Our code and documentation are available at https://github.com/mbaigorria/booking-challenge2021-recsys. 2
The ACM WSDM WebTour 2021 challenge proposed by Booking.com is an instance of the multi-destinations trip planning problem. Using an anonymized dataset of bookings, the goal is to recommend the next possible destination to a user in the context of a session. 2.1
The dataset is composed of over a million anonymized reservations. Each hotel reservation is a part of a user’s trip (identified by utrip_id) and includes multiple features such as the city, country, check-in, and checkout times. In the test set, these trips include at least four consecutive reservations. An overview of the dataset statistics is shown in Table 1.
Type Users Sessions Cities Accomodations MinSessMioanx lenMgtehdian Train 200,153 217,686 39,901 1,166,835 1 48 5 Test 68,502 70,662 21,305 378,667 4 44 5 Table 1: Descriptive statistics of the training and test sets
For each session in the test set, the city and country of the last booking are concealed. However, at inference time, other variables such as the check-in day of the next booking are also available. 2.2
For every trip in the test set, the correct city to be recommended is concealed. The metric used to evaluate the recommendations was precision@4. We can understand this metric as the percentage of times the correct city was in the top 4 recommendations for each trip. For this problem, the metric is equivalent to the recall@4, since there is only one relevant item to recommend per trip. 3
We designed our approach to be sequence and distance aware. Predicting the next destinations is a problem that must capture the notion of physical distance between cities and the sequential nature of the user’s decision-making process. We should also be able to handle variable-length user trips. It is also worth remembering user trips may be censored, as users do not necessarily book every trip on the platform.
Given some highly dimensional categorical variables such as afiliate_id or city_id, we should also learn latent representations (also known as embeddings) to explicitly avoid using one-hot encodings. The large dimensionality of the output space can also be challenging. In this specific case, there are 39,901 possible cities to recommend. In the training set, only 52% of these cities appear as the last element of a trip. If we are not careful, any softmax-based model to predict the sequence’s final booking is unlikely to make all cities predictable.
In Section 3.1, we explain how we tackled these challenges, the problems and trade-ofs we faced, and some interesting future research directions.
concat
concat
3.1
Our neural network architecture can be divided into three components. First, our model concatenates the embeddings of all features for each user trip booking. These concatenated vectors are then fed into a many-to-many RNN encoder, outputting the next city’s probability mass function given the previous cities at every time step. We finally recommend the top 4 cities in the probability mass function of the final step. You can see our general architecture diagram in Figure 2. 3.1.1
Feature engineering and representation.
Feature engineering. Using the provided dataset, we created 14 features by using the available information of both the current and the next bookings. We were cautious to not introduce any type of data leakage. The variables used were: • City and country of the current booking. • Country in which the current booking made. • Country in which the next booking is made. • Check-in day, month, and year of the current booking. • Check-in day of the next booking. • Duration of stay (days) of the current booking. • Duration of stay (days) of the next booking. • Device class of the current booking. • Transition days between a checkout and the next check-in. • Current and next booking afiliate_id (e.g. direct, third-party referral, paid search engine, etc.).
The transition days between bookings can potentially help us capture censored bookings. The model can learn that an extended transition time could mean the user booked on another platform, depending on the city to be recommended.
Feature representation. We embed all categorical and numerical
features. Numerical features have a low cardinality, so they should be
easily learned by the model. This allowed us not to have to worry
about feature scaling. The embedding dimension of all numerical
features is set to 10 and for categorical features to 25 except for
the device_id (5) and the city_id (128). To input these features into
our model, we then concatenate all the embeddings at each time
step. We also tried to merge all feature embeddings through
multiplication, but our model degraded. This is similar to the findings
by Mizrahi et al. for a similar dataset [
subsequences Session length
Instead of using a many-to-one RNN, we decided to use a manyto-many architecture to predict the next booked city at each time step. This many-to-many architecture has the same number of parameters as the many-to-one architecture. We apply a decoder over all the hidden states for each sequence instead of just the last one. This is identical to training a many-to-one architecture with an augmented dataset containing all session subsequences beginning from the first element in a single batch. However, for a session of length , the many-to-one architecture must do forward passes with a number of sequential operations in O (2) to get all the hidden states. On the other hand, the many-to-many model computes the same hidden states but linearly in .
We apply recurrent dropout with a probability of 0.1 to both of
the recurrent encoders. We also use dropout on the concatenated
input embeddings with a probability of 0.3. This is done not only to
regularize our model [
We concluded that tying the city embeddings with the output layer does not only lead to a faster training time because the network has a lower number of parameters, but it also has a better performance across diferent encoder configurations. You can see Table 3 for more details. 3.2
We calculate the cross-entropy loss of the probability mass function over cities at every sequence step for each trip. Since we train the model in batches, we then average all these cross-entropy losses for every trip in a batch.
We have validated that the longer a user trip is, the easier it is for our model to predict the next booking. Table 2 shows the sequence length distribution for diferent datasets once we remove the last observation in every sequence. We do this since we do not 0.243 0.243 0.215 0.119 0.069 0.041 0.025 0.016 0.010 0.006 0.006 1,186,491 (>4x more) 217,573
When considering all subsequences, the session length distribution becomes more concentrated on shorter sessions. This type of data augmentation comes at a cost since the underlying data generating process may be diferent for short and long trips.
Shorter sequences will also dominate the gradient estimates for each batch. We can correct this bias by weighting our loss function’s cross-entropy outputs by the reverse cumulative frequency of the dataset sequence length. This way, each length’s sequences would have the same contribution to gradient updates as in the non-augmented dataset. Our experiments in Table 3 show weighting the loss function degraded the model’s performance. However, this could be related to how we distributed the trips in a batch to train the model. This will be further discussed in Section 4.2.
It is also worth pointing out that augmenting the dataset explicitly has the risk of data leakage, using a validation trip that our model has already seen during training time. The many-tomany model avoids this by design, processing all subsequences of a session in one forward pass inside a single batch. 3.3
We trained a stratified ten-fold average cross-validation ensemble
using the Adam optimizer [
We obtained some additional data by unifying the training and the test set. We were careful to split the dataset into folds with a similar sequence length distribution. We considered the last probability mass function at inference time and extracted the top 4 recommendations with the maximum probability, precisely as in a many-to-one model.
In this section, we dive into the implementation details and empirical evaluations of our proposed recommendation approach.
Our model was implemented in PyTorch [
We pre-loaded all batches in GPU, and when doing 10-fold crossvalidation, we were careful to keep the folds as balanced as possible, also shufling the batches at every epoch. Using a many-to-many RNN architecture instead of doing explicit data augmentation allowed us to process four times the amount of sequences at roughly the same computational cost.
Model Type Many To Many Many To One
GRU LSTM GRU LSTM
True False True False True False True False
UNWEIGHTED
WEIGHTED UNWEIGHTED
WEIGHTED
WEIGHTED UNWEIGHTED UNWEIGHTED
WEIGHTED UNWEIGHTED
WEIGHTED
WEIGHTED UNWEIGHTED
WEIGHTED UNWEIGHTED
WEIGHTED UNWEIGHTED UNWEIGHTED
WEIGHTED
Based on the implementation discussed in Section 4, we measured the performance of our model’s diferent configurations by building a new test set of 5,600 trips with 24,400 bookings from the training set. This test set had a similar trip length distribution as the challenge organizers’ original test set.
Our results confirm GRUs seem to perform better than LSTMs for this task. We can also see the benefit of using many-to-many RNNs, which outperform many-to-one RNNs across all configurations. Tying the encoder and decoder weights also led to higher performance.
It is natural to wonder why the weighted version of the model did not perform as well as the unweighted version. We believe that this is related to our implementation. When we sorted the trips to avoid zero-padding, we biased each gradient update’s mean and variance. Even if every batch has a fixed number of trips, this does not imply that it has the same number of augmented sequences. This can be solved using variable-sized batches, or avoiding sorting altogether.
One of the main challenges in Session-Aware recommender systems is efectively handling a large dimensional output space. This paper described how a many-to-many RNN could outperform the many-to-one configuration in such a setting. We showed how this extension is equivalent to doing data augmentation in a manyto-one RNN, with the pitfall of potentially biasing our gradient estimates. We finally showed the advantages of tying the feature embedding weights with the parameterization of the decoder.
The efect of having an uneven sequence length distribution in every batch and how to efectively handle the bias introduced by considering all subsequences in the many-to-many architecture can be an interesting future research direction. Attempting to do multi-task learning at every time step by also predicting other future attributes like the country might further boost our predictive performance.
In the dataset provided by Booking.com, around 12% of the users
had multiple bookings. We attempted to model this by
concatenating user trips with a separator token without success. This context
can potentially be modeled using Hierarchical RNNs, or
Transformer based architectures [
We want to thank Sole Pera, Roberto Martín Pozzer, Leonardo Baldassini, Fabricio Previgliano, and Charlie Giudice for their useful comments on an earlier version of this work. We would also like to thank the ACM WSDM WebTour 2021 workshop organizers for the opportunity to participate in such an exciting challenge.