Recommending destinations of trips based on user behavior is an important task for travel agencies such as Booking.com. The Booking.com Challenge - WebTour 2021 ACM WSDM workshop is aimed at building models for this task; the goal is to predict the final destination of multi-destination trips, based on a large dataset of over a million-trip reservations at Booking.com with date, destination, etc. In this paper, I present my approach where I leverage recent advances in language modeling techniques including Transformer. The approach, which used only a sequence of the visited cities for input, showed a top-4 accuracy of 0.4720 on the final result leaderboard. Full code is available here: https://github.com/sakatani/BookingcomChallenge2021.
• Information systems → Recommender systems; • Computing methodologies → Neural networks.
Booking.com is the world’s largest online travel agency, which
is being used by millions of users to find accommodations.
Recommending destinations based on user behavior is an important
task for travel agencies such as Booking.com [
Booking.com Challenge [
Every record in the data was a user’s reservation and contained
information such as city of stay, country of stay, reservation ID,
user ID, reservation date, check-in date, check-out date, afiliation
channel, and country where the reservation was made. That
information other than the city of stay was expected to play an important
role in the performance of inference, as shown in a previous study
[
The model used in my approach is illustrated in Figure 1, where the positional encoding of a Transformer is replaced with a single-layer LSTM.
For modeling multi-destination trips, an RNN-based model has
been proposed [
Although Transformer-based models have achieved a number of state-of-the-arts techniques in the field of natural language processing, they have one computational cost weakness against conventional RNN models when considering its use in the generative task for destination recommendation. As the Transformer does not preserve the hidden state unlike RNNs but uses positional encoding to represent the sequential nature of tokens, it is expected to be computationally more expensive than RNNs because it needs to recompute the entire history in the context window at each time step.
To address this problem, a model called LSTM + Transformer, in
which the positional encoding of Transformer is replaced with a
single-layer LSTM, has been proposed in a previous study [
All the experiments were conducted on Google Colaboratory and it was made sure that Intel(R) Xeon(R) CPU @ 2.30GHz, 250GB RAM, and NVIDIA Tesla T4 GPUs were allocated.
The LSTM-Transformer combined model consisted of one LSTM
layer and five Transformer layers, with 512 dimensions of the LSTM
hidden layer and Transformer, 1024 dimensions of the feedforward
Transformer layers, and eight attention heads. The batch size was
Sakatani, Y.
set to 16. The Adam optimizer was used during training. The
learning rate was linearly warmed up to 7 × 10−4 with 4000 iterations,
and then decayed in proportion to the inverse square root of the
iteration number according to the formula in [
In addition to the LSTM-Transformer combined model, a model
consisting of six layers of Transformers and a model consisted of
LSTM or GRU were also prepared for comparison. The six-layer
Transformer model was identical to the LSTM-Transformer model,
except that a Transformer layer was assigned instead of the LSTM
layer. The LSTM and GRU models have two layers and 512 hidden
units. The GRU model was based on the previous study on the
RNN-based multi-destination trip model [
The models were evaluated using perplexity, which was based on the negative log-likelihood loss, and top-4 accuracy on the local evaluation data according to the rules of the challenge. Figure 2 shows the perplexity during training on the training and validation datasets. The perplexity of each model on the training dataset decreased steadily with the number of iterations. On the validation dataset, both the perplexity of Transformer and LSTM-Transformer combined model decreased steadily until at least 350,000 iterations, while that of the GRU and LSTM model decreased shakingly and then increased after 50,000 iterations.
The perplexity and top-4 accuracy on the local evaluation dataset were then evaluated using the checkpoint with the minimum perplexity on the validation dataset (Table 2). The LSTM-Transformer combined model showed the lowest perplexity and highest top-4 accuracy. The Transformer model performed as well as the LSTMTransformer combined model. These results indicate that the replacement of the positional encoding of Transformer with the LSTM layer does not adversely afect the accuracy for this type of task, but rather it can have a positive efect on performance. The LSTM and GRU models showed smaller perplexities than the Transformerbased models during training on both the training and validation datasets, whereas they showed worse perplexities and top-4 accuracy on the evaluation data than the Transformer-based models. A possible explanation for this is that the RNN-based models might have been overfitting to the training or validation dataset.
Finally, the local evaluation dataset was split into 18% for validation and 82% for training, then the LSTM-Transformer model was re-trained with the datasets. The predictions submitted to the ifnal result leaderboard were generated by the retrained model and showed 0.4720 for the top-4 accuracy score.
While this is irrelevant for the evaluation of the challenge, the
time required for each model to generate tokens for 10000 steps
was measured three times on both GPU and CPU (Table 2). The
processing time of the LSTM-Transformer combined model was
reduced to 59% of that of the Transformer model on the CPU and
90% on the GPU. These results indicate that replacing positional
encoding of Transformer with LSTM reduces the computational cost.
The processing time on GPU, however, was not reduced as much as
on CPU. The LSTM-Transformer model may have some processing
bottlenecks on GPU, as pointed out in a previous study[
Perplexity
Top-4 accuracy 4
In this paper, I describe my approach to the Booking.com Challenge WebTour 2021 ACM WSDM workshop. The approach showed the LSTM substitution of positional encoding can have a positive efect on both the prediction and computational performance.
Because of time constraints, the approach did not make use of
any information other than the visited destinations for building
the models. Although the LSTM-Transformer model used only
sequences of the visited cities for input, it achieved the 13th score
on the final result leaderboard. A previous study has shown that
the prediction performance of RNN models can be improved by
combining contextual information such as users’ home country
with a sequence of visited destinations [
The efect of replacing the positional encoding with LSTM on the computational cost was briefly investigated, but not fully explored in this study. It is expected that the LSTM-Transformer model reduces the computational cost especially when multiple destinations are predicted as the model has the advantage that the expansion of the context window does not afect the computational cost for the input. Another future work involves an investigation on how replacing positional encoding with LSTM afects the computational cost of Transformer-based models in generative tasks for destination recommendation. steps.
192.5 (2.0) 324.5 (3.8)