Next destination prediction problem by users' traveling sequence has provoked many researchers' attention recently. It is a perfect benchmark for a graph neural network or graph embedding algorithm, which have achieved state of the art for many kinds of tasks involving graph information. In this paper, we explore this problem by recurrent neural network and we have achieved 0.5741 top-4 accuracy.
In this paper, we will introduce our model and method employed
in the WebTour 2021 Challenge [
As the preprocessing, We pad/crop users’ destination sequences to
the same length 20 in order to use recurrent neural network. User
sequences with length less than 2 are removed since they will not
appear in our testing data. We use the data from both training data
and test data for Word2Vec[
The structure [Figure:1] of our model is shown blow. The input
of our model are destination sequences and manual features. We use
LSTM [
= ( [, ( [1, 2, ..., ])]) where are embedding vector of diferent cities.
We have 2 outputs, next destination and next country. The extra country information introduced here help us to decrease the loss by a lot. We apply softmax activation function to the output and use cross-entropy to optimize our model. 2.1
Since Booking.com allows us to use part of the information about the next destination, we made some manual features concerning the check-out date of the last destination, check-in date of the next destination and the duration the traveler will stay. The extra features improve our top-4 accuracy by 1.5%.
Our features include the features concerning time, • Duration of last trip. • Day of the month for check in. • Day of the week for check in. • Day of the month for check out.
• Day of the week for check out.
The sinus and cosinus value of the time are used. For example, day of the week sin = sin(day of the week/7) day of the week cos = cos(day of the week/7) Other features like afiliate_id, device_class and booker_country are also included. The source is processed by a trainable embedding layer. The additional information improved the final result by 1.5%.
We use a 2 layers MLP to process the input of features. It is then concatenated to the output from RNN. We make the final prediction by a densely connected layer, with the activation function softmax. All the cities are treated as the promising destination (this part can be improved by carefully select positive and negative samples).
Our training strategy is 20 k-fold cross-validation with learning rate decay strategy, ’reduce lr on plateau’. We trained our model with Adam optimizer with learning rate 0.001 and batch size 8096. What contributes most to our final result is the two phase training strategy. 3.1
Pretraining has been proved to be eficient and efective for many tasks. Although RNN is not as capable as transformer, we can still improve its performance by using
We pretrain our RNN model by using the sub-sequences from the complete sequence. Assume that the complete sequence is, (0, 1, 2, ..., ) (0), (0, 1), (0, 1, 2), ..., (0, 1, 2, ..., −1)
where denotes the destinations. 3.2
Pretraining can also be problematic because some intermediate destinations are not directly connected with user’s origin. Thus some sub-sequences are noisy and the data distribution of subsequences is diferent from that of the original data. To handle this diference of data distribution, we only need to fine-tune our model with the original training data.
The following [Table 1] is the comparison of top-4 accuracy for local validation data between diferent settings.
LSTM+features+pretrain+finetune+test data top-4 accuracy 52.51% 54.02% 54.95% 55.53% 57.49%
In this paper, we start from a very simple RNN model for next destination prediction. The main improvements are made through understanding the data and making the most of the data through data exploration. We have shown that pretraining for sequential recommendation system is beneficial and fine-tuning can improve the performance further. Our final result is competitive to SOTA.
Yuanzhe Zhou, Shikang Wu, and Chenyang Zheng