Deep learning based methods have been used successfully in recommender system problems. Approaches using recurrent neural networks, transformers, and attention mechanisms are useful to model users' long- and short-term preferences in sequential interactions. To explore diferent session-based recommendation solutions, Booking.com recently organized the WSDM WebTour 2021 Challenge, which aims to benchmark models to recommend the final city in a trip. This study presents our approach to this challenge. We conducted several experiments to test diferent state-of-the-art deep learning architectures for recommender systems. Further, we proposed some changes to Neural Attentive Recommendation Machine (NARM), adapted its architecture for the challenge objective, and implemented training approaches that can be used in any sessionbased model to improve accuracy. Our experimental result shows that the improved NARM outperforms all other state-of-the-art benchmark methods.
• Computer systems organization → Embedded systems; • Information systems → Recommender systems; • Computing methodologies → Sequential decision making.
Deep learning approaches for recommender systems have garnered
significant attention owing to their potential for modeling
longand short-term user preferences. Models using recurrent neural
networks (RNNs), transformers, and attention mechanisms are handy
for text, presenting encouraging results when used in session-based
recommendation problems [
In general, session-based approaches use only the information
from the item’s interaction sequence in the session as a predictor of
the next item. However, in some domains, contextual information
within and between sessions and global information are essential
to modeling user behavior. Recommending travel destinations has
several particularities, such as the sequence of cities on a trip can
contain noise, financial constraints may lead to diversions, and
destinations are highly correlated to the moment in time and duration
of the trip [
The WSDM WebTour 2021 Challenge [
Booking.com is the world’s largest online travel agency. It is a
platform where millions of travelers find accommodations for their
trips, and millions of accommodation providers list their hotels,
apartments, guest houses, and other lodgings. [
Booking.com recently organized the WSDM WebTour 2021
Challenge [
This section describes each approach used in the present study to improve our final model. First, we chose a state-of-the-art sessionbased model to improve (described in 3.1). Further, we created new features from statistical information about users and cities and added time modeling to focus on the travel problem particularity (described in 3.2 and 3.3). Also, we have reduced the efect of the imbalance dataset using specific loss functions and multitask modeling (described in 3.5 and 3.4). Finally, we improve the generalization of noise information through data augmentation (described in 3.6).
Each approach can be applied separately in other session-based models. 3.1
We use the architecture of the neural attentive recommendation
machine (NARM) [
Our approach uses categorical and dense features of the user, city, and trip combined with the trip history. The core of the NARM module is the same as the original paper, and we just changed the size of the inputs, the bottleneck with hidden representation, and the outputs.
The features are concatenated and follow two paths. The first group of features passes through an attention layer before being fed into the NARM module. It is an important step for relating diferent positions of the same input sequence. In the second path, the features bypass the feature bottleneck generated by the NARM module, which improves the decoder by providing it with more contextual information on the session.
Figure 1 shows the final architecture proposed with modifications described in this study. 3.2
Our approach uses statistical features from users, cities, and trips combined with the trip’s history information. For every trip, we extract more than 30 statistical features. Below we provide examples of the important features we use in the present study: User Statistics • Number of trips • Number of cities visited • Average number of unique cities visited per trip • Average trip size • Average trip duration • Most frequently traveled month • Number of unique cities the user visited • Number of unique countries the user visited • Device used for booking • Booker country City Statistics • City interactions that have preceded the current trip • City interactions that have preceded the current trip by user • City interactions in the current trip • Country interactions that have preceded the current trip • Country interactions that have preceded the current trip by user
Furthermore, we employ user statistical features to create user embedding from an autoencoder trained on the same training split data. This approach proved helpful because only 0.7% of the users have made more than one trip; the vast majority are new users. In addition to providing a dense representation of the user features, the autoencoder ensures that similar users are close to each other in the created vector space. Moreover, the autoencoder was trained separately from the recommendation model. Figure 2 shows user embeddings colored by the most frequently traveled month. 3.3
In terms of tourism, a trip’s location is merely a piece of information that can tell how good the place is for a trip. The date (moment in time) and duration of the trip are as crucial as, for example, sometimes, the traveler can only visit certain places only in specific seasons or specific time windows. Sometimes, it is not feasible to stay for a few days more.
Embeddings are highly useful for modeling categorical features
such as the location, and RNNs are remarkably good for capturing
sequential data’s time-dependence. However, they are limited when
the events have irregular time intervals, such as duration of stay on
a specific city trip. Therefore, we need to incorporate time interval
explicitly into the model as a time-space embedding. In this case,
we use the approach presented in [
Therefore, each city embedding presented in a trip follows through a time embedding layer. We incorporate time (start trip month) and duration (how long users stayed in a city) information. In the end, we obtain a space-time embedding. Hybrid Model with Time Modeling for Sequential Recommender Systems 3.4
The data about the cities in our dataset are imbalanced and have
high dimensionality. In contrast, the data about countries are less
imbalanced and have lesser dimensionality. Thus, we adapt our
model to predict both to try to improve models’ gradient signal.
According to [
We used two targets in the present study. The city of the next step interaction was the main target that we used to evaluate the model, and the city’s country was used as the second target. The second target was used only for regularization and as inductive bias in the primary target. Both targets use the same loss function, and the final loss is a combination of the two. 3.5
Focal loss [
Formally, the focal loss is expressed as follows:
() = − (1 − ) () where is adjusts the rate at which easy examples are downweighted and is a prefixed value between 0 and 1 to balance the positive-labeled samples and negative-labeled samples. We use Equation 2 for calculation loss.
( , ) = ( ) + (1 − ) ( ) where is a balance parameter, and and are obtained from the target prediction. (1) (2) 3.6
Data augmentation techniques have been widely used to enhance
image [
The first is a sequence preprocessing step proposed in [
For each sample in training, we randomly choose a step in the trip to change. Given an input training trip [1, 2, 3, 4], we change it in three ways: remove a step as a dropout layer and generate sequences such as [1, 2, 4]; replace with a mask token (unknown) and generate sequences with the same size but with mask [1, 2, 3, ] ; and replace with a similar city, such as [1, 2, 3, 4] where 2 and 2 are similar cities. The mask token will be the same as that used in production/inference mode when the model does not embed a specific city, and the similarity definition is the same as that we used for the Item-KNN model.
We apply both methods to make the model less susceptible to overfitting or noise information. 4
In this section, we present the experimental settings and results. 4.1
We randomly partition the training dataset by trip, using 90% of
data for training and 10% for model validation. We filtered trips
with less than four cities or more than 10 and trips with a duration
of more than 22 days in the training split. These values came from
the initial analysis and are outliers. Table 1 shows the statistics of
the dataset used in our experiments.
• Popularity: Popular predictor recommends the most
popular city in last city’s country on the current trip.
• Item-KNN: This baseline recommends the most similar city
to the last city on current trip. The similarity is defined as
the cosine distance between the trip vector of the city. It is
similar to the co-occurrence approach. [
We choose the NARM model, which shows the best performance among the models listed above, such as our baseline model to improve during the competition. Therefore, our approach is an adaptation of the NARM model. Some approaches were modeled for the click prediction problem. For our experiments, we adapted the last layer of all deep learning models to produce a ranking list of all items, = [1, 2, 3, ..., ], that can occur on the current trip. Furthermore, we trained our model using the same loss function. 4.3
All our models were trained using similar parameters. We used 50-dimensional embeddings for all category features, and we normalized dense features using standard deviation. Optimization was carried out using Rectified Adam (RAdam), with a 0.001 learning rating and 0.01 weight decay, with mini-batch size fixed at 64. We truncated the trip history size using a fixed window of 10 time-steps with padding. The number of epochs was defined by early stopping using 10 steps, and we used cross-entropy as a loss function for most models or focal loss with 1 and 3 .
Finally, we use the MARS-Gym framework [
Each user’s embeddings are presented in the 2-D plane in Figure 2. We can see that the month is a good predictor of user behavior, and, in general, users who travel in the same month also share similarities. We can also notice that there is a mixed cluster in the center, maybe belonging to users who are likely to travel on diferent dates with greater frequency. Furthermore, we can use this representation for new users to improve a cold-start recommendation.
Each model’s performance on the test dataset is summarized in Table 2. We can see that the Item-KNN method performs similarly to the Popularity method, which is a cold start method. Therefore, there is no benefit in adopting the Item-KNN method given the complexity posed by cold recommendations.
SASRec and Caser are very diferent approaches, the former based on attention mechanism and later on horizontal and vertical convolutions. Both showed similar results, with Caser performing slightly better. However, the best base baseline model was NARM. We can see that the original NARM outperformed state-of-the-art baselines using the same input information. Therefore, we chose NARM in this study to improve the architecture and training phase.
We evaluate two versions of NARM. In NARM V1, we use only improvements that were applied in the training and regularization steps, with the same input data were used for other baseline models, but applying approaches 3.3, 3.4, 3.5, 3.6. NARM V1 outperforms the original NARM with approximately +4.62% accuracy. This is a promising finding, as this indicates that these techniques can now be used for any other session-based models that are in need of improvement.
Finally, we apply all approaches present in this study to NARM V2, and it represents our final approach for the challenge. Figure 1 shows our final architecture. We found that NARM V2 outperforms the original NARM with approximately +9.66% accuracy. This supports the idea that when it is not possible to obtain an item or user metadata, session statistics can also be used as features. 5
In this study, we present our approach for the WSDM WebTour 2021 Challenge. We conducted several experiments using diferent session-based models to recommend the next destination on a trip. We modified the existing NARM model to add contextual information to the session, space-time modeling, and approaches to reduce the negative efect of class imbalance in the training phase. Our results show that it is possible to enhance the performance of stateof-the-art models through simple changes. The improved NARM outperforms all baseline models, and that the implemented training approaches can be used in any session-based recommendations model. Hybrid Model with Time Modeling for Sequential Recommender Systems