Recommending a hotel for vacations or a business trip can be a challenging task due to the large number of alternatives and considerations to take into account. In this study, a recommendation engine is designed to identify relevant hotels based on features of the facilities and the context of the trip via flight information. The system was designed as a cascaded machine learning pipeline, with a model to predict the conversion probability of each hotel and another to predict the conversion of a set of hotels as presented to the traveller. By analysing the feature importance of the model based on sets of hotels, we are able to construct optimal lists of hotels by selecting individual hotels that will maximise the probability of conversion.
• Computing methodologies → Machine learning;
In the United States, the travel industry is estimated to be the third
largest industry after the automotive and food sectors and
contributes to approximately 5% of the gross domestic product. Travel
has experienced rapid growth as users are willing to pay for new
experiences, unexpected situations, and moments of meditation
[
Finding the best alternative can become a complicated and
timeconsuming process. Consumers used to rely mostly on
recommendations from other people by word of mouth, known products from
∗Both authors contributed equally to this research.
advertisements [
Many players have emerged in the past decades mediating the communication between the consumers and the suppliers. One type of player is the Global Distribution System (GDS), which allows customer-facing travel agencies (online or physical) to search and book content from most airlines and hotels. Increased conversion is a benefical goal for the supplier and broker as it implies more revenue for a lower cost of operation, and for the traveller, as it implies quicker decision making and thus less time spent on search and shopping activities.
In this study, we aim to increase the conversion rate for hospitality recommendations after users book air travel. In Section 2, the problem is formulated in order to highlight the considerations which separate this work from many recommender system paradigms. Section 3 presents the main techniques and concepts used in this study. In Section 4, a brief overview is given of the industry data used in this study. Section 5 discusses the results obtained for diferent machine learning models including feature analysis. A discussion of the main outcomes of this study is provided in Section 6. 2 2.1
Booking a major holiday is typically a yearly or bi-yearly activity for
travellers, requiring research for destinations, activities and pricing.
According to a study from Expedia [
Researchers have tried to relate touristic behavioural patterns
to psychological needs and expectations by 1) defining a
characterization of travel personalities and 2) building a computational
model based on a proper description of these profiles [
In order to clearly define our goal, let us first define some terminology: • Hotel Conversion: a hotel recommendation leads to a conversion when the user books a specific hotel. • Hotel Model: machine learning model trained to predict the conversion probability of individual hotels. • Passenger Name Record (PNR): digital record that contains information about the passenger data and flight details. • Session: after a traveller completes a flight booking through a reservation system, a session is defined by the context of the flight, the context of the reservation, and a set of five recommended hotels proposed by the recommender system. • Session Conversion: a session leads to a conversion when the user books any of the hotels suggested during the session. • Session Model: machine learning model trained using features related with the session context and hotels, its output is the conversion probability of the session.
The end goal of the recommender system is to increase session conversion. We can estimate the probability of booking of a list of hotels using the session model, and thus we can compare diferent lists using the session model to determine the one which will maximise the probability of conversion of the session. Note that in this case conversion is defined as a selection or "click" of a hotel on the interface, rather than a booking. 2.3
The content sold through a GDS is diverse, including flight segments, hotel stays, cruises, car rental, and airport-hotel transfers. The core GDS business concerns the delivery of appropriate travel solutions to travel retailers. Therefore, state-of-the-art recommendation engines capable of analysing historical bookings and automatically recommending the appropriate travel solutions need to be designed. Figure 1 shows an outline of the rule-based recommendation system currently in use. After a user books a flight, information related to the trip is sent to the recommender engine.
However, this system does not take into account valuable information such as the context of the request (e.g. where did the booking originate from?), details about the associated flight (e.g. how many days is the user staying in the city?) nor historical recommendations (e.g. are similar users likely to book similar hotels?), which are key assets to fine tune the recommendations.
The problem is novel due to the richness of available data sources
(bookings, ratings, passenger information) and the variety of
distribution channels: indirect through travel agencies or direct
(website, mobile, mailbox). However, it is important to consider that
by design, no personally identifiable information (PII) or traveller
specific history is used as part of the model, which therefore
excludes collaborative-filtering or content-based approaches. The
contributions of this work are:
• The combination of data feeds to generate the context of
travel, including flights booked by traveller, historical hotels
proposed and booked at destination by other travellers, and
hotel content information.
• The definition of a 2-stage machine learning recommender
tailored for travel context. Two machine learning models are
required to build the new recommendation set. The output
of the first machine learning algorithm (prediction of the
probability of hotel booking) is a key input for the second
algorithm, based on the idea of [
Using machine learning and the historical dataset of recommendations, we can train a model which is capable of predicting with high confidence whether a proposed set of recommended hotels leads to a booking.
Once we have fit the model, we can evaluate other combinations of hotels and recommend a list of hotels to the user that maximizes the conversion. Instead of proposing a completely new set of hotels, we decide to modify the existing suggestions given by the existing rule-based system. Our approach, shown in Figure 2, removes one of the initial hotels and introduces an additional one that increases the conversion probability:
We have identified two diferent ways to select the hotel that is going to be introduced within the set of recommendations: • We can create and evaluate all possible combinations and choose the one with the highest conversion probability. This means, each time one out of the five hotels from the initial list is removed, and a new one from the pool of hotels is inserted. However, this brute force solution is computationally ineficient and time-consuming (e.g., in Paris this results in 5*1,653 diferent combinations for a single swap, the length of the list multiplied by the number of available hotels). • Alternatively, a hotel from the list of selected hotels can be replaced with an available hotel, based on some criteria. Typically, the criteria might be the price of the hotel room, or the average review score, or a combination of multiple indicators. In this work, the criteria used to optimise the overall list of hotels is determined via feature analysis.
Nevertheless, the last solution presents some challenges that need to be discussed and solved: (1) How to study the feature importance of complex non-linear models? (2) How to best interpret the feature importance in an unbalanced dataset? (3) How many features should be used during the selection process of building an optimal list? Initially, we are facing a multi-objective optimization problem since the choice of a hotel for enhancing the conversion probability might depend on diferent features. Furthermore, the existence of categorical features makes this optimization even harder. Can we convert it into a univariate optimization problem?
The novelty of this study comes from the use of two related works
to address the above points. First, we design a two-stage cascaded
machine learning model [
Ensembling techniques consist in combining the decisions of
multiple classifiers in order to reduce the test error on unseen data. After
studying the bias-variance decomposition of the error in bagging
and boosting, Kohavi observed that the reduction of the error is
mainly due to reduction in the variance [
There are several advantages of using cascade generalization over other ensemble algorithms: • The new attributes are continuous since they are probability class distributions. • Each classifier has access to the original attributes and any new attribute included at lower levels is considered exactly in the same way as any of the original attributes. • It does not use internal cross validation which afects the computational eficiency of the method. • The new probabilities can act as a dimensionality reduction technique. The relationship between the independent features and the target variable are captured by these new attributes.
As will be shown in further sections, this last point is a key aspect of the proposed system, as the probabilities generated by the hotel model can be used to directly select new hotels to include in the recommendation. However, the session model uses aggregated features from the hotel model, and as such an interpretable feature analysis is required to determine how best to select hotels based on their features. 3.3
Machine learning has grown in popularity in the last decade by
producing more reliable, more accurate, and faster results in areas
such as speech recognition [
In this section, we focus on the work from Ribeiro et al. [
In practice, LIME creates interpretable explanations for an individual sample by fitting a linear model to a set of perturbed variations of the sample and the resulting predictions as output from a complex-model. 3.5
The selection of which machine learning model to use highly depends on the problem nature, constraints and limitations that are trying to be solved. In this work, algorithms from diferent families of machine learning were investigated. Specifically, the Naive Bayes Classifier (NB) and Generalised linear Model (GLM) were investigated as linear models, Random Forests (RF), Gradient Boosting Machines (GBMs) were used to evaluate Decision Tree based ensembles and fully connected Neural Networks (NN) were also assessed. Furthermore, the model ensembling technique of Stacking (STK) was also assessed. Stacking comprises of learning a linear model to predict the target variable based on the output probabilities of multiple machine learning algorithms as features. 3.6
The first step is to train a machine learning model on individual
hotels, as shown is Figure 3. The features used for training this
model are not exclusively related to hotels, but also with the session
and flight context. Evaluating this model, we get the probability
that a certain hotel will be booked for a given location. The model
is learned by framing the problem as a supervised classification
problem, using the conversion (i.e. click) as a label. Note that for the
hotel model, the probabilities of conversion are independent of other
hotels presented in the session. This leads to several advantages:
• Cold start problem: the model does not penalise items or
users that have not been recommended yet, since no hotel
identifier or personally identifiable information is used. [
Note that the features used as input to the hotel model are discussed in Section 4. The second machine learning model predicts whether a session leads to a conversion or not, see Figure 4. A session is composed of five diferent hotels and the aim of the recommender system is to propose a set of hotels that results in the user booking any one of them. Aggregates of the features from the Hotel Model (contextual, passenger, and hotel features) are used, as well as the hotel probabilities obtained from the hotel model. The numerical features related with the hotels are aggregated in diferent ways (max, min, std and avg of price and probability for example). The features related with the context do not change (e.g. attributes about the session or the flight) as these are identical for each element in the session. The Session Model estimates the conversion probability of the session using contextual and content information. Thus, part of the session builder is to create and evaluate new lists of hotels to determine whether these lists will result in higher conversion probability than the original list. Figure 5 shows how this process is performed. First, a reference session with the recommendations, given by an existing rule based system, is scored. For each of the proposed hotels, we estimate the booking probability of each hotel using the Hotel Model. Next, we can calculate the booking probability at session level, using the probabilities of the Hotel Model as an input feature of the Session Model. Then, we aim to improve the conversion probability of the session by removing one of the hotels from the list and introducing a new one. After including the new hotel, if the booking probability of the current session is greater than the probability of the previous session, then this new hotel list is the one that will be proposed to the user.
A rule must be defined to select the hotel to remove and which
new hotel to introduce in the recommendation list. Once we have
trained the Session Model, we can analyse the feature importance of
the variables for the positive cases that were correctly classified (i.e.
true positive cases). With the Local Interpretable Model-Agnostic
Explanations model [
Note that the LIME analysis is performed only on true positive cases from the training set. In this dataset, the classes are highly imbalanced due to a low conversion rate, as such standard feature analysis techniques may be overly influenced by negative samples, i.e., sessions which did not result in clicks. As LIME is designed to be used on individual decisions, a linear model is fitted and analysed for each true positive. The feature weights for each linear model are then averaged, given a feature importance ranking for all correctly classified converted sessions. 3.9
As with many conversion problems, the classes are highly imbalanced, and as such the metrics used to assess performance must be carefully chosen.
F-measure (Fβ ). The generalization of the F1 metric is given by
[
β 2P + R β is a parameter that controls a balance between precision P and recall R. When β = 1, F1 comes to be equivalent to the harmonic mean of P and R. If β > 1, F becomes more recall-oriented (by placing more emphasis on false negatives) and if β < 1, it becomes more precision oriented (by attenuating the influence of false negatives). Common used metrics are the F2 and F0.5 scores.
Area Under the ROC curve. The receiver operating characteristic (ROC) curve is created by plotting the true positive rate (TPR) against the false positive rate (FPR) at various threshold levels. However, this can present an optimistic view of a classifier performance if there is a large skew in the class distribution because the metric takes into account true negatives.
Average Precision (AP). The precision-recall curve is a similar evaluation measure that is based on recall and precision at diferent threshold levels. An equivalent metric is the Average Precision (AP) which is the weighted mean of precisions achieved at each threshold, with the increase in recall from the previous threshold as the weight:
AP = Õ
(Rn − Rn−1)Pn n
Precision-recall curves are better for highlighting diferences
between models for unbalanced datasets due to the fact that they
evaluate the fraction of true positives among positive instances. In
highly imbalanced settings, the AP curve will likely exhibit larger
diferences and will be more informative than the area under the
ROC curve. Note that the relative ranking of the algorithms does
not change since a curve dominates in ROC space if and only if it
dominates in PR space [
Each row contains information regarding the context of the session, the recommended hotel, and whether the recommendation led to a conversion. In particular, the features are the number of recommendations (from 1 to 5), date of the recommendation, country where the booking was made, country where the passenger is traveling, hotel identifier, hotel provider identifier, price of the hotel at time of the recommendation, price currency and whether the recommendation led to a conversion. Additionally, the logs were enriched with supplementary information regarding each hotel including a hotel numerical rating (from 0 to 5), hotel categorical rating and the hotel chain. information (e.g., flights number, dates) and the passenger information (e.g., name, gender, and somethime passport details). A PNR may also include many other data elements such as payment information (currency, total price, etc), additional ancillary services sold with the ticket (such as extra baggage and hotel reservation) and other airline related information (cabin code, special meal request, etc).
For the purpose of this study, we retrieve and extract features related with the air travel of the traveller. These include the date of PNR creation, airline code, origin city, destination city, date of departure, time of departure, date of arrival, time of arrival, days between the departure and booking date, travel class, number of stops (if any), duration of the flight in minutes (including stops) and the number of days the passenger is staying at the destination. 5
Table 1 shows the results of the experiment comparing diferent algorithms for the hotel model in terms of AUC, AP, F1 and F0.5 scores. In Figure 6, the ROC and AP curves can be seen in detail. The low AUC value for the GLM model and Naive Bayes Classifier suggest that linear classification techniques do not lead to the best results and more complex models are needed to correctly represent the data. The non-linear techniques have closer results, with the Random Forest obtaining the best values for AP, F1 and F0.5. A Stacked Ensemble using all the previous models is created but it does not improve the previous outcome. 5.1
The PNR data is an important attribute since it contains rich attributes related to the trip and passenger. However, is this case personally identifiable information is not used in the recommender system, thus the PNR features help to provide context about the trip rather than the traveller. Incorporating this data to the models substantially enhanced their performance, as can be observed in Figure 6. Features of the PNR including the number of travellers in the booking and trip duration, among others, contributed to an increase in area under the PR curve from 0.183 to 0.249. 5.2
After we have trained the hotel model, we predict individually the probability of conversion of a hotel. Then, we create the sessions based on 5 recommended hotels.
In Table 2 the results are shown. In this case, the best model for both AUC and AP is the Stacked Ensemble composed of a Random Forest, a Generalized Linear Model and a Naïve Bayes Classifier. Although the F0.5 score of the GBM model is slightly better than the STK model, the latter clearly outperforms the rest of the metrics. 5.3
After the Session model has been trained, we analyse its feature importance to study which variables contribute the most to the model using LIME. Concretely, we evaluate the model on the true positive instances from the training dataset, since we want to optimise the conversion.
As can be seen in Figure 7, the most important features according to LIME are all derived from the hotel model: the standard deviation, maximum, and average individual hotel conversion probabilities. Some features which are important to the model such as "market" (country where the booking is made from), the flight class of service, the destination city, and arrival and departure times of the flight can not be used to manipulate the results of the session builder, as these are all part of context of the recommendation. Features extracted from prices (the diference between the average price and the minimum, and the ratio of the lowest price to the average price) are also considered important by the LIME model, but rank lower than many hotel conversion probability features.
As the standard deviation of the individual hotel conversions is the most important criteria, the following rule for the session builder is defined: from the original hotel list remove one hotel with the closest conversion probability to the mean conversion probability of the list, and replace it with the hotel with the highest conversion probability from the set of available hotels for a particular city. 5.4
Results from the hotel list builder are shown in Table 3 for the two largest cities in the dataset and for the complete dataset. For both cities, we observe a large increase in conversion when using the LIME based session builder. However, a brute force approach to evaluating all possible lists does lead to higher conversion rates, at the cost of a significant increase in processing time. When we consider the complete dataset, we once again observe a large increase in conversion from the baseline for the LIME model. With respect to brute force, we observe that the LIME session builder performs much closer to the brute force builder in terms of conversion. This is attributed to the impact of smaller cities in the complete dataset, and thus less choice in hotels for the builders, resulting in the LIME session builder finding the optimal list. Additionally, on the complete dataset, the processing time of the brute force builder is 2.8 times the duration of the LIME builder, whereas larger gains were observed on the individual cities, where more options for hotels were available. 6
In this study, an algorithm was created to improve hotel recommendations based on historical hotel bookings and flight booking attributes. Diferent machine learning models are used in a cascaded fashion. First, a model estimates the conversion probability of the individual hotels independently. Note that adding trip context, via PNR based features, resulted in better PR AUC. The output of the ifrst model is then combined with aggregates of the hotels in the list in order to create a feature vector for the session model to estimate the conversion probability that any hotel in the list will be converted. LIME analysis revealed that the hotel model conversion probabilities are the most important features, specifically the standard deviation, mean and maximum individual hotel conversion probabilities in the list. This allows for a simple heuristic to be defined to increase the session conversion probability. In this study, a single change is performed in the list of hotels, however this could be expanded to allow multiple changes.
Variations on this pipeline could also be considered, for instance
LIME is used in this study for feature importance ranking in the
session builder, however recently a similar methodology was proposed
using a mixture regression model referred to as LEMNA [
Here, the session builder relies on insights gained from analysis of the feature importance ranking of the session model using LIME over all sessions which lead to a conversion. Thus, the same heuristic is applied to all datapoints in the session builder. However, a key aspect of LIME is that it provides an interpretation of a model for a single datapoint. As such, an evolution of the approach would be to compute the most important features for each recommendation in real time, and to use the information to build an optimal hotel list based on the attributes most likely to lead to conversion.