=Paper=
{{Paper
|id=Vol-3219/short1
|storemode=property
|title=An Online Experiment of a Price-Based Re-Rank Algorithm
|pdfUrl=https://ceur-ws.org/Vol-3219/short1.pdf
|volume=Vol-3219
|authors=Emanuele Cavenaghi,Lorenzo Camaione,Paolo Minasi,Gabriele Sottocornola,Fabio Stella,Markus Zanker
|dblpUrl=https://dblp.org/rec/conf/recsys/CavenaghiCMSSZ22
}}
==An Online Experiment of a Price-Based Re-Rank Algorithm==
https://ceur-ws.org/Vol-3219/short1.pdf
An Online Experiment of a Price-Based Re-Rank
Algorithm
Emanuele Cavenaghi1 , Lorenzo Camaione4 , Paolo Minasi4 , Gabriele Sottocornola1 ,
Fabio Stella3 and Markus Zanker1,2
1
Free University of Bozen-Bolzano, Bolzano, Italy
2
University of Klagenfurt, Austria
3
University of Milano-Bicocca, Milano, Italy
4
Bravonext SA t/a lastminute.com, Chiasso, Switzerland
Abstract
Recommender Systems were created to support users in situations of information overload. However,
users are consciously or unconsciously influenced by many factors in their decision making. In this
paper, we focused our attention on the influence of price in user decision-making in the context of online
hotel search and booking. First, we analyzed a historical dataset from a meta-search booking platform to
evaluate the influence of different factors on user click behavior. Then, we performed an online A/B test
on the same meta-search booking platform, in which we compared the current policy with a price-based
re-rank policy. Our experiments suggested that, although in offline observations properties with lower
prices tended to have a higher Click-Through Rate, in an online context a price-based re-rank was only
sufficient to achieve an improvement in Click-Through Rate for the first position on the recommended
list.
Keywords
Recommender Systems, Learning to rank, Tourism, Meta-search Booking Platform, Online Hotel Search
1. Introduction
Recommender Systems (RSs) are algorithms developed for helping users to find items of interest.
The massive volume of information available on the web leads to the problem of information
overload and, thus, increases the need for delivering effective and timely recommendations.
The main idea behind these methods is to know users’ interests, based on their feedback on past
interactions with items in order to recommend new unseen items matching their preferences.
RSs are extensively applied in the E-Tourism domains [1, 2] to recommend destinations/travel
packages [3, 4, 5], points of interest [6, 7, 8] or restaurants [9, 10]. In the context of recommending
appropriate accommodations to travellers it is fundamental to exploit both contextual features
(such as season and place) as well as user’s preferences. In the last years, many RSs were
RecSys Workshop on Recommenders in Tourism (RecTour 2022), September 22th, 2022, co-located with the 16th ACM
Conference on Recommender Systems, Seattle, WA, USA
$ ecavenaghi@unibz.it (E. Cavenaghi); lorenzo.camaione@lastminute.com (L. Camaione);
paolo.minasi@lastminute.com (P. Minasi); gabriele.sottocornola@unibz.it (G. Sottocornola); fabio.stella@unimib.it
(F. Stella); Markus.Zanker@unibz.it (M. Zanker)
0000-0002-0235-0421 (E. Cavenaghi); 0000-0001-9983-2330 (G. Sottocornola); 0000-0002-4805-5516 (M. Zanker)
© 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
Workshop
Proceedings
http://ceur-ws.org
ISSN 1613-0073
CEUR Workshop Proceedings (CEUR-WS.org)
developed in order to recommend hotels in the context of online booking. Some approaches
were based on traditional RSs techniques such as Collaborative Filtering [11], like [12, 13, 14]
also considering multi-criteria ratings [15], and Content-Based approaches [16], like [17, 18].
Instead, other works proposed domain-specific approaches. For example, Levi et al. [19] used
text reviews as the main source of information to make recommendations, [20] built specific
topic models from textual reviews and Lin et al.[21] designed an app where users can search
and browse hotel reviews.
This work aims to find answers to the following research questions in the context of Online
Hotel Search:
• Do offered tourism properties1 with lower prices in recommendation lists have a higher
Click-Through Rate (CTR)?
• Is a price-based re-rank of these offered properties sufficient to achieve a higher CTR?
• Does the Online Travel Agency (OTA)2 associated with offered properties influence the
CTR?
To answer these questions, firstly, we analysed historical data. Specifically, our dataset was
collected on a meta-search booking platform that compares the prices of offered properties from
different OTAs. Then, to answer the second and the third research questions, we ran an A/B
test to compare the RS used by the company with a re-rank algorithm based on price. In both,
the historical dataset and the A/B test, the company had no information about the anonymous
users and their history of previous interactions with the site. Moreover, there was no explicit
feedback (e.g., user ratings specific to properties), but we had to rely on implicit feedback, in
our case user clicks.
This paper is organized as follows, in Section 2, we report a brief discussion on the historical
dataset. Then, in Section 3, we describe a simple re-rank algorithm based on relative prices,
and, in Section 4, we present the results of the A/B test. Finally, in Section 5, we investigate the
influence of the OTA on the results of the A/B test.
2. Dataset Description
In this section, we report the results of our data analysis to provide an answer to the first research
question: “Do offered tourism properties with lower prices in recommendation lists have a higher
CTR?”. The dataset consists of roughly 130, 000 recommended lists, each composed by 25
properties showed on the same page, collected on a meta-search booking platform in the period
between Nov 2021 - Apr 2022 on searches made on 14 Italian cities. In each recommended list,
each property can be presented with a different OTA. As a result, a property can be presented
with different OTAs in different lists and in each list multiple OTAs are presented. Given the
small number of users that look beyond the first page, we restricted the analysis to the first
page.
First, considering the CTR for different rank positions in the recommendation list, as reported
in Figure 1, the strong effect of the rank position becomes evident as already stated by Joachims
1
With the term properties we refer to any type of accommodation like hotels, apartments, guest houses, etc.
2
The OTA is the external company on which the user can book the property.
Figure 1: CTR for each rank position in the recommendation list. The x-axis reports the rank position
in the recommendation list, while the y-axis reports the CTR achieved in that rank position.
(a) Price higher than 0.75 quantile. (b) Price lower than 0.25 quantile.
Figure 2: Difference between conditioned CTR and a-priori CTR, showed in Figure 1, for each rank
position in the recommended list.
et al. [22]. Here, the authors indicated that users’ clicking decisions were influenced by the
relevance of the results, but also by the order in which they were presented. For the rest of the
paper, we will refer to the CTR in Figure 1 as the a-priori CTR.
In order to answer the first research question, we report in Figure 2 the difference between
the CTR distribution conditioned on price and the a-priori CTR. We ran two experiments with
the following filter conditions: values higher than the 0.75 quantile and values lower than the
0.25 quantile within each recommended list. Specifically, in Figure 2a, we computed the CTR
distribution taking into account only the offered properties that had a price higher than the
0.75 quantile price within each recommendation list, i.e., we removed all the properties with a
price lower than the 0.75 quantile in each recommendation list, and we subtracted the a-priori
CTR distribution to obtain the plot. Same procedure was applied to compute the conditioned
CTR distribution for properties with a price lower than 0.25 quantile, displayed in Figure 2b. In
this analysis, we just ignored the cheaper and expensive properties without changing the rank
position.
For each rank position in the list, the CTR was higher than the a-priori CTR if we considered
properties with a price lower than the 0.25 quantile. This clearly means that lower prices
positively influenced the users’ propensity of clicking on a property and the opposite happened
if we considered properties with a price higher than the 0.75 quantile: for higher prices the
CTR was lower. The answer to the question: “Do offered tourism properties with lower prices in
recommendation lists have a higher CTR?” is clearly yes. Price influences the user decision-
making both positively and negatively as stated in Lockyer [23] and Stávková et al. [24]
3. Re-rank Algorithm
To answer the second research question, we implemented a simple and efficient algorithm to
re-rank the top-25 list of offered properties as recommended by the current algorithm. Since
the algorithm only re-ranks the top-25 items, it is ensured that all properties presented to users
are of comparable quality with the baseline. To re-rank the properties, we computed a score
and reordered the properties from highest to lowest score. The score, reported in Equation 1, is
composed by two logistic functions with two means:
1 1
𝑦 =𝛼· +𝛽· (1)
1 + 𝑒𝑘𝑥𝑖 −𝜇1 1 + 𝑒𝑘𝑥𝑖 −𝜇2
Where 𝛼, 𝛽 ∈ [0, 1] manage the weight of the two functions while 𝛼+𝛽 = 1, 𝑦 represents the
score, 𝑥𝑖 the price of the property 𝑖 and 𝑘 controls the speed by which the function approaches
the limits (i.e., 0 and 1). Finally, the two means, 𝜇1 and 𝜇2 , represent respectively the mean price
for the type of accommodation3 of the property 𝑖 within the recommendation list and the median
price of the properties within the recommended list (regardless of the type of accommodation).
For 𝜇2 , we used the median instead of the mean to reduce the impact of outlier prices, for
example, the price of 5-stars hotels. 𝜇1 allows us to account in a simple way the quality-price
ratio, because a user may prefer to pay more for higher quality accommodations. While 𝜇2
controls for the absolute price of the properties, because, as showed in Figure 2b, users tend to
click on properties associated with a lower price.
In the following experiments, we use 𝛼 = 𝛽 = 0.5. We select these values using the results
from offline experiments on the dataset described in Section 2, because running multiples online
experiments with different values of the two hyper-parameters was not possible.
4. A/B Test Results
The A/B test was conducted on the company’s website for 20 days (between June and July 2022),
and, in the end, nearly 1 million searches were conducted by users worldwide. We compared
the Baseline policy used by the company with the Re-rank policy described in Section 3. The
results, in terms of CTR for each rank position, are reported in Figure 3. The confidence interval
at 95% is reported by the black line on the top of the bars.
Figure 3 clearly depicts that, for the first position, the CTR achieved by the Re-rank policy
was statistically significant higher (more than 2%) than the Baseline policy. Instead, for all
rank positions after the third, the Baseline policy achieved a slightly higher CTR, even if the
difference was less than 0.5% and close to zero for bottom positions. The increase in the first
position was expected, and the results confirmed our hypotheses. However, we also expected
3
With type of accommodation we refer to the different type of properties, e.g., apartment, guest house, hotels
with 3 stars, etc.
Figure 3: Results from the A/B test in terms of CTR for each rank position.
an improvement for more top ranked positions while from the third rank position we observed
a decrease.
To further analyse the user click behaviour, given that we can not disclose the results in terms
of conversion rates, we computed the CTR for session (SCTR). The SCTR is defined as the ratio
between the number of clicked sessions and the total number of sessions: # of #clicked sessions
of session .A
session is clicked if at least one of the recommended item received a click. The Re-rank policy
achieved a SCTR of 23.48%, while the Baseline policy achieved a slightly higher SCTR of 24.16%.
The difference between the two policies was very small and showed that the increase in CTR
in the first position for the Re-rank policy was compensated by the decrease for all the other
positions.
Given the results from the A/B test, the answer to the second research question, “Is a price-
based re-rank of these offered properties sufficient to achieve a higher CTR?”, is clearly yes if
the main goal is to improve the CTR in the top position of the list. Despite the data analysis
results, which showed that lower prices were a key factor to improve the CTR, a policy that
re-rank items by price was only sufficient to improve the CTR w.r.t. the Baseline policy in the
first position of the recommended list. However, since users usually pay more attention to the
item in first position, this can be considered a good result even if the SCTR slightly decreased
with the Re-rank policy.
5. Influence of the OTA
To further study the differences in CTR and SCTR metrics between the two policies, we analysed
further variables with potential influence on user decision-making. Among the considered
variables, such as average rating, number of reviews and location of the properties, the Online
Travel Agency (OTA) presented with each property emerged as one of the key factors. Here, we
focused on the influence of OTAs because it is important to the company’s business and we
already analysed the other variables in [25].
Figure 4 depicts the CTR at each rank position for the most common OTA and for all the other
Figure 4: CTR comparison for each rank position between the most common OTA and the other OTAs.
(a) Count of impressions for the most common OTA. (b) Count of impressions for all the other OTAs.
Figure 5: Count of the impressions for the most common OTA and for the other OTAs.
OTAs. Since we can not disclose the names of the OTAs, we only distinguished between the
most common OTA and the other OTAs. The most common OTA always had a CTR that was
significantly higher than the CTR of the other OTAs at least for the first 15 positions, which
means that users preferred this OTA to the others. One reason for this preference could be that
the most common OTA might be more trusted by users.
This difference in CTR between OTA groups, joined to the number of recommendations
for each OTA group, reported in Figure 5, could explain the difference in SCTR identified
between the two policies. From Figure 5a, we can see that the Re-rank policy recommended
properties with the most common OTA less frequently at top-ranked positions while favouring
more frequently other OTAs, Figure 5b.
Thus, by favouring lower price offers the Re-rank policy pushed less well-known OTAs
to top-ranked positions and exposed them to higher levels of users’ attention. Their lower
likelihood of being clicked, however, seems to have neutralized the positive price effect and
resulted in an overall decrease in the SCTR.
The answer to the third research question, “Does the OTA associated with offered properties
influence the CTR?”, is yes. Although price and rank positions were identified as the most
important features that influenced users’ decision-making, there were also other factors, in
our case the OTA, that could impact users’ decision and thus overall performance metrics of
a ranking policy. At the end, in our case, a price-based re-rank algorithm that also keep the
balance for the OTA feature would probably have improved the baseline, whereas considering
only the price was sufficient to achieve a marginal improvement.
6. Conclusions
In this paper, we studied how the price influenced user click behaviour in online hotel search.
We started by analysing a historical dataset collected in a meta-search booking platform in which
as expected the price showed a strong influence on CTR. To verify this fact, we ran an online
A/B test on the company’s website to compare a Baseline policy with a price-based re-rank
policy that shuffles the top-25 offered properties in recommendation lists.
The results showed that the re-rank policy improved the CTR for top rank position. This
confirmed that price was a key factor influencing users’ click behaviour in according to previous
works (such as Lockyer [23] and Stávková et al. [24]) where many factors influenced user
decision-making, such as cleanliness and quality of properties. However, in the context of RSs,
it is usually very difficult, or nearly impossible, to assess the true quality of items. Instead,
we found that even a more identifiable feature, such as the OTA associated with a property,
influenced the user decision. For example, in our case the most common OTA achieved a higher
CTR for every rank position compared to the other OTAs and seems to be favoured more by
users.
This work consequently highlights the many influence factors and biases on users decision-
making in online travel search that are disregarded in most offline datasets by presenting the
outcome of a price-based re-rank strategy.
References
[1] J. Borràs, A. Moreno, A. Valls, Intelligent tourism recommender systems: A survey, Expert
systems with applications 41 (2014) 7370–7389.
[2] K. Chaudhari, A. Thakkar, A comprehensive survey on travel recommender systems,
Archives of Computational Methods in Engineering 27 (2020) 1545–1571.
[3] Q. Liu, Y. Ge, Z. Li, E. Chen, H. Xiong, Personalized travel package recommendation, in:
2011 IEEE 11th international conference on data mining, IEEE, 2011, pp. 407–416.
[4] F. Lorenzi, S. Loh, M. Abel, Personaltour: a recommender system for travel packages, in:
2011 IEEE/WIC/ACM International Conferences on Web Intelligence and Intelligent Agent
Technology, volume 2, IEEE, 2011, pp. 333–336.
[5] Z. Yu, H. Xu, Z. Yang, B. Guo, Personalized travel package with multi-point-of-interest
recommendation based on crowdsourced user footprints, IEEE Transactions on Human-
Machine Systems 46 (2015) 151–158.
[6] T. Ruotsalo, K. Haav, A. Stoyanov, S. Roche, E. Fani, R. Deliai, E. Mäkelä, T. Kauppinen,
E. Hyvönen, Smartmuseum: A mobile recommender system for the web of data, Journal
of Web Semantics 20 (2013) 50–67.
[7] W.-S. Yang, S.-Y. Hwang, itravel: A recommender system in mobile peer-to-peer environ-
ment, Journal of Systems and Software 86 (2013) 12–20.
[8] C. Binucci, F. De Luca, E. Di Giacomo, G. Liotta, F. Montecchiani, Designing the content
analyzer of a travel recommender system, Expert Systems with Applications 87 (2017)
199–208.
[9] A. Gupta, K. Singh, Location based personalized restaurant recommendation system for
mobile environments, in: 2013 International Conference on Advances in Computing,
Communications and Informatics (ICACCI), IEEE, 2013, pp. 507–511.
[10] J. Zeng, F. Li, H. Liu, J. Wen, S. Hirokawa, A restaurant recommender system based on user
preference and location in mobile environment, in: 2016 5th IIAI International Congress
on Advanced Applied Informatics (IIAI-AAI), IEEE, 2016, pp. 55–60.
[11] P. Resnick, N. Iacovou, M. Suchak, P. Bergstrom, J. Riedl, Grouplens: An open architecture
for collaborative filtering of netnews, in: Proceedings of the 1994 ACM conference on
Computer supported cooperative work, 1994, pp. 175–186.
[12] R. Saga, Y. Hayashi, H. Tsuji, Hotel recommender system based on user’s preference
transition, in: 2008 IEEE International Conference on Systems, Man and Cybernetics, IEEE,
2008, pp. 2437–2442.
[13] M. Nilashi, O. bin Ibrahim, N. Ithnin, N. H. Sarmin, A multi-criteria collaborative filtering
recommender system for the tourism domain using expectation maximization (em) and
pca–anfis, Electronic Commerce Research and Applications 14 (2015) 542–562.
[14] Z. Zhang, Y. Morimoto, Collaborative hotel recommendation based on topic and sentiment
of review comments, in: proceeding of the 9th forum for information and engineering,
DEIM, 2017.
[15] D. Jannach, M. Zanker, M. Fuchs, Leveraging multi-criteria customer feedback for satis-
faction analysis and improved recommendations, J. Inf. Technol. Tour. 14 (2014) 119–149.
URL: https://doi.org/10.1007/s40558-014-0010-z. doi:10.1007/s40558-014-0010-z.
[16] M. J. Pazzani, D. Billsus, Content-based recommendation systems, in: The adaptive web,
Springer, 2007, pp. 325–341.
[17] M. P. O’Mahony, B. Smyth, Learning to recommend helpful hotel reviews, in: Proceedings
of the third ACM conference on Recommender systems, 2009, pp. 305–308.
[18] J. Srisuan, A. Hanskunatai, The ensemble of naïve bayes classifiers for hotel searching, in:
2014 International Computer Science and Engineering Conference (ICSEC), IEEE, 2014, pp.
168–173.
[19] A. Levi, O. Mokryn, C. Diot, N. Taft, Finding a needle in a haystack of reviews: cold start
context-based hotel recommender system, in: Proceedings of the sixth ACM conference
on Recommender systems, 2012, pp. 115–122.
[20] M. Rossetti, F. Stella, M. Zanker, Analyzing user reviews in tourism with topic models, J.
Inf. Technol. Tour. 16 (2016) 5–21. URL: https://doi.org/10.1007/s40558-015-0035-y. doi:10.
1007/s40558-015-0035-y.
[21] K.-P. Lin, C.-Y. Lai, P.-C. Chen, S.-Y. Hwang, Personalized hotel recommendation using
text mining and mobile browsing tracking, in: 2015 IEEE International Conference on
Systems, Man, and Cybernetics, IEEE, 2015, pp. 191–196.
[22] T. Joachims, L. Granka, B. Pan, H. Hembrooke, F. Radlinski, G. Gay, Evaluating the
accuracy of implicit feedback from clicks and query reformulations in web search, ACM
Transactions on Information Systems (TOIS) 25 (2007) 7–es.
[23] T. Lockyer, The perceived importance of price as one hotel selection dimension, Tourism
Management 26 (2005) 529–537.
[24] J. Stávková, L. Stejskal, Z. Toufarová, et al., Factors influencing consumer behaviour,
ZEMEDELSKA EKONOMIKA-PRAHA- 54 (2008) 276.
[25] E. Cavenaghi, L. Camaione, P. Minasi, G. Sottocornola, F. Stella, M. Zanker, An analysis
of user click behaviour in online hotel search, in: Proceedings of the Italian Information
Retrieval Workshop, 2022.