Recommender Systems were created to support users in situations of information overload. However, users are consciously or unconsciously influenced by many factors in their decision making. In this paper, we focused our attention on the influence of price in user decision-making in the context of online hotel search and booking. First, we analyzed a historical dataset from a meta-search booking platform to evaluate the influence of diferent factors on user click behavior. Then, we performed an online A/B test on the same meta-search booking platform, in which we compared the current policy with a price-based re-rank policy. Our experiments suggested that, although in ofline observations properties with lower prices tended to have a higher Click-Through Rate, in an online context a price-based re-rank was only suficient to achieve an improvement in Click-Through Rate for the first position on the recommended list.
Recommender Systems (RSs) are algorithms developed for helping users to find items of interest. The massive volume of information available on the web leads to the problem of information overload and, thus, increases the need for delivering efective and timely recommendations. The main idea behind these methods is to know users’ interests, based on their feedback on past interactions with items in order to recommend new unseen items matching their preferences.
RSs are extensively applied in the E-Tourism domains [
This work aims to find answers to the following research questions in the context of Online Hotel Search: • Do ofered tourism properties1 with lower prices in recommendation lists have a higher
Click-Through Rate (CTR)? • Is a price-based re-rank of these ofered properties suficient to achieve a higher CTR? • Does the Online Travel Agency (OTA)2 associated with ofered properties influence the
CTR?
To answer these questions, firstly, we analysed historical data. Specifically, our dataset was collected on a meta-search booking platform that compares the prices of ofered properties from diferent OTAs. Then, to answer the second and the third research questions, we ran an A/B test to compare the RS used by the company with a re-rank algorithm based on price. In both, the historical dataset and the A/B test, the company had no information about the anonymous users and their history of previous interactions with the site. Moreover, there was no explicit feedback (e.g., user ratings specific to properties), but we had to rely on implicit feedback, in our case user clicks.
This paper is organized as follows, in Section 2, we report a brief discussion on the historical dataset. Then, in Section 3, we describe a simple re-rank algorithm based on relative prices, and, in Section 4, we present the results of the A/B test. Finally, in Section 5, we investigate the influence of the OTA on the results of the A/B test.
In this section, we report the results of our data analysis to provide an answer to the first research question: “Do ofered tourism properties with lower prices in recommendation lists have a higher CTR?”. The dataset consists of roughly 130, 000 recommended lists, each composed by 25 properties showed on the same page, collected on a meta-search booking platform in the period between Nov 2021 - Apr 2022 on searches made on 14 Italian cities. In each recommended list, each property can be presented with a diferent OTA. As a result, a property can be presented with diferent OTAs in diferent lists and in each list multiple OTAs are presented. Given the small number of users that look beyond the first page, we restricted the analysis to the first page.
First, considering the CTR for diferent rank positions in the recommendation list, as reported in Figure 1, the strong efect of the rank position becomes evident as already stated by Joachims 1With the term properties we refer to any type of accommodation like hotels, apartments, guest houses, etc. 2The OTA is the external company on which the user can book the property. et al. [22]. Here, the authors indicated that users’ clicking decisions were influenced by the relevance of the results, but also by the order in which they were presented. For the rest of the paper, we will refer to the CTR in Figure 1 as the a-priori CTR.
In order to answer the first research question, we report in Figure 2 the diference between the CTR distribution conditioned on price and the a-priori CTR. We ran two experiments with the following filter conditions: values higher than the 0.75 quantile and values lower than the 0.25 quantile within each recommended list. Specifically, in Figure 2a, we computed the CTR distribution taking into account only the ofered properties that had a price higher than the 0.75 quantile price within each recommendation list, i.e., we removed all the properties with a price lower than the 0.75 quantile in each recommendation list, and we subtracted the a-priori CTR distribution to obtain the plot. Same procedure was applied to compute the conditioned CTR distribution for properties with a price lower than 0.25 quantile, displayed in Figure 2b. In this analysis, we just ignored the cheaper and expensive properties without changing the rank position.
For each rank position in the list, the CTR was higher than the a-priori CTR if we considered properties with a price lower than the 0.25 quantile. This clearly means that lower prices positively influenced the users’ propensity of clicking on a property and the opposite happened if we considered properties with a price higher than the 0.75 quantile: for higher prices the CTR was lower. The answer to the question: “Do ofered tourism properties with lower prices in recommendation lists have a higher CTR?” is clearly yes. Price influences the user decisionmaking both positively and negatively as stated in Lockyer [23] and Stávková et al. [24]
To answer the second research question, we implemented a simple and eficient algorithm to re-rank the top-25 list of ofered properties as recommended by the current algorithm. Since the algorithm only re-ranks the top-25 items, it is ensured that all properties presented to users are of comparable quality with the baseline. To re-rank the properties, we computed a score and reordered the properties from highest to lowest score. The score, reported in Equation 1, is composed by two logistic functions with two means:
1 1 = · 1 + − 1 + · 1 + − 2 (1)
Where , ∈ [
In the following experiments, we use = = 0.5. We select these values using the results from ofline experiments on the dataset described in Section 2, because running multiples online experiments with diferent values of the two hyper-parameters was not possible.
The A/B test was conducted on the company’s website for 20 days (between June and July 2022), and, in the end, nearly 1 million searches were conducted by users worldwide. We compared the Baseline policy used by the company with the Re-rank policy described in Section 3. The results, in terms of CTR for each rank position, are reported in Figure 3. The confidence interval at 95% is reported by the black line on the top of the bars.
Figure 3 clearly depicts that, for the first position, the CTR achieved by the Re-rank policy was statistically significant higher (more than 2%) than the Baseline policy. Instead, for all rank positions after the third, the Baseline policy achieved a slightly higher CTR, even if the diference was less than 0.5% and close to zero for bottom positions. The increase in the first position was expected, and the results confirmed our hypotheses. However, we also expected 3With type of accommodation we refer to the diferent type of properties, e.g., apartment, guest house, hotels with 3 stars, etc. an improvement for more top ranked positions while from the third rank position we observed a decrease.
To further analyse the user click behaviour, given that we can not disclose the results in terms of conversion rates, we computed the CTR for session (SCTR). The SCTR is defined as the ratio between the number of clicked sessions and the total number of sessions: # of clicked sessions . A # of session session is clicked if at least one of the recommended item received a click. The Re-rank policy achieved a SCTR of 23.48%, while the Baseline policy achieved a slightly higher SCTR of 24.16%. The diference between the two policies was very small and showed that the increase in CTR in the first position for the Re-rank policy was compensated by the decrease for all the other positions.
Given the results from the A/B test, the answer to the second research question, “Is a pricebased re-rank of these ofered properties suficient to achieve a higher CTR?”, is clearly yes if the main goal is to improve the CTR in the top position of the list. Despite the data analysis results, which showed that lower prices were a key factor to improve the CTR, a policy that re-rank items by price was only suficient to improve the CTR w.r.t. the Baseline policy in the ifrst position of the recommended list. However, since users usually pay more attention to the item in first position, this can be considered a good result even if the SCTR slightly decreased with the Re-rank policy.
To further study the diferences in CTR and SCTR metrics between the two policies, we analysed further variables with potential influence on user decision-making. Among the considered variables, such as average rating, number of reviews and location of the properties, the Online Travel Agency (OTA) presented with each property emerged as one of the key factors. Here, we focused on the influence of OTAs because it is important to the company’s business and we already analysed the other variables in [25].
Figure 4 depicts the CTR at each rank position for the most common OTA and for all the other (a) Count of impressions for the most common OTA. (b) Count of impressions for all the other OTAs. OTAs. Since we can not disclose the names of the OTAs, we only distinguished between the most common OTA and the other OTAs. The most common OTA always had a CTR that was significantly higher than the CTR of the other OTAs at least for the first 15 positions, which means that users preferred this OTA to the others. One reason for this preference could be that the most common OTA might be more trusted by users.
This diference in CTR between OTA groups, joined to the number of recommendations for each OTA group, reported in Figure 5, could explain the diference in SCTR identified between the two policies. From Figure 5a, we can see that the Re-rank policy recommended properties with the most common OTA less frequently at top-ranked positions while favouring more frequently other OTAs, Figure 5b.
Thus, by favouring lower price ofers the Re-rank policy pushed less well-known OTAs to top-ranked positions and exposed them to higher levels of users’ attention. Their lower likelihood of being clicked, however, seems to have neutralized the positive price efect and resulted in an overall decrease in the SCTR.
The answer to the third research question, “Does the OTA associated with ofered properties influence the CTR?”, is yes. Although price and rank positions were identified as the most important features that influenced users’ decision-making, there were also other factors, in our case the OTA, that could impact users’ decision and thus overall performance metrics of a ranking policy. At the end, in our case, a price-based re-rank algorithm that also keep the balance for the OTA feature would probably have improved the baseline, whereas considering only the price was suficient to achieve a marginal improvement.
In this paper, we studied how the price influenced user click behaviour in online hotel search. We started by analysing a historical dataset collected in a meta-search booking platform in which as expected the price showed a strong influence on CTR. To verify this fact, we ran an online A/B test on the company’s website to compare a Baseline policy with a price-based re-rank policy that shufles the top-25 ofered properties in recommendation lists.
The results showed that the re-rank policy improved the CTR for top rank position. This confirmed that price was a key factor influencing users’ click behaviour in according to previous works (such as Lockyer [23] and Stávková et al. [24]) where many factors influenced user decision-making, such as cleanliness and quality of properties. However, in the context of RSs, it is usually very dificult, or nearly impossible, to assess the true quality of items. Instead, we found that even a more identifiable feature, such as the OTA associated with a property, influenced the user decision. For example, in our case the most common OTA achieved a higher CTR for every rank position compared to the other OTAs and seems to be favoured more by users.
This work consequently highlights the many influence factors and biases on users decisionmaking in online travel search that are disregarded in most ofline datasets by presenting the outcome of a price-based re-rank strategy. [7] W.-S. Yang, S.-Y. Hwang, itravel: A recommender system in mobile peer-to-peer environment, Journal of Systems and Software 86 (2013) 12–20. [8] C. Binucci, F. De Luca, E. Di Giacomo, G. Liotta, F. Montecchiani, Designing the content analyzer of a travel recommender system, Expert Systems with Applications 87 (2017) 199–208. [9] A. Gupta, K. Singh, Location based personalized restaurant recommendation system for mobile environments, in: 2013 International Conference on Advances in Computing, Communications and Informatics (ICACCI), IEEE, 2013, pp. 507–511. [10] J. Zeng, F. Li, H. Liu, J. Wen, S. Hirokawa, A restaurant recommender system based on user preference and location in mobile environment, in: 2016 5th IIAI International Congress on Advanced Applied Informatics (IIAI-AAI), IEEE, 2016, pp. 55–60. [11] P. Resnick, N. Iacovou, M. Suchak, P. Bergstrom, J. Riedl, Grouplens: An open architecture for collaborative filtering of netnews, in: Proceedings of the 1994 ACM conference on Computer supported cooperative work, 1994, pp. 175–186. [12] R. Saga, Y. Hayashi, H. Tsuji, Hotel recommender system based on user’s preference transition, in: 2008 IEEE International Conference on Systems, Man and Cybernetics, IEEE, 2008, pp. 2437–2442. [13] M. Nilashi, O. bin Ibrahim, N. Ithnin, N. H. Sarmin, A multi-criteria collaborative filtering recommender system for the tourism domain using expectation maximization (em) and pca–anfis, Electronic Commerce Research and Applications 14 (2015) 542–562. [14] Z. Zhang, Y. Morimoto, Collaborative hotel recommendation based on topic and sentiment of review comments, in: proceeding of the 9th forum for information and engineering, DEIM, 2017. [15] D. Jannach, M. Zanker, M. Fuchs, Leveraging multi-criteria customer feedback for satisfaction analysis and improved recommendations, J. Inf. Technol. Tour. 14 (2014) 119–149.
URL: https://doi.org/10.1007/s40558-014-0010-z. doi:10.1007/s40558-014-0010-z. [16] M. J. Pazzani, D. Billsus, Content-based recommendation systems, in: The adaptive web,
Springer, 2007, pp. 325–341. [17] M. P. O’Mahony, B. Smyth, Learning to recommend helpful hotel reviews, in: Proceedings of the third ACM conference on Recommender systems, 2009, pp. 305–308. [18] J. Srisuan, A. Hanskunatai, The ensemble of naïve bayes classifiers for hotel searching, in: 2014 International Computer Science and Engineering Conference (ICSEC), IEEE, 2014, pp. 168–173. [19] A. Levi, O. Mokryn, C. Diot, N. Taft, Finding a needle in a haystack of reviews: cold start context-based hotel recommender system, in: Proceedings of the sixth ACM conference on Recommender systems, 2012, pp. 115–122. [20] M. Rossetti, F. Stella, M. Zanker, Analyzing user reviews in tourism with topic models, J.
Inf. Technol. Tour. 16 (2016) 5–21. URL: https://doi.org/10.1007/s40558-015-0035-y. doi:10. 1007/s40558-015-0035-y. [21] K.-P. Lin, C.-Y. Lai, P.-C. Chen, S.-Y. Hwang, Personalized hotel recommendation using text mining and mobile browsing tracking, in: 2015 IEEE International Conference on Systems, Man, and Cybernetics, IEEE, 2015, pp. 191–196. [22] T. Joachims, L. Granka, B. Pan, H. Hembrooke, F. Radlinski, G. Gay, Evaluating the accuracy of implicit feedback from clicks and query reformulations in web search, ACM Transactions on Information Systems (TOIS) 25 (2007) 7–es. [23] T. Lockyer, The perceived importance of price as one hotel selection dimension, Tourism
Management 26 (2005) 529–537. [24] J. Stávková, L. Stejskal, Z. Toufarová, et al., Factors influencing consumer behaviour,
ZEMEDELSKA EKONOMIKA-PRAHA- 54 (2008) 276. [25] E. Cavenaghi, L. Camaione, P. Minasi, G. Sottocornola, F. Stella, M. Zanker, An analysis of user click behaviour in online hotel search, in: Proceedings of the Italian Information Retrieval Workshop, 2022.