Recommender Systems (RSs) are algorithms developed for helping users to find items of interest. The massive volume of information available on the web leads to the problem of information overload and, thus, increases the need for delivering effective and timely recommendations. The main idea behind these methods is to know users' interests, based on their feedback on past interactions with items in order to recommend new unseen items matching their preferences.
RSs are extensively applied in the E-Tourism domains [1,2] to recommend destinations/travel packages [3,4,5], points of interest [6,7,8] or restaurants [9,10]. In the context of recommending appropriate accommodations to travellers it is fundamental to exploit both contextual features (such as season and place) as well as user's preferences. In the last years, many RSs were developed in order to recommend hotels in the context of online booking. Some approaches were based on traditional RSs techniques such as Collaborative Filtering [11], like [12,13,14] also considering multi-criteria ratings [15], and Content-Based approaches [16], like [17,18]. Instead, other works proposed domain-specific approaches. For example, Levi et al. [19] used text reviews as the main source of information to make recommendations, [20] built specific topic models from textual reviews and Lin et al. [21] designed an app where users can search and browse hotel reviews.
This work aims to find answers to the following research questions in the context of Online Hotel Search:
• Do offered tourism properties1 with lower prices in recommendation lists have a higher Click-Through Rate (CTR)? • Is a price-based re-rank of these offered properties sufficient to achieve a higher CTR? • Does the Online Travel Agency (OTA)2 associated with offered properties influence the CTR?
To answer these questions, firstly, we analysed historical data. Specifically, our dataset was collected on a meta-search booking platform that compares the prices of offered properties from different OTAs. Then, to answer the second and the third research questions, we ran an A/B test to compare the RS used by the company with a re-rank algorithm based on price. In both, the historical dataset and the A/B test, the company had no information about the anonymous users and their history of previous interactions with the site. Moreover, there was no explicit feedback (e.g., user ratings specific to properties), but we had to rely on implicit feedback, in our case user clicks.
This paper is organized as follows, in Section 2, we report a brief discussion on the historical dataset. Then, in Section 3, we describe a simple re-rank algorithm based on relative prices, and, in Section 4, we present the results of the A/B test. Finally, in Section 5, we investigate the influence of the OTA on the results of the A/B test.
In this section, we report the results of our data analysis to provide an answer to the first research question: "Do offered tourism properties with lower prices in recommendation lists have a higher CTR?". The dataset consists of roughly 130, 000 recommended lists, each composed by 25 properties showed on the same page, collected on a meta-search booking platform in the period between Nov 2021 -Apr 2022 on searches made on 14 Italian cities. In each recommended list, each property can be presented with a different OTA. As a result, a property can be presented with different OTAs in different lists and in each list multiple OTAs are presented. Given the small number of users that look beyond the first page, we restricted the analysis to the first page.
First, considering the CTR for different rank positions in the recommendation list, as reported in Figure 1, the strong effect of the rank position becomes evident as already stated by Joachims et al. [22]. Here, the authors indicated that users' clicking decisions were influenced by the relevance of the results, but also by the order in which they were presented. For the rest of the paper, we will refer to the CTR in Figure 1 as the a-priori CTR.
In order to answer the first research question, we report in Figure 2 the difference between the CTR distribution conditioned on price and the a-priori CTR. We ran two experiments with the following filter conditions: values higher than the 0.75 quantile and values lower than the 0.25 quantile within each recommended list. Specifically, in Figure 2a, we computed the CTR distribution taking into account only the offered properties that had a price higher than the 0.75 quantile price within each recommendation list, i.e., we removed all the properties with a price lower than the 0.75 quantile in each recommendation list, and we subtracted the a-priori CTR distribution to obtain the plot. Same procedure was applied to compute the conditioned CTR distribution for properties with a price lower than 0.25 quantile, displayed in Figure 2b. In this analysis, we just ignored the cheaper and expensive properties without changing the rank position.
For each rank position in the list, the CTR was higher than the a-priori CTR if we considered properties with a price lower than the 0.25 quantile. This clearly means that lower prices positively influenced the users' propensity of clicking on a property and the opposite happened if we considered properties with a price higher than the 0.75 quantile: for higher prices the CTR was lower. The answer to the question: "Do offered tourism properties with lower prices in recommendation lists have a higher CTR?" is clearly yes. Price influences the user decisionmaking both positively and negatively as stated in Lockyer [23] and Stávková et al. [24]
To answer the second research question, we implemented a simple and efficient algorithm to re-rank the top-25 list of offered properties as recommended by the current algorithm. Since the algorithm only re-ranks the top-25 items, it is ensured that all properties presented to users are of comparable quality with the baseline. To re-rank the properties, we computed a score and reordered the properties from highest to lowest score. The score, reported in Equation 1, is composed by two logistic functions with two means:
Where 𝛼, 𝛽 ∈ [0, 1] manage the weight of the two functions while 𝛼+𝛽 = 1, 𝑦 represents the score, 𝑥 𝑖 the price of the property 𝑖 and 𝑘 controls the speed by which the function approaches the limits (i.e., 0 and 1). Finally, the two means, 𝜇 1 and 𝜇 2 , represent respectively the mean price for the type of accommodation3 of the property 𝑖 within the recommendation list and the median price of the properties within the recommended list (regardless of the type of accommodation). For 𝜇 2 , we used the median instead of the mean to reduce the impact of outlier prices, for example, the price of 5-stars hotels. 𝜇 1 allows us to account in a simple way the quality-price ratio, because a user may prefer to pay more for higher quality accommodations. While 𝜇 2 controls for the absolute price of the properties, because, as showed in Figure 2b, users tend to click on properties associated with a lower price.
In the following experiments, we use 𝛼 = 𝛽 = 0.5. We select these values using the results from offline experiments on the dataset described in Section 2, because running multiples online experiments with different values of the two hyper-parameters was not possible.
The A/B test was conducted on the company's website for 20 days (between June and July 2022), and, in the end, nearly 1 million searches were conducted by users worldwide. We compared the Baseline policy used by the company with the Re-rank policy described in Section 3. The results, in terms of CTR for each rank position, are reported in Figure 3. The confidence interval at 95% is reported by the black line on the top of the bars.
Figure 3 clearly depicts that, for the first position, the CTR achieved by the Re-rank policy was statistically significant higher (more than 2%) than the Baseline policy. Instead, for all rank positions after the third, the Baseline policy achieved a slightly higher CTR, even if the difference was less than 0.5% and close to zero for bottom positions. The increase in the first position was expected, and the results confirmed our hypotheses. However, we also expected an improvement for more top ranked positions while from the third rank position we observed a decrease.
To further analyse the user click behaviour, given that we can not disclose the results in terms of conversion rates, we computed the CTR for session (SCTR). The SCTR is defined as the ratio between the number of clicked sessions and the total number of sessions: # of clicked sessions # of session . A session is clicked if at least one of the recommended item received a click. The Re-rank policy achieved a SCTR of 23.48%, while the Baseline policy achieved a slightly higher SCTR of 24.16%. The difference between the two policies was very small and showed that the increase in CTR in the first position for the Re-rank policy was compensated by the decrease for all the other positions.
Given the results from the A/B test, the answer to the second research question, "Is a pricebased re-rank of these offered properties sufficient to achieve a higher CTR?", is clearly yes if the main goal is to improve the CTR in the top position of the list. Despite the data analysis results, which showed that lower prices were a key factor to improve the CTR, a policy that re-rank items by price was only sufficient to improve the CTR w.r.t. the Baseline policy in the first position of the recommended list. However, since users usually pay more attention to the item in first position, this can be considered a good result even if the SCTR slightly decreased with the Re-rank policy.
To further study the differences in CTR and SCTR metrics between the two policies, we analysed further variables with potential influence on user decision-making. Among the considered variables, such as average rating, number of reviews and location of the properties, the Online Travel Agency (OTA) presented with each property emerged as one of the key factors. Here, we focused on the influence of OTAs because it is important to the company's business and we already analysed the other variables in [25].
Figure 4 depicts the CTR at each rank position for the most common OTA and for all the other OTAs. Since we can not disclose the names of the OTAs, we only distinguished between the most common OTA and the other OTAs. The most common OTA always had a CTR that was significantly higher than the CTR of the other OTAs at least for the first 15 positions, which means that users preferred this OTA to the others. One reason for this preference could be that the most common OTA might be more trusted by users. This difference in CTR between OTA groups, joined to the number of recommendations for each OTA group, reported in Figure 5, could explain the difference in SCTR identified between the two policies. From Figure 5a, we can see that the Re-rank policy recommended properties with the most common OTA less frequently at top-ranked positions while favouring more frequently other OTAs, Figure 5b.
Thus, by favouring lower price offers the Re-rank policy pushed less well-known OTAs to top-ranked positions and exposed them to higher levels of users' attention. Their lower likelihood of being clicked, however, seems to have neutralized the positive price effect and resulted in an overall decrease in the SCTR.
The answer to the third research question, "Does the OTA associated with offered properties influence the CTR?", is yes. Although price and rank positions were identified as the most important features that influenced users' decision-making, there were also other factors, in our case the OTA, that could impact users' decision and thus overall performance metrics of a ranking policy. At the end, in our case, a price-based re-rank algorithm that also keep the balance for the OTA feature would probably have improved the baseline, whereas considering only the price was sufficient to achieve a marginal improvement.
In this paper, we studied how the price influenced user click behaviour in online hotel search. We started by analysing a historical dataset collected in a meta-search booking platform in which as expected the price showed a strong influence on CTR. To verify this fact, we ran an online A/B test on the company's website to compare a Baseline policy with a price-based re-rank policy that shuffles the top-25 offered properties in recommendation lists.
The results showed that the re-rank policy improved the CTR for top rank position. This confirmed that price was a key factor influencing users' click behaviour in according to previous works (such as Lockyer [23] and Stávková et al. [24]) where many factors influenced user decision-making, such as cleanliness and quality of properties. However, in the context of RSs, it is usually very difficult, or nearly impossible, to assess the true quality of items. Instead, we found that even a more identifiable feature, such as the OTA associated with a property, influenced the user decision. For example, in our case the most common OTA achieved a higher CTR for every rank position compared to the other OTAs and seems to be favoured more by users.
This work consequently highlights the many influence factors and biases on users decisionmaking in online travel search that are disregarded in most offline datasets by presenting the outcome of a price-based re-rank strategy.