Recommender systems in online marketplaces face the challenge of balancing multiple objectives to satisfy various stakeholders, including customers, providers, and the platform itself. This paper introduces Juggler-MAB, a hybrid approach that combines meta-learning with Multi-Armed Bandits (MAB) to address the limitations of existing multi-stakeholder recommendation systems. Our method extends the Juggler framework, which uses meta-learning to predict optimal weights for utility and compensation adjustments, by incorporating a MAB component for real-time, context-specific refinements. We present a two-stage approach where Juggler provides initial weight predictions, followed by MAB-based adjustments that adapt to rapid changes in user behavior and market conditions. Our system leverages contextual features such as device type and brand to make fine-grained weight adjustments based on specific segments. To evaluate our approach, we developed a simulation framework using a dataset of 0.6 million searches from Expedia's lodging booking platform. Results show that Juggler-MAB outperforms the original Juggler model across all metrics, with NDCG improvements of 2.9%, a 13.7% reduction in regret, and a 9.8% improvement in best arm selection rate.
Recommender systems often focus solely on user satisfaction. However, in many real-world applications,
particularly in online marketplaces, multiple stakeholders’ interests need to be considered [
The Juggler framework [
To address these limitations, we propose a two-step approach that combines meta-learning (Juggler) with Multi-Armed Bandits for multi-stakeholder recommendations for real-time weight adjustments. This approach aims to: 1) provide more granular weight adjustments based on specific segments (e.g., device type, brand) and 2) adapt quickly to changes in trafic patterns without requiring frequent retraining of the main Juggler model. Our research questions are: • Can the integration of MAB with Juggler improve the performance and adaptability of multistakeholder recommendations in online marketplaces? • Are contextual features useful to improve the MAB’s efectiveness at making the right decisions? SURE workshop held in conjunction with the 18th ACM Conference on Recommender Systems (RecSys), 2024, in Bari, Italy.
CEUR
ceur-ws.org
The rest of this paper is organized as follows: Section 2 presents the related work, while Section 3 introduces the proposed hybrid solution. Section 4 covers the experimental setup used to validate the proposal and while Section 5 reports the results to the research questions. Lastly, Section 6 highlights the main conclusions and avenues for future work.
The Juggler framework [
Multi-Armed Bandits (MAB) are a class of reinforcement learning algorithms that balance exploration
and exploitation in decision-making processes [
The integration of meta-learning and bandit algorithms has been explored in other domains, such as
algorithm selection [
Several studies have addressed the challenge of balancing multiple objectives in recommender
systems. Rodriguez et al. [
Recent developments in industry have led to the creation of self-service platforms for deploying
contextual bandits, such as AdaptEx [
We present a hybrid approach that combines the Juggler framework’s meta-learning capabilities [
The Juggler framework selects from five pre-configured options for utility and compensation weights,
providing a coarse adjustment of the recommendation strategy based on the search context. These
options range from lower relevance and compensation to higher relevance and compensation, as
described in [
The MAB component introduces fine-grained adjustments to the Juggler-predicted weights. Each arm of the bandit represents a small corrective measure to be applied to the utility and compensation weights to improve relevance.
The key features of our MAB implementation include: 1. Contextual arms: The contextual bandits consider contextual features (e.g., device type, brand) when selecting arms. 2. Reward function: We use Normalized Discounted Cumulative Gain (NDCG) as a proxy for Conversion Rate, allowing for ofline simulation and evaluation.
balance exploration and exploitation efectively [
+ ) ⋅ ) ⋅ where and
corrective weights determined by the MAB.
are the weights predicted by Juggler, and and are the
We formulate our contextual MAB problem as follows: let be the set of arms, where each arm ∈ represents a pair of corrective weights ( , features such as device or brand. The reward is defined as the NDCG of the resulting ranking. The goal is to find a policy ∶ → that maximizes the expected cumulative reward: ). The context ∈
at time includes =1 max [∑ ( , ( ))] where is the time horizon.
We explored various methods to combine Juggler’s predictions with MAB corrections, ultimately settling on the additive approach described above. We carefully selected contextual variables that would help identify under-performing segments in the Juggler model, such as device type and brand. Balancing multiple objectives in a single reward function required careful consideration. We chose NDCG as an initial approach due to its widely accepted usage, with plans to explore more complex multi-objective reward functions in future work.
To evaluate our hybrid approach, we developed a custom simulator that allows us to test various configurations ofline using historical data. The simulator, built on Expedia data, enables to: 1. Replay historical searches and user interactions. Data is loaded on a daily basis, consisting of data for each property in each search and the respective user clicks and bookings. 2. Apply the Juggler-MAB model to generate new rankings. The MAB is sampled (ppotentially using contextual data) and the retrieved arm is included in the ranking formula, yielding the simulated score and the final ranking. 3. Evaluate the performance using both immediate (e.g., clicks) and delayed (e.g., bookings) feedback.
The reward function evaluates the simulated rankings and information about the arm sampled, reward and contextual information (if any) is provided to the MAB, to update its internal state.
The simulation framework provides a safe environment to test and refine our approach before considering online deployment.
We used a dataset of 0.6 million searches from Expedia’s lodging booking platform, covering a period of 31 consecutive days. The data has over 600000 distinct properties across approximately 41000 distinct destinations, with feedback sparsity over 96%. The dataset includes features such as device type, brand, destination, and historical user interactions.
We compared several variants of the proposed Juggler-MAB hybrid approach against the original
Juggler model [
Distribution of reward value. • -greedy: a classical bandit using a vanilla implementation of the canonical algorithm. We have used = 0.1 and = 0.3 . • Recursive Least Squares with Thompson Sampling (RLS): a contextual bandit using a linear model with a vector of means and a matrix of variances-covariances.
The experiments use the actual production Juggler model predictions for each search. This improves
the reliability of Juggler’s predictions, which in turn leads to more robust estimates of the MAB’s
efect. We then implemented the MAB component using the AdaptEx SDK [
Our Juggler-MAB hybrid approach outperformed the Juggler baseline across all metrics for all bandits proposed. The NDCG improvements range from +0.8% for GT bandit, all the way to +2.9% in several RLS bandits. In terms of regret, we achieve a reduction of 13.7% and an improvement in best arm selection rate of 9.8%.
The -greedy algorithms provide very strong baselines, especially when = 0.1 . GT bandit is clearly the worst bandit, but yet useful since it outperforms the baseline. Among the contextual bandits, the best one across all metrics is the . Interestingly, when using more contextual features, we did not achieve better performance. Further investigations are required to identify what matters to define the context.
We performed Wilcoxon signed-rank tests and observed no statistical diference between all RLS
bandits. The Critical Diference [
Figure 2 shows the learning dynamics for all bandits across all days in the data sample. To improve interpretation, we include only the best contextual bandit. The Juggler-MAB demonstrated fast adaptation to changing conditions. We observed that the MAB component was able to make fine-grained adjustments to the Juggler predictions, resulting in improved performance.
We inspect now Juggler-MAB’s efect on lodging ranking top-10 average statistics in Table 2. The results are reported as diferences to the Juggler baseline, as we cannot expose the sensitive raw data.
Metric daily price guest rating star rating margin % margin $
The results show a clear pattern for all bandits: average daily price decreases and guest and star ratings increase as NDCG improves. On the contrary, margin % and margin $ decreases, which could pose problems to the marketplace objectives and long term health. The expectations, to be validated via AB test, is that the increase in relevance will lead to an improvement in conversion rate which can ofset the impact in profit per transaction.
Diving now deeper into the arms selection per bandit, we present Figure 3. The results show a clear and expected preference towards arms lower compensation weights, as they are not aligned with the NDCG reward. However, it is interesting to observe that the best bandit has learned that not only is it ideal to decrease compensation, but also to increase or decrease relevance depending on the context.
Despite the overall positive results, we identified two limitations. First, the reward function considers
only a single dimension of the problem (i.e. relevance), which explains the impact to the
compensation component. Future work will address this limitation by using multi-objective optimization
techniques [
In this paper, we presented a novel hybrid approach combining Meta-Learning with Multi-Armed Bandits for multi-stakeholder recommendations in online travel marketplaces. Our Juggler-MAB system demonstrated significant improvements over existing methods. Key contributions of our work include 1) an integration of meta-learning and contextual bandits for recommendation systems and 2) empirical evidence of the efectiveness of our approach in a large-scale, real-world setting. Based on our findings and the limitations identified, we propose the following directions for future research: 1. Online testing: Conduct A/B tests in a production environment to validate the performance of
Juggler-MAB under real-world conditions and user behaviors
2. Dynamic arm space: Explore methods for dynamically adjusting the arm space of the MAB
component based on observed performance and changing market conditions.
3. Fairness considerations: Incorporate explicit fairness constraints or objectives into the MAB
formulation to ensure equitable treatment of diferent provider segments [