=Paper=
{{Paper
|id=Vol-3924/short5
|storemode=property
|title=Addressing bias in Recommender Systems: A Case Study on Data Debiasing Techniques in Mobile Games
|pdfUrl=https://ceur-ws.org/Vol-3924/short5.pdf
|volume=Vol-3924
|authors=Yixiong Wang,Maria Paskevich,Hui Wang
|dblpUrl=https://dblp.org/rec/conf/robustrecsys/WangPW24
}}
==Addressing bias in Recommender Systems: A Case Study on Data Debiasing Techniques in Mobile Games==
https://ceur-ws.org/Vol-3924/short5.pdf
Addressing bias in Recommender Systems: A Case Study on Data
Debiasing Techniques in Mobile Games
Yixiong Wang1,† , Maria Paskevich1,∗,† and Hui Wang1
1
King, Malmskillnadsgatan 19, 111 57 Stockholm, Sweden
Abstract
The mobile gaming industry, particularly the free-to-play sector, has been around for more than a decade, yet it still experiences rapid
growth. The concept of games-as-service requires game developers to pay much more attention to recommendations of content in their
games. With recommender systems (RS), the inevitable problem of bias in the data comes hand in hand. A lot of research has been
done on the case of bias in RS for online retail or services, but much less is available for the specific case of the game industry. Also,
in previous works, various debiasing techniques were tested on explicit feedback datasets, while it is much more common in mobile
gaming data to only have implicit feedback. This case study aims to identify and categorize potential bias within datasets specific to
model-based recommendations in mobile games, review debiasing techniques in the existing literature, and assess their effectiveness on
real-world data gathered through implicit feedback. The effectiveness of these methods is then evaluated based on their debiasing
quality, data requirements, and computational demands.
Keywords
Recommender systems, In-game recommendation, Debiasing, Mobile games
1. Introduction publicly available datasets; (3) to adapt and apply debias-
ing strategies, originally developed for explicit feedback
In the context of mobile gaming, delivery of content to play- data, to the implicit feedback data specific to King, and (4)
ers through recommendations plays an important role. It to evaluate and compare the efficacy of different methods
could include elements such as, for example, in-game store based on the quality of debiasing, data requirements, and
products or certain parts of content. However, RSs used computational complexity.
within this context are susceptible to bias due to (1) lim-
ited exposure: unlike in webshops (e.g. Amazon), available
placements for sellable products in mobile games are often 2. Related work
limited, and showing one product to a user means that alter-
natives would not be displayed; (2) the common approach of The existing literature on addressing debiasing techniques in
segmenting content through fixed heuristics before adopting RS presents a well-structured and categorized list of method-
RS introduces biases in the training data, which influences ologies [2][3]. It suggests that the selection of particular
the development of these models. Traditionally, at King debiasing techniques should depend on the specific types
we have been addressing these biases by either training of bias present in the data, as well as on the availability of
models on biased data, or by establishing holdout groups unbiased data samples. In recommender systems for mobile
of users who would receive random recommendations for games, various types of bias can arise, including but not
a period of time in order to collect a uniform dataset that limited to selection bias, exposure bias, position bias, and
reflects user preference in an unbiased way. Although the conformity bias. Some of the relevant methods to debias the
second approach allows the collection of unbiased data, it data in these cases could be The Inverse Propensity Scoring
could compromise user experience for a segment of players, (IPS) [4] method, which deals with selection and exposure
and may lead to significant operational costs and poten- biases by weighting observations inversely to their selection
tial revenue losses. In previous studies, researchers have probability, and does so without need for unbiased data. Yet
primarily focused on data derived from explicit feedback, the method could potentially result in high variance due
where users rate items using a numerical scale, and vari- to the challenges in accurately estimating propensities. Po-
ous debiasing techniques are tested on this data. However, tential solutions to the high variance issue of IPS method
within the realm of mobile gaming, obtaining explicit feed- include, for example, using Doubly Robust (DR) learning
back affects from user experience, making it challenging to [5] that introduces a novel approach to loss functions as
collect. As an alternative, data is often collected through a combination of IPS-based models with imputation-based
implicit feedback [1], where user preferences are inferred models. The combination of two models assures doubly
from behaviors such as impressions, purchases, and other robustness property when either of the two components
interactions. Given these challenges, our objectives in this (propensity estimation or imputed data) remains accurate.
study are: (1) to identify and categorize potential bias within This method, though, relies on having an unbiased data
our datasets; (2) to conduct a review of existing literature sample to work. Another option is model-agnostic and bias-
on debiasing techniques and assess their effectiveness on agnostic solutions like AutoDebias [6], which are based on
meta-learning to dynamically assign weights within the RS,
aiming to neutralize biases across the board. A potential
RobustRecSys: Design, Evaluation, and Deployment of Robust Recom- benefit of such solution is that it doesn’t require knowing the
mender Systems Workshop @ RecSys 2024, 18 October, 2024, Bari, Italy.
∗
Corresponding author. types of bias present in the data, but as a downside, it also
†
These authors contributed equally. relies on randomized samples. In addition, the process of fit-
Envelope-Open wyx.ei.99@gmail.com (Y. Wang); maria.paskevich@king.com ting multiple models makes training more computationally
(M. Paskevich); maddy.hui.wang@king.com (H. Wang) demanding. Despite the advances and variety of available
Orcid 0000-0001-8904-2052 (Y. Wang); 0009-0006-6211-1824 debiasing techniques, applying Recommendation Systems
(M. Paskevich); 0009-0004-4190-9410 (H. Wang)
© 2025 Copyright for this paper by its authors. Use permitted under Creative Commons License
to mobile gaming content remains a relatively untapped
Attribution 4.0 International (CC BY 4.0).
CEUR
ceur-ws.org
Workshop ISSN 1613-0073
Proceedings
Table 1
The sizes and feedback types of all datasets used in this study.
A key difference is that the open datasets (COAT and YahooR3!)
provide explicit feedback, while the proprietary datasets (A, B,
and C) offer only implicit feedback (purchase/no purchase). Set A,
a proprietary dataset, lacks randomized data, limiting debiasing
options.
Dataset Biased samples Unbiased samples Feedback type
COAT 311k 54k Explicit
yahooR3! 12.5k 75k Explicit
Set A 47.6k - Implicit
Set B 100k 218k Implicit
Set C 980k 1.2mln Implicit
some items displayed in more appealing placements while
others are not visible on the first screen (Fig. 1). Another
Figure 1: Examples of content placements in Candy Crush Soda bias, selection bias, arises from imbalanced product impres-
Saga (left) and Candy Crush Saga (right), highlighting biases: sions, where certain items—such as conversion offers—are
selection bias with a prominently placed product (left) and ex-
shown to users more frequently, resulting in significantly
posure bias with limited visibility, where products are hidden
behind the ”More Offers” button (right). higher exposure for those items.
3.2. Selection of Debiasing techniques
area, with most of the publications focusing on building The primary reasoning for the selection of debiasing tech-
recommendations [7] [8] [9], and not on issues of imbalance niques for this study was based in a literature review, and
and bias. Previous efforts at King introduced DFSNet [10], included the applicability of each method to the specific
an end-to-end model-specific debiasing technique that en- biases present in the propreitery datasets—namely, selec-
ables training an in-game recommender on an imbalanced tion bias, exposure bias, and position bias. Further, it was
dataset without randomized data. This work aims to en- imperative to evaluate techniques across two dimensions:
rich King’s debiasing toolkit by exploring model-agnostic those that require randomized datasets and those that do
solutions, specifically focusing on the challenges of content not, as well as to examine methodologies that are agnostic
recommendations within mobile games. However, the archi- to any particular type of bias. Given the identified biases in
tecture of DFSNet is complex, involving multiple modules, the datasets, we adopted several debiasing techniques: (1)
which can make the implementation and maintenance chal- Matrix Factorisation (MF) as a baseline model, Inverse
lenging. Moreover, it requires constant feedback loops over Propensity Scoring (IPS), a method that does not require
time and the model’s performance is highly dependent on randomized data collection and primarily addresses selec-
the quality and recency of the training data. tion and exposure biases. (2) Doubly Robust learning,
that tackles the same biases but, unlike IPS, requires a ran-
domized dataset. And (3) AutoDebias (DR), a bias-agnostic
3. Methodology technique that also needs randomized data. Each method
was tested across all datasets to evaluate model performance
3.1. Datasets and complexity. We initially applied MF to biased dataset 𝐷𝑇
Our study utilized two public datasets (COAT[4], ya- to establish metrics for comparison, we denote our baseline
hooR3![13]) to validate theoretical results and three pro- model as MF(biased), then compared these outcomes with
prietary datasets from King (Set A, Set B, Set C) that are the results from the debiasing methods.
focused on user-item interactions in game shops within
Match-3 Game A and Match-3 Game B (Fig.1). The sizes of 3.3. Evaluation metrics
each dataset, along with their respective feedback types, are
provided in Table 1. We aimed to observe the effectiveness For models’ evaluation, we use metrics that assess both
of different techniques on datasets collected with explicit predictive power of the models (RMSE and AUC), as well as
feedback (public datasets), and those with implicit feedback quality of ranking (NDCG@5) and inequality and diversity
(King’s datasets). Explicit feedback is typically collected by in the recommendations (Gini index and Entropy):
asking users to rate items on a numerical scale, for example
• NDCG@5 assesses the model’s ability to rank rele-
from 1 to 5, where 1 indicates disinterest, 2 signifies dissat-
vant items in the recommendation list:
isfaction, and 5 shows a preference. In contrast, Implicit
feedback (as in the proprietary datasets) involves a binary 𝑘
DCG@k 2𝑟𝑒𝑙𝑖 − 1
response from users: purchase or non-purchase. This setup NDCG@k = , DCG@k = ∑ ,
IDCG@k 𝑖=1 log2 (𝑖 + 1)
makes it harder to accurately measure user preferences. As
discussed in the Introduction, mobile games often have lim- where IDCG@k is the ideal DCG@k and 𝑟𝑒𝑙𝑖 repre-
ited space for displaying sellable products, which is the case sents items ordered by their relevance up to position
for all three proprietary datasets. This limitation leads to ex- k.
posure bias in the data. Additionally, placement of different
products within the game shop creates positional bias, with
Figure 2: Debiasing results on open datasets (COAT and yahooR3!). The graphs show the percentage change in metrics
(AUC, RMSE, NDCG@5, Gini, and Entropy) for various models relative to MF(biased). AUC is plotted against other metrics to
demonstrate the trade-off between diversity gains in recommendation systems and potential compromises in predictive power.
Different models are represented by colors, training times are indicated by point sizes, and dataset types are distinguished by
shapes.
Table 2
Percentage improvement of various models compared to MF(biased) across open datasets. The best results for each metric are
highlighted in bold.
Dataset Model RMSE AUC NDCG Gini Entropy Training time (sec)
IPS -2.53% -0.26% -1.18% 0.62% -0.29% 8.82%
COAT DR 3.86% -1.57% 2.75% -18.88% 6.16% 194.12%
AutoDebias -5.06% 0.39% 3.73% 0.16% 0.00% 767.65%
IPS -29.70% -0.55% 0.73% -6.33% 0.82% -22.98%
yahooR3! DR -30.39% -0.83% 0.00% 1.22% -0.12% 412.56%
AutoDebias -36.89% 1.79% 20.70% -58.15% 4.26% 3215.87%
• RMSE measures the magnitude of prediction errors Additionally, we include Training Time, defined as the
of exact rating predictions: time required for each model to reach saturation, measured
in seconds. This metric provides insights into the compu-
1 tational complexity and the resources required by different
RMSE = ∑ (𝑟 ̂ − 𝑟 )2 ,
|𝑅| (𝑢,𝑖)∈𝑅 𝑢𝑖 𝑢𝑖 methodologies.
√
where |𝑅| denotes the total number of ratings in the
dataset, 𝑟𝑢𝑖
̂ and 𝑟𝑢𝑖 are predicted and true ratings for 4. Experimentation
all user-item pairs (𝑢, 𝑖).
• AUC reflects how well the model distinguishes be- We regard biased data as training set, 𝐷𝑇 . When it comes
tween positive and negative interactions: to randomized data, following the strategies as mentioned
+
(|𝐷te +
|+1)⋅|𝐷te |
in [11], we split it into 3 parts: 5% for randomised set 𝐷𝑈
∑(𝑢,𝑖)∈𝐷te+ rank𝑢,𝑖 − 2 to help training as required by DR and Autodebias, 5% for
AUC = + | ⋅ (|𝐷 | − |𝐷 + |)
, validation set 𝐷𝑉 to tune hyper-parameters and incur early-
|𝐷te te te
stopping mechanism to prevent overfitting, the rest 90% for
where 𝐷te + is the number of positive samples in test
test set 𝐷𝑇 𝑒 to evaluate the model. For conformity reasons,
set 𝐷te , and 𝑟𝑎𝑛𝑘𝑢,𝑖 denotes the position of a positive the data split strategy mentioned above is applied to both
feedback (𝑢, 𝑖). In experimentation, AUC mainly open datasets and proprietary datasets. For this project, we
served as a metric to prevent overfitting and help deploy a training pipeline on Vertex AI [12], integrating
fine-tunning in validation phase. components such as data transformation powered by Big-
• Gini index measures inequality in the recommen- Query, model training and evaluation, as well as experiment
dations distribution. The higher coefficient indicates tracking. The training pipeline retrieves data from the data
higher inequality warehouse to train models and produces artifacts that are
𝑛 later integrated into an experiment tracker. By adopting
∑𝑖=1 (2𝑖 − 𝑛 − 1) 𝜙(𝑖)
𝐺= this artifact-based approach, we address the inherent chal-
𝑛
𝑛 ⋅ ∑𝑖=1 𝜙(𝑖) lenge of reproducibility in operationalizing ML projects, as
Where 𝜙𝑖 is the popularity score of the 𝑖-th item, with it provides all the necessary components to reproduce ex-
the scores 𝜙𝑖 arranged in ascending order (𝜙𝑖 ≤ 𝜙𝑖+1 ), periments. Each experiment is run up to 10 times on Vertex
and 𝑛 represents the total number of items. AI with the same hyper parameters, but varying random
• Entropy measures the diversity in the distribution seeds to get estimation on the variability of the results.
of recommended items with higher values indicating A pipeline plays a pivotal role in enhancing machine
higher diversity. learning processes within the industry by automating each
𝑛
step from data fetching to model evaluation. For this project,
𝐸𝑛𝑡𝑟𝑜𝑝𝑦 = − ∑ 𝑝𝑖 log(𝑝𝑖 ), a training pipeline was implemented on Vertex AI, encom-
𝑖=1 passing components such as data transformation utilizing
where 𝑛 is a total number of items u in a dataset and BigQuery, model training, model evaluation, and experi-
𝑝𝑖 is a probability of an item being recommended. ment tracking. All the experiments were conducted within
Figure 3: Debiasing results on internal datasets (Set A, Set B and Set C). The graphs show the percentage change in metrics
(AUC, RMSE, NDCG@5, Gini, and Entropy) for various models relative to MF(biased). AUC is plotted against other metrics to
demonstrate the trade-off between diversity gains in recommendation systems and potential compromises in predictive power.
Different models are represented by colors, training times are indicated by point sizes, and dataset types are distinguished by
shapes.
Table 3
Percentage improvement of various models compared to MF(biased) across internal datasets. The best results for each metric
are highlighted in bold.
Dataset Model RMSE AUC NDCG Gini Entropy Training time (sec)
Set A IPS 20.95% -0.97% -1.53% -3.06% 0.41% -4.72%
IPS -8.61% 3.18% -0.14% 3.29% -0.02% -12.23%
Set B DR -45.40% 7.07% 0.68% -0.54% 0.00% 386.46%
AutoDebias -26.46% -1.25% -0.48% 3.26% -0.02% -63.26%
IPS 39.01% -23.46% -29.36% -9.47% 9.04% -15.50%
Set C DR 7.74% -13.76% -28.44% -5.36% 5.47% 14.74%
AutoDebias 64.50% 2.61% -0.01% 1.72% -2.47% 233.93%
this framework, ensuring consistency, efficiency, and preci- 5.2. Internal Datasets
sion throughout the development lifecycle.
For the internal datasets, the results are less consistent
across the datasets and debiasing techniques (Table 3). This
5. Experimentation results may be due to the fact that internal datasets employed im-
plicit feedback when collecting data, where user preferences
The absolute results of all experiments, including confi- are inferred from their impression and purchase records.
dence intervals, are presented in Table 4. In this section, This can introduce biases due to the lack of negative sam-
we report the percentage improvement of various debias- ples and overrepresentation of user interactions, potentially
ing techniques compared to the baseline model, which was skewing the models towards popular items.
trained on biased data (MF(biased) model). Set A is a relatively small dataset (Table 1), and the lack of
randomized data limits our options to only using IPS. As a re-
5.1. Open Datasets sult, some metrics, such as RMSE and AUC, actually worsen
(Table 3), which we might accept as a trade-off to achieve
For the COAT dataset, the results show varying degrees of better balance in recommendations. However, NDCG@5
improvement across different metrics (Table 2). The top per- also does not improve. On the positive side, IPS enhances
forming method (AutoDebias), exhibited the best improve- diversity metrics, with Gini improving by 3.06% and En-
ments in RMSE (-5.06%), AUC (0.39%) and NGCG@5 (3.73%) tropy by 0.41%, while also reducing computational cost by
with low changes in Gini (0.16%) and no improvement in En- 4.27%. Overall, applying this method increases model diver-
tropy. DR also provided higher gains in NDCG@5 (2.75%), sity with comparable training time, but comes at the cost of
and performed better in Gini (-18.88%) and Entropy (6.16%), accuracy.
but at a cost of higher RMSE (3.86%) and lower AUC (-1.57%). Set B demonstrates substantial improvements with DR,
While AutoDebias outperformed other techniques when it including a 45.40% reduction in RMSE, a 7.07% increase in
comes to improving predictive power of the model (AUC, AUC, and gains in NDCG@5 (0.68%) and Gini (-0.54%), mak-
RMSE), it was not very efficient in terms of Gini and En- ing the model perform better in both accuracy and diversity.
tropy, and has a significantly higher computational cost. However, this comes at a significant computational cost,
This highlights a trade-off between improved accuracy and increasing training time by 386.46%. Given the total number
increased resource requirements. of samples being 318k, this leads to a considerably longer
For YahooR3! dataset, again, AutoDebias results in training process. AutoDebias ranks second in RMSE im-
the highest improvement in RMSE (-36.89%), AUC (1.79%), provement (-26.46%), while IPS shows a positive gain in
NDCG@5 (20.70%), as well as Gini (-58.15%) and Entropy AUC (3.18%). However, DR is the only method that consis-
(4.26%), but did so also with dramatically increased compu- tently improves outcomes of NDCG@5, Gini, and Entropy.
tational cost (3216%). IPS provides a balanced performance For Set C, the largest dataset with nearly 2.2 million
with improvements in RMSE (-29.70%) and Entropy (0.82%) samples, AutoDebias achieves the highest improvement
at a lower computational cost (-22.98%), making it a practical in AUC (2.61%) and maintains stable NDCG@5. However,
choice for resource-constrained environments. it underperforms compared to the baseline and other tech-
Table 4
Performance metrics across different models and datasets, with 95% confidence intervals.
Dataset Model RMSE AUC NDCG@5 Gini Entropy Training time (sec)
MF (uniform) 1.00 ± 0.02 0.54 ± 0.01 0.36 ± 0.02 0.64 ± 0.01 4.91 ± 0.02 2.00 ± 1.60
MF (biased) 0.75 ± 0.01 0.77 ± 0.01 0.51 ± 0.01 0.64 ± 0.04 4.9 ± 0.11 3.40 ± 1.00
COAT IPS 0.73 ± 0.01 0.76 ± 0.01 0.50 ± 0.01 0.65 ± 0.04 4.89 ± 0.10 3.70 ± 2.30
DR 0.78 ± 0.02 0.75 ± 0.01 0.52 ± 0.01 0.52 ± 0.01 5.20 ± 0.03 10.00 ± 6.90
AutoDebias 0.71 ± 0.01 0.77 ± 0.02 0.53 ± 0.01 0.64 ± 0.06 4.90 ± 0.14 29.50 ± 9.6
MF (uniform) 0.73 ± 0.01 0.57 ± 0.01 0.43 ± 0.01 0.41 ± 0.01 6.58 ± 0.01 4.80 ± 1.20
MF (biased) 0.86 ± 0.01 0.73 ± 0.01 0.55 ± 0.01 0.41 ± 0.01 6.58 ± 0.01 60.50 ± 12.20
yahooR3! IPS 0.61 ± 0.01 0.72 ± 0.01 0.55 ± 0.01 0.39 ± 0.01 6.63 ± 0.02 46.60 ± 16.10
DR 0.60 ± 0.04 0.72 ± 0.01 0.55 ± 0.01 0.42 ± 0.01 6.57 ± 0.01 310.10 ± 54.60
AutoDebias 0.54 ± 0.01 0.74 ± 0.01 0.66 ± 0.01 0.17 ± 0.01 6.86 ± 0.01 2006.10 ± 1541.00
MF (biased) 0.82 ± 0.07 0.54 ± 0.02 0.56 ± 0.02 0.36 ± 0.01 2.83 ± 0.01 694.30 ± 163.30
Set A
IPS 0.99 ± 0.02 0.54 ± 0.01 0.55 ± 0.01 0.35 ± 0.02 2.84 ± 0.02 661.5 ± 85.9
MF (uniform) 0.61 ± 0.00 0.92 ± 0.01 0.97 ± 0.00 0.10 ± 0.00 1.77 ± 0.00 2891.00 ± 126.90
MF (biased) 0.81 ± 0.06 0.89 ± 0.00 0.97 ± 0.00 0.10 ± 0.00 1.80 ± 0.00 2123.90 ± 441.3
Set B IPS 0.74 ± 0.14 0.92 ± 0.01 0.97 ± 0.00 0.10 ± 0.00 1.77 ± 0.00 1864.10 ± 86.70
DR 0.44 ± 0.02 0.95 ± 0.01 0.96 ± 0.01 0.10 ± 0.01 1.77 ± 0.00 10332.00 ± 2486.30
AutoDebias 0.56 ± 0.02 0.88 ± 0.01 0.96 ± 0.01 0.10 ± 0.00 1.77 ± 0.00 780.30 ± 153.70
MF (uniform) 0.92 ± 0.04 0.25 ± 0.02 0.07 ± 0.01 0.52 ± 0.01 2.52 ± 0.02 775.90 ± 265.00
MF (biased) 0.62 ± 0.01 0.84 ± 0.01 0.80 ± 0.01 0.65 ± 0.01 2.18 ± 0.02 650.80 ± 114.70
Set C IPS 0.86 ± 0.06 0.64 ± 0.05 0.56 ± 0.08 0.59 ± 0.01 2.37 ± 0.02 549.90 ± 128.30
DR 0.67 ± 0.02 0.72 ± 0.05 0.57 ± 0.09 0.61 ± 0.02 2.29 ± 0.05 746.70 ± 140.00
AutoDebias 1.02 ± 0.03 0.86 ± 0.04 0.78 ± 0.02 0.66 ± 0.02 2.12 ± 0.04 2173.20 ± 1826.10
niques in RMSE, Gini, Entropy, and training time, which References
increases significantly by 233.93%. IPS, on the other hand,
delivers poor results in RMSE (39.01%), AUC (-23.46%), and [1] Oard, Douglas W.& Jinmook Kim. Implicit feedback for
NDCG@5 (-29.36%), but excels in Gini (-9.47%) and Entropy recommender systems. Proceedings of the AAAI work-
(9.04%) without adding to the training time. shop on recommender systems. Vol. 83. 1998.
[2] Jiawei Chen and Hande Dong and Xiang Wang and Fuli
Feng and Meng Wang & Xiangnan He, Bias and Debias
6. Conclusion and Future work in Recommender System: A Survey and Future Directions.
ACM Trans. Inf. Syst. 41, 3, Article 67 (2023)
Implementing more accurate and less biased models is cru- [3] Harald Steck, Training and testing of recommender sys-
cial to avoiding the perpetuation of negative feedback loops tems on data missing not at random. In Proceedings of
and the overexposure of certain items caused by segmen- the 16th ACM SIGKDD international conference on
tation heuristics in retraining data. This approach also Knowledge discovery and data mining (KDD 2010). As-
enhances data quality, which is essential for fine-tuning sociation for Computing Machinery, New York, NY,
models. A recommender system that diversifies content USA, 713–722
exposure improves user experience by ensuring that vis- [4] Tobias Schnabel, Adith Swaminathan, Ashudeep Singh,
ibility is not limited to only the most popular items. In Navin Chandak, and Thorsten Joachims, Recommenda-
our experiments, Inverse Propensity Scoring (IPS) stands tions as Treatments: Debiasing Learning and Evaluation.
out for its simplicity and model-agnostic nature, requiring In Proceedings of the 33rd International Conference on
no randomized data collection and fewer training epochs. International Conference on Machine Learning - Vol-
However, the improvements it offers are somewhat limited. ume 48 (ICML 2016). JMLR.org, 1670–1679.
AutoDebias excels in improving accuracy metrics, but at [5] Quanyu Dai, Haoxuan Li, Peng Wu, Zhenhua Dong,
substantially higher computational costs and sometimes Xiao-Hua Zhou, Rui Zhang, Rui Zhang, and Jie Sun, A
poorer performance in Gini and Entropy. DR still offers Generalized Doubly Robust Learning Framework for Debi-
strong improvement in observed metrics, including Gini asing Post-Click Conversion Rate Prediction. In Proceed-
and Entropy. So while each debiasing method has its own ings of the 28th ACM SIGKDD Conference on Knowl-
trade-offs, significant performance gains still depend on the edge Discovery and Data Mining (KDD 2022). Associ-
challenging task of collecting randomized datasets, as high- ation for Computing Machinery, New York, NY, USA,
lighted in our introduction. Potential future work includes: 252–262.
(1) adopting online reinforcement learning approach such [6] Jiawei Chen, Hande Dong, Yang Qiu, Xiangnan He, Xin
as Multi-Armed Bandit (MAB) [14, 15, 16] for data collec- Xin, Liang Chen, Guli Lin, and Keping Yang, AutoDebias:
tion, including contextual bandit models, (2) developing Learning to Debias for Recommendation. In Proceedings
and testing combined debiasing models which can com- of the 44th International ACM SIGIR Conference on
bine strengths of different debiasing techniques to mitigate Research and Development in Information Retrieval
various biases simultaneously while optimizing for compu- (SIGIR 2021). Association for Computing Machinery,
tational efficiency. New York, NY, USA, 21–30.
[7] Andrés Villa, Vladimir Araujo, Francisca Cattan, and
Denis Parra, Interpretable Contextual Team-aware Item
Recommendation: Application in Multiplayer Online Bat-
tle Arena Games. In Proceedings of the 14th ACM Con-
ference on Recommender Systems (RecSys 2020)
[8] Qilin Deng, Kai Wang, Minghao Zhao, Zhene Zou,
Runze Wu, Jianrong Tao, Changjie Fan, and Liang Chen
Personalized Bundle Recommendation in Online Games.
In Proceedings of the 29th ACM International Confer-
ence on Information & Knowledge Management (CIKM
2020)
[9] Meng Wu, John Kolen, Navid Aghdaie, and Kazi A.
Zaman. Recommendation Applications and Systems at
Electronic Arts. In Proceedings of the Eleventh ACM
Conference on Recommender Systems (RecSys 2017)
[10] Cao, Lele & Asadi, Sahar & Biasielli, Matteo & Sjöberg,
Michael. Debiasing Few-Shot Recommendation in Mo-
bile Games. Workshop of ACM Conference on Recom-
mender Systems (RecSys 2020)
[11] Liu, Dugang and Cheng, Pengxiang and Dong, Zhen-
hua and He, Xiuqiang and Pan, Weike and Ming, Zhong
A general knowledge distillation framework for counter-
factual recommendation via uniform data. Proceedings
of the 43rd International ACM SIGIR Conference on
Research and Development in Information Retrieval
(SIGIR 2020)
[12] Google. 2024, Vertex AI. Retrieved December 1, 2023
from https://cloud.google.com/vertex-ai,
[13] Marlin, Benjamin M., and Richard S. Zemel. Collabo-
rative prediction and ranking with non-random missing
data. Proceedings of the third ACM conference on Rec-
ommender systems (RecSys 2009)
[14] Felício, Crícia Z., Klérisson VR Paixão, Celia AZ Barce-
los, and Philippe Preux. A multi-armed bandit model
selection for cold-start user recommendation. In Proceed-
ings of the 25th conference on user modeling, adapta-
tion and personalization, pp. 32-40. 2017.
[15] Wang, Lu, Chengyu Wang, Keqiang Wang, and Xi-
aofeng He. Biucb: A contextual bandit algorithm for
cold-start and diversified recommendation. In 2017 IEEE
International Conference on Big Knowledge (ICBK), pp.
248-253. IEEE, 2017.
[16] Wang, Qing, Chunqiu Zeng, Wubai Zhou, Tao Li,
S. Sitharama Iyengar, Larisa Shwartz, and Genady Ya
Grabarnik. Online interactive collaborative filtering using
multi-armed bandit with dependent arms. IEEE Trans-
actions on Knowledge and Data Engineering 31, no. 8
(2018): 1569-1580.