Mobile gaming has become increasingly popular due to the growing usage of smartphones in day to day life. In recent years, this advancement has led to an interest in the application of in-game recommendation systems. However, the in-game recommendation is more challenging than common recommendation scenarios, such as e-commerce, for a number of reasons: (1) the player behavior and context change at a fast pace, (2) only a few items (few-shot) can be exposed, and (3) with an existing hand-crafted heuristic recommendation, performing randomized explorations to collect data is not a business choice that is preferred by game stakeholders. To that end, we propose an end-to-end model called DFSNet (Debiasing FewShot Network) that enables training an in-game recommender on an imbalanced dataset that is biased by the existing heuristic policy. We experimentally evaluate the performance of DFSNet both in an ofline setup on a validation dataset and online in a real-time serving environment, illustrating the correctness and efectiveness of the trained model.
• Information systems → Recommender systems; • Computing methodologies → Neural networks. In-game recommendation, debiasing, mobile game, feedback loop, few-shot recommendation, A/B test Reference Format: Lele Cao, Sahar Asadi, Matteo Biasielli, and Michael Sjöberg. 2020. Debiasing Few-Shot Recommendation in Mobile Games. In 3rd Workshop on Online Recommender Systems and User Modeling (ORSUM 2020), in conjunction with the 14th ACM Conference on Recommender Systems, September 25th, 2020, Virtual Event, Brazil.
As smartphones expand the gaming market [
In-game recommender systems utilize user interaction data that
describes historical behavior and current context of individual
players to expose each player the right item at the right time. However,
despite a few in-game recommendation trials [
Our literature survey (till the date when this paper is written)
shows that none of the related works [
There are many scenarios where in-game recommendations could be applied. In Figure 1, we exemplify a couple of examples for two of the King1 games: Candy Crush Soda Saga (CCSS) and Candy Crush Saga (CCS). We notice that some occasions allow only one item (a.k.a. one-shot) to be exposed at a time such as Figure 1b, while others (e.g., Figure 1a) can display a few more (a.k.a. few-shot) items. Items can have no values specified as shown in these two examples, or have values attached. To simplify the introduction of our method and experiments, we use the one-shot setup where only one item k with value vk can be recommended upon each trigger of an exposure opportunity. We will show that our approach can be easily applied to scenarios with few-shot exposure and items with no values. The overall optimization objective is to maximize the expected value of the potentially clicked items. In this section, we present a walk-through of our debiasing few-shot recommendation approach. 2.1
Each sample in the dataset corresponds to a complete item exposure event triggered at time t for a player. As illustrated in Figure 2, we calculate the player features, noted as x ∈ RD , using historical data of the last N days before the time t . The D-dimensional features fall into two categories: behavioral (e.g., the total number of game rounds played) and contextual (e.g., the latest inventory status). In addition, at time t , the exposed item k (following a heuristic policy) is recorded. Within the time window that item k is exposed, we log if the player eventually clicks on it or not, which is treated as a
binary label y ∈ {0, 1}2. The raw dataset is extremely biased due to the presence of the existing heuristic policy, and it is imbalanced concerning the label and distribution of exposed item types.
In this section, we propose to train a debiasing few-shot network, DFSNet, to perform a few-shot in-game recommendation using only the heavily biased and imbalanced dataset. The goal of DFSNet is to rank K items where the k-th item has a value vk for a player (that is represented by a D-dimensional feature vector x ∈ RD ), in order to maximize the expected click value. As shown in Figure 3, DFSNet consists of three modules: preference predictors, confidence predictors, and meta ranking. The training is conducted in a minibatch fashion; the input is a matrix X = {x(m) M }m=1 ∈ RM ×D , where M is the number of samples in each mini-batch and D is the number of features in each sample. For the sake of conciseness, we use the general terms x and X to denote any sample and mini-batch, respectively. 2.2.1 Preference Predictors. For a player x, the preference predictor module (cf. the red dashed bounding box in Figure 3) predicts the probability yˆk that player x will click item k if this item is exposed. During training, the mini-batch X is firstly divided into K subsets (noted as X1, . . . , XK ), so that the k-th subset Xk ∈ RMk ×D only contains the Mk players that were exposed to the k-th item. As a result, each item k has its own architectural branch, which sequentially propagates Xk through a sample balancer and a preference predictor, and eventually yields the click/non-click probability DYk ∈ RMk ×2.
Since the number of clicked items usually represents a small
fraction of the entire exposed item set, there are far more negative
samples (y=[
XHk = Xk+ ∪ XHk−, where XHk ∈ R[max(min(Mk+, Mk−),1)+Mk+]×D . (1)
This minority subsampling balancer is conceptually similar to the
negative sampling in [
The output of the Sample Balancer from the k-th branch (i.e. XHk
in Figure 3) is then fed to a preference predictor implemented with
a 4-layer Deep Neural Network (DNN) binary classifier. The ELU
(Exponential Linear Unit) activation function [
Lp = 21K kXK=1 MH1k mXMH=k1 −yk(m) ∗ log yˆ k(m) 1 , (2) where “∗” represents the element-wise multiplication, MHk = max(min(M+, M−), 1) + M+ is the number of samples in XHk , k k k the notation y(m) is the label (one-hot encoded vector) of the m-th k sample in XHk , and yˆ k(m) is the predicted probability vector for the same sample. 2.2.2 Confidence Predictors. To explicitly model the bias from the pre-dominant heuristic, we introduce a confidence predictor module (cf. the green dashed bounding box in Figure 3) to DFSNet. The confidence predictors estimate the probability ck that player x has recently been exposed to the item k. Thus, cˆk can be treated as an approximation of the confidence we have for the predicted click probability yˆk . Similar to preference predictors, this module also employs K branches (for K items), each of which has a sample balancer and a DNN binary classifier.
The mini-batch input X ∈ RM ×d is fed into the sample balancer of each branch indiscriminately. To prevent the confidence predictors from simply memorizing the heuristic rule and lose the generalization capability, it is important to remove the features (if any) that are used in heuristic policy, hence X’s second dimension d may be smaller than the original dimension D. The sample balancer in this module first divides X into two subsets Xk ∈ RMk ×d and X¬k ∈ R(M −Mk )×d , where Xk only contains Mk players exposed to item k, and X¬k has the rest M − Mk samples. Due to the preexisting heuristic policy, the item exposure was not randomized, making the size of Xk and X¬k imbalanced. To that end, we need a sample balancer for each branch to produce a balanced mini-batch Xk using
Xk = X¬k ∪ Xk′ ∈ R[max(M −Mk,1)]×d Xk ∪ X¬′k ∈ R[max(Mk,1)]×d where Xk′ and X¬′k are obtained via minority subsampling (without replacement); specifically, the former term contains max(M − Mk , 1) samples randomly selected from Xk , and the latter contains max(Mk , 1) randomly picked samples from X¬k . Mk denotes the number of samples in Xk , hence Xk ∈ RMk ×d .
Xk is then fed to a confidence predictor implemented in the same way as in preference predictors except that each DNN is scaled proportionally to factor Mk / PK
k=1 Mk . The loss Lc to optimize this module has a similar form to Equation (2): , Mk ≥ M − Mk , (3) , Mk < M − Mk
Lc = 21K kXK=1 M1k mXM=k1 −ck(m) ∗ log cˆk(m) 1 , (4)
where ck(m) ∈ {0, 1}2 is the constructed confidence label specifying
if the m-th player/sample in Xk actually saw item m or not, and cˆk(m)
is the predicted confidence probability vector for the same sample.
The preference and confidence predictors are jointly optimized with
a total loss L = Lp + Lc .
2.2.3 Meta Ranking. During model serving/prediction, the sample
balancers will be omitted, meaning that the input X ∈ RM ×D
(representing M players) will be directly fed to DNNs in all branches,
in order to simultaneously generate real-valued preference (DY ∈
[
, otherwise Rk = yˆk (6) cˆk · yˆk
We can conveniently assume that K items are already sorted by their values v, hence the propensity scores R = [R1, . . . , RK ] are also sorted accordingly. An overly drastic change of item exposure (e.g. a player who used to see item 1 according to the heuristic which suddenly gets item K from a newly deployed recommender system) may undermine the player experience and game ecosystem. To avoid that situation, it is a good practice to enforce a heuristic deviation threshold (noted as ks ∈ {1, . . . , K − 1}) in the online production environment. Specifically, we mask Rk with (7) 0 RHk = Rk , |kh − k | > ks , , otherwise where kh is the item from the pre-existing heuristic policy. With HR = [RH1, . . . , RHK ], both one-shot and few-shot in-game recommendation are possible. When recommending items based on HR, we can sometimes choose to apply ϵ-greedy to slowly accumulate more diversified data for follow-up model iterations. 3
We apply DFSNet to a real-time item recommendation scenario for the CCSS game. There is a total of five items ( K =5) in this scenario, yet only one item k can be shown on the mobile screen when the player triggers the exposure event. The item k has a value vk . Items are sorted by value in an ascending order, i.e., v1 < v2 < v3 < v4 < v5. If a player clicks on the exposed item k, a value vk will be added to the game ecosystem; and we choose to maximize the value of the clicked item. The details of the concrete use case and items are considered to be sensitive proprietary information and therefore anonymized in this paper.
As illustrated in Figure 4, the raw dataset is collected (cf.
Section 2.1) using a Flink2 based stateful streaming platform [
Events
Item Game Clients
Game Servers
Events
Item
Features
Prediction Streaming Cluster (Flink-based)
Tensorflow Serving Cluster Online (with A/B test capability) Data (BigQuery)
Trained Model
Machine Learning
Platform Offline (e.g., min-max, z-score, and logarithmic) to numerical features and perform either one-hot encoding or embedding to categorical features. The dataset pre-processing and model development is carried out on a machine learning platform developed by King.
We will present both ofline (training and evaluation) and online
(serving and monitoring) evaluation of DFSNet in the following
sections. DFSNet is implemented in Tensorflow 3; the preference and
confidence DNNs are scaled as depicted in Figure 3. The training
is carried out with Adam optimizer [
To perform ofline model evaluation, we create a validation dataset (noted as U = {x(u ) U
}u=1) by randomly selecting 1% data from the raw dataset (thus U ≈ 0.22 million), and use the rest for training. 3.1.1 Generalization evolvement during training. We evaluate the performance of the current model on the validation dataset during the training. Since the datasets are highly imbalanced, accuracy is not an informative metric to monitor during training. We also find that recall and precision are having a hard time competing with each other (showing no clear trend) during the training, hence not ideal for monitoring the training performance. AUC-ROC (Area Under the Curve of Receiver Operating Characteristics), on the other hand, is a stable metric that reliably tells how much the model is capable of distinguishing between classes; therefore, the evolution of peritem AUC-ROCs (see Figure 5) indicates how the generalization ability of the model improves during the training process. At the end of the training, we also measure the recall and precision for each item, which are visualized as red bars in Figure 12. 3.1.2 Policy change quantization: heuristic vs. DFSNet. We use the trained DFSNet to obtain predictions on the validation dataset. We ifrst measure the overall change of item exposure distribution. The results are reported in Figure 6a. In our experiment, we observed no significant change in item allocation for players due to the strong confidence constraint imposed, yet there is a slight shift towards the higher-valued items. The ratio of players that see a diferent item (than heuristic) is about 7.4%. To decompose the policy change, 3https://www.tensorflow.org we illustrate, in Figure 6b, a Policy Transition Matrix (PTM), where each cell at position (i, j) indicates the ratio of players who were supposed to get item j, according to heuristic policy, but are now exposed to item i according to DFSNet. It can be seen that the diagonal has the majority of the unchanged exposures, and each row largely follows a truncated normal distribution. 3.1.3 Distribution of preference and confidence predictions. For each sample in the validation dataset, DFSNet produces ten probabilities: five click probabilities ( yˆ1 to yˆ5) and five confidence probabilities (cˆ1 to cˆ5). Figure 7 visualizes yˆk and cˆk jointly to answer four questions: (1) Does yˆk reflect the low click ratio of item k? The five red area plots on the diagonal are the distributions of yˆk , all of which show that clicking tends to be a rare event. (2) Does cˆk match the exposure ratio of item k? The five green bar plots on the diagonal represent the distributions of cˆk ; the majority of exposures come from item 1, which coincides with the heuristic item exposure distribution in Figure 6a. (3) Does yˆk show general item preference? The lower triangular portion has pair-wise scatter plots of click probabilities. Each data point in the plot for item i and j has a coordinate of (yˆi , yˆj ), thus if the point is below the line of yˆi =yˆj , the corresponding player prefers item i over j, and vice versa. To examine the general trend, we fit linear models (red straight lines going through the original points) for pair-wise plots. We observe that in average, players prefer items with lower values. (4) Can DFSNet be confident with multiple items for the same player? The upper triangular portion in Figure 7 contains pair-wise scatter plots of confidence probabilities cˆk . Every point in the plot for item i and j is located at (cˆi , cˆj ). Intuitively, implied by Equation (5), the points (representing players) in the green shaded areas are likely eligible to more than one item. 3.1.4 Best-efort estimation of recall, precision, and uplifts. On the ofline validation dataset U ∈ RU ×D , it is impossible to measure the “quality” of a recommendation that is diferent than what was actually exposed; hence, a sub-optimal solution is to create a subset (from U) containing only the players for whom both DFSNet and the heuristic policy recommended the same item. We use U′ ∈ RU ′×D (U ′ < U ) to denote that subset. On that subset, we measure peritem recall and precision for preference predictors (cf. the red (a) Item 1 (b) Item 2 (c) Item 3 (d) Item 4 (e) Item 5 (f) ROC for all items (g) Item 1 (h) Item 2 (i) Item 3 (j) Item 4 (k) Item 5 (l) ROC for all items bars in Figure 12). To provide uplift baselines of average click rate ( #_clicked_items ) and click value ( total_value_of_clicked_items ), we #_items #_clicked_items calculate both metrics for both the heuristic and DFSNet policies. The results are presented in Table 1. Ofline uplifts will be then compared with the ones obtained during online model serving (cf. Section 3.2.3). 3.2
After the DFSNet model is trained and validated in an ofline environment, it is then deployed in a Tensorflow Serving 4 cluster. As illustrated in Figure 4, a prediction client (sharing the entire feature collection logic described in Section 2.1) is also deployed on the streaming cluster. To validate the online performance of the DFSNet model, we run an A/B test on a small fraction of players on CCSS.
For each game player in the test group, the prediction client makes a request to the DFSNet prediction service (in real-time) as soon as any pre-defined triggering event emerges. In the life cycle of a real-time recommendation system, it is often required to iterate on the model serving periodically (cf. Figure 8 for an example of two serving iterations) to incorporate bug fixes or new models trained on more recent data.
During online serving, we track several metrics (aggregated into temporal windows of 5 minutes) to monitor the key system performance, some examples of which include model response time, model exceptions, and model raw output distribution. The definition of those system metrics remains the same for diferent recommendation models. These metrics are indicators of the system health, and therefore, they play a critical role in the validity of the model. To monitor model performance, we log all features, predictions, and labels in a BigQuery5 database, and visualize them
in a dashboard that is updated on hourly basis. We will hereafter emphasize our online evaluation on several key perspectives, all of which are adapted from the monitoring dashboard. 3.2.1 Heuristic deviation trend. The foremost questions to answer about model performance are twofold: (1) what the scale of the model impact is and (2) how this impact evolves along the timeline. To answer these two questions, we illustrate the overall heuristic deviation trend of two adjacent model serving iterations in Figure 8, where the red curve shows the ratio of players (in the DFSNet A/B test group) for which DFSNet and the heuristic policy recommended diferent items. As reported in Section 3.1.2, this ratio is approximately 7.4% (represented by a straight green line) when measured on the ofline validation dataset. So, the expectation is that the ratio of impact should reach around 7.4%; this trend can be clearly seen in each model serving iteration. However, some input features need individual players to respond to certain game components, which takes about three days in the use case discussed here; and ta tic 0.10
s e i tg r a eu 0.08 th h rsey tahn 0.06 la le fop nubd 0.04 to tn 0.02 i a e r r
e ehT iffd 0.00 ay1 D ay5
D Previous model serving ay10 D ay15 D items are served using the heuristic policy to players that still have incomplete features. As a result, for each model serving iteration, the ratio always starts from a fairly low point before reaching 7.4%. Furthermore, between two subsequent iterations, the ratio drops, in this use case, for three days while rebuilding features before picking up the ascending trend again. We believe that the daily monitoring of the heuristic deviation trend helps tracking the scale of the model impact efectively. 3.2.2 Player Transition Matrix (PTM). To have a better insight of the underlying changes contributing to the overall model impact, we break down the impact analysis further using PTMs. Here, a PTM is a 2D matrix that puts players into each grid cell according to how their experience changed from the default heuristic policy (rows) to the model policy (columns). Figure 9 summarizes the results over a period of 14 days since Day11 (cf. Figure 8) and illustrates four diferent PTMs (a-d) that enable model impact decomposition from four diferent perspectives: (a) Grid cell (i, j ) denotes the number of players that are now exposed to the item i but would have originally got item j. The gray-scale background of Figure 9a is also used in (c) and (d). (b) Calculated by dividing each number in (a) by the sum of the corresponding row. It should have the highest ratio values on the diagonal, as obtained in the ofline evaluation and reported in Figure 6b. Because the presence of incomplete features leads to a policy fallback (to the heuristic, as mentioned in Section 3.2.1), the results presented in Figure 9b are more conservative than those shown in Figure 6b. (c) To inspect how the moved players impact the click probability on PTM, we calculate the click percentage (of item i) for the player cohort in each grid cell, and then overlay the percentage values on top of Figure 9a’s gray-scale background, resulting in Figure 9c. The blue values in the diagonal cells are the click percentage for the control group. Each of-diagonal cell (i, j )i,j contains the players that are moved from cell (j, j ) to cell (i, i ). Intuitively, we expect the model to guarantee the click percentage in (i, j )i,j to be larger than either percentage values in cell (i, i ) or (j, j ); we use red boxes to highlight the cells that fail to satisfy that expectation. In practice, it is acceptable to have a few red boxes as long as most of the densely populated cells satisfy the expectation. (d) The PTM in Figure 9d principally serves the same purpose as Figure 9c except that it computes the average click value of item i instead.
Based on the PTM analysis, we expect to see an improvement in the user engagement with the examined game feature compared to using the heuristic solution. Further analysis of the PTM can help us to understand better user behaviors. In our analysis, we used an eight-dimensional space to describe players behavior. Figure 10 shows diferent user behavioral patterns (in the form of radar charts) on top of the grayscale background from Figure 9a. The KPI calculation and the actual values are considered to be sensitive proprietary data and therefore removed from the charts. 3.2.3 Uplifts of click ratio, count, and value. The previous sections presented a drill-down process of analyzing the model behavior in comparison with the heuristic policy. We now zoom out and compare the item click dynamics with the control group. We choose to focus on the accumulated 14-day uplift of three metrics: click count, click ratio, and click value (these metrics are defined in Section 3.1.4). The online uplift is computed by subtracting the metrics (normalized by the population size of A/B test groups) of the control group from the ones of the DFSNet group. Hence, uplift can have both positive and negative values. All uplift values are considered sensitive proprietary data and therefore scaled.
Figure 11a shows that DFSNet group is losing clicking counts
on items 1 and 4 while gaining more click counts on other items;
therefore, the players moved away from those buckets are mostly
item clickers, and they bring more absolute click counts for items 2,
3, and 5. However, the click ratio of item 5 is reversed in Figure 11b,
which is a consequence of the lower click ratio in the player cohort
moved to item 5. Figure 11c shows the uplift of accumulated click
value for each item. We observe that the loss of click value (from
items 1 and 4) is compensated by the increased click value for items
2, 3, and 5, leading to a net positive value uplift (approximately
+0.71% over the control group). As a result, the ofline uplift
estimations (Table 1) are overly optimistic compared to that measured
online.
(a) Number of players “moved” from j to i.
(b) Ratio version of (a): each row sums to 1.
(c) Click percentage of item i for players who should see item j
according to the heuristic.
(d) Average click value of item i for players who should see item j
according to the heuristic.
3.2.4 Iterating on the DFSNet model. The dataset used to train
DFSNet is highly biased due to the pre-existing heuristic rules. Our
approach achieves debiasing by incorporating confidence
predictors, thus demonstrating a mild impact of less than 8% (cf. Figure 8).
Nonetheless, that impact continuously changes the players’
experiences, which nudges the input feature distributions around. This
creates a direct feedback loop, which gradually compromises the
generalization and discriminative capability of the model being served.
It is a form of analysis debt [
In Figure 12, the red bars represent the precision and recall estimated using a subset of the validation dataset (as explained in Section 3.1.4), while the green bars in Figure 12a and 12b are respectively precision and recall calculated 14 days after the model got deployed (a snapshot on Day24 in Figure 8). We observe that online precision and recall reach much higher values than the ofline reference initially, hence we argue that the ofline evaluation tends to underestimate the true values of precision and recall. Figure 12c and 12d reflect the situation four days later. The trend is clear: in four days, both precision and recall have declined significantly; when the majority of green bars go under the red bars, it is probably the time to retrain/finetune the DFSNet using fresher data. 4
In-game recommendation aims to enable providing more relevant items to each player. The in-game recommendation use cases usually allow exposing only a few items at a time; thus, change in the choice of items can have a large impact on game dynamics leading to a short feedback loop. In addition, player preferences change quickly due to the change in the game dynamics and player context. As a result, the model gets outdated sooner in real-time prediction. In-game item exposures are mostly dominated by some hand-crafted heuristics, which heavily bias the data, and randomized exploration to train an unbiased recommendation model is usually not favored by stakeholders. We propose DFSNet that enables training an unbiased few-shot recommender using only the biased and imbalanced data.
During training, AUC-ROC is a stable indicator of the modelâĂŹs generalization ability. We demonstrate several ways to estimate the model performance ofline on a validation dataset. We also evaluate the online DFSNet performance in an A/B test. We start with monitoring the overall model impact by looking at the heuristic deviation trend. Then, we further decompose the model impact using
PTMs. We carried out data analysis to understand user behaviors and discern the key factors causing players to be exposed to a diferent item than the heuristic recommendation. This work proposes a solution to address the problem of bias and imbalanced data in the domain of in-game recommender systems. We suggest ofline and proxy metrics as a way to have an estimate of model online performance. We discuss and showcase the challenges of an online solution in an A/B test. The comparison between the control and DFSNet test groups show a net +0.71% uplift of click value, which is less optimistic than the best-efort ofline estimation. We show that (a) Precision of click prediction up till Day24. (b) Recall of click prediction up till Day24. (c) Precision of click prediction up till Day28. (d) Recall of click prediction up till Day28. continuous comparison of ofline and online precision/recall can help determine the appropriate time to retrain the model. Further analysis is required before putting the proposed solution live. In the scenario presented in this paper, we chose the click-through rate as one of the evaluation metrics, which is widely adopted in e-commerce. However, this metric might not be a good proxy for business metrics for an in-game recommender system. Future work will explore the choice of metrics that constitute a better proxy of the model’s online performance.
In addition, future work includes (1) designing long-term labels that better approximate the business targets, (2) explicitly modeling the interactions between diferent in-game features to eliminate the implicit feedback loop, and (3) replacing model iteration with online reinforcement learning approaches.