Recommender systems have long grappled with optimizing user satisfaction using only implicit user feedback. Many approaches in the literature rely on complicated feedback modeling and costly user studies. We propose online recommender systems as a candidate for the recently introduced Interaction Grounded Learning (IGL) paradigm. In IGL, a learner attempts to optimize a latent reward in an environment by observing feedback with no grounding. We introduce a novel personalized variant of IGL for recommender systems that can leverage explicit and implicit user feedback to maximize user satisfaction, with no feedback signal modeling and minimal assumptions. With our empirical evaluations that include simulations as well as experiments on real product data, we demonstrate the efectiveness of IGL for recommender systems.
listening sessions [11], half of the clicked on content was actually disliked by users.
The last decade has seen unprecedented growth in e- Challenge 2: Incorporating multiple implicit feedback commerce, social media and digital streaming oferings, signals requires manual feature engineering. In addition resulting in users that are overwhelmed with content to clicks, user implicit feedback can include dwell time and choices. Online recommender systems ofer a way [3], mouse movement [12], scroll information [13] and to alleviate this information overload and improve user gaze [14]. One popular approach uses dwell time to filter experience by providing personalized content. Unfor- out noisy clicks, with the reasoning that satisfied users tunately, optimizing user satisfaction is challenging be- stay on pages longer [3]. Although the industry standard cause explicit feedback indicating user satisfaction is rare is 30+ seconds of dwell time for a “meaningful” click, this in practice [1]. To resolve the problem of data sparsity, number actually varies depending on the page topic, readpractitioners rely on implicit signals such as clicks [2] or ability and content length [15]. It is equally challenging dwell time [3] as a proxy for user satisfaction. However, to incorporate other signals, for example, behaviors such designing an optimization objective using implicit signals as viewport time, dwell time and scroll patterns have a is nontrivial, and many modern recommender systems complicated temporal relationship and represent prefersufer from the following challenges. ence in diferent phases [ 10]. There is an extensive body
Challenge 1: No one implicit signal is the true user satis- of work on modeling diferent implicit feedback signals faction signal. User clicks are the most readily available [16, 17], however these niche models may not generalsignal, and the Click-Through Rate (CTR) metric has be- ize well across a diverse user base, or stay relevant as come the gold standard for evaluating the performance of recommender systems and their users evolve. online recommendation systems [4]. Yet there are many To tackle these challenges, we propose online recominstances when a user will interact via clicks and be un- mender systems as a candidate for Interaction-Grounded satisfied with the content. The most familiar of these is Learning (IGL) [18]. IGL is a learning paradigm where clickbait, where poor quality content attracts user clicks a learner optimizes for latent rewards by interacting by exploiting cognitive biases such as caption bias [5], with the environment and associating observed feedback position bias [6] or the curiosity gap [7, 8]. Optimization with the unobservable true reward. Although IGL was of the CTR will naturally promote clickbait items that originally inspired by brain-computer interface applicaprovide negative user experiences and cause distrust in tions, in this paper we demonstrate that the framework, the recommender system [9]. Recent studies show that when utilizing a diferent generative assumption and augclicks may even be a signal of user dissatisfaction. In lab- mented with an additional latent state, is also well suited oratory studies of online news reading [10] and Spotify for recommendation applications. Existing approaches ORSUM@ACM RecSys 2022: 5th Workshop on Online Recommender such as reinforcement learning and traditional contextual Systems and User Modeling, jointly with the 16th ACM Conference on bandits sufer from the choice of reward function. HowRecommender Systems, September 23rd, 2022, Seattle, WA, USA ever IGL resolves the 2 above challenges while making © 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License minimal assumptions about the value of observed user CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g ACttEribUutRion W4.0oInrtekrnsahtioonpal (PCCroBYce4.0e).dings (CEUR-WS.org) feedback. Our new approach is able to incorporate both et al. [19] loosen full conditional independence by conexplicit and implicit signals, leverage ambiguous user sidering context conditional independence, i.e. ⟂ |, . feedback and adapt to the diferent ways in which users For our setting, this corresponds to the user feedback interact with the system. varying for combinations of preference and content, but
Our Contributions. We introduce IGL for recom- remaining consistent across all users. Neither of these mender systems, allowing us to leverage implicit and two assumptions are applicable in the setting of online explicit feedback signals and mitigate the need for re- content recommendation because diferent users interward engineering. We present the first IGL strategy for act with recommender systems in diferent ways. This context-dependent feedback, the first use of inverse kine- is evidenced by our production data from a real world matics as an IGL objective, and the first IGL strategy image recommendation system (see Sec. 4.3) along with for more than two latent states. Using simulations and existing results in the literature [20, 21]. By assuming real production data, we demonstrate that recommender user-specific communication rather than item-specific systems require at least 3 reward states, and that IGL communication, we allow for personalized reward learnis able to address two big challenges for modern online ing. recommender systems. Number of Latent Reward States. Prior work shows the binary latent reward assumption, along with an assumption that rewards are rare under a known reference 2. Background on policy, is suficient for IGL to succeed. Specifically, opInteraction-Grounded Learning timizing the contrast between a learned policy and the oblivious uniform policy is able to succeed when feedProblem Statement. Consider a learner that is inter- back is both context and action independent [18]; and acting with an environment while trying to optimize optimizing the contrast between the learned policy and their policies without access to any grounding or explicit all constant-action policies succeeds when the feedback reward signal. At each time step, the stationary environ- is context independent [19]. ment generates a context ∈ which is sampled i.i.d. Although the binary latent reward assumption (e.g., from a distribution 0. The learner observes the context satisfied or dissatisfied) appears reasonable for recomand then selects an action ∈ from a finite action set. mendation scenarios, it fails to account for user indiferIn response, the environment jointly generates a latent ence versus user dissatisfaction. This observation was reward and feedback vector ( , ) ∈ ℛ × conditional on ifrst motivated by our production data, where a 2 state (, ) . However, the learner is only able to observe and IGL policy would sometimes maximize feedback signals not . Since the latent reward can be either deterministic with obviously negative semantics. Assuming users igor stochastic, let (, ) ∶= (,) [ ] denote the expected nore most content most of the time [22], negative feedreward after choosing action for context . In the IGL back can be as dificult to elicit as positive feedback, and a setting, the context space and feedback vector space 2 state IGL model is unable to distinguish between these can be arbitrarily large. Let ∈ Π ∶ → Δ( ) denote extremes. Hence, we posit a minimal latent state model a stochastic policy, with corresponding expected return for recommender systems involves 3 states: (i) = 1 , ( ) ∶= (,)∼ 0× [ ] . In IGL, the learner’s goal is to when users are satisfied with the recommended content, ifnd the optimal policy ∗ = argmax∈Π ( ) , while only (ii) = 0 , when users are indiferent or inattentive, and able to observe context-action-feedback (, , ) triples. (iii) = −1 , when users are dissatisfied.
In the recommender system setting, the context is the user, the action is the recommended content and 3. Derivations the feedback is the user feedback. Unfortunately existing IGL approaches ([18], [19]) leverage assumptions We now address the first of the previously mentioned designed for classification and control tasks which are challenges from Sec. 1. For the recommender system seta poor fit for recommendation scenarios: (i) context- ting, we use the assumption that ⟂ |, , namely that independence of the feedback and (ii) binary latent re- the feedback is independent of the displayed content wards. given the user and their disposition toward the disFeedback Dependence Assumptions. It is information played content ∈ {−1, 0, 1} . Thus, we assume that users theoretically impossible to solve IGL without assump- may communicate in diferent ways, but a given user tions about the relation between , and [19]. In the expresses satisfaction, dissatisfaction and indiference in ifrst paper on IGL, the authors assumed full conditional the same way. independence of the feedback on the context and chosen The statistical dependence of on frustrates the action, i.e. ⟂ , | . For recommender systems, this un- use of learning objectives which utilize the product of desirably implies that all users communicate preferences marginal distributions over (, ) . Essentially, given aridentically for all content. In the following paper, Xie bitrary dependence upon , learning must operate on
In this section we motivate our inverse kinematics strategy using exact expectations. When acting according to any policy (|)
, we can imagine trying to predict the action taken given the context and feedback; the posterior distribution is (| , ) = = (|) = (|) = (|) (|) ( |, ) ∑ ∑ ∑ ( |) ( | , , )
( |) ( | , ) ( |) ( | , ) ( |) = ∑ ( | , )
( |, ) (, ) ∑ ( |, ) (|) ( |, ) ( |, ) We arrive at an inner product between a reward decoder term ( | , )
and a reward predictor term (|,)(|) ∑ (|,)(|)
.
Direct extraction of a reward predictor using maximum
tion (
biguate the extreme events. We assume partial reward in(Bayes rule) ( ⟂ |, ) treme event disambiguation to one-sided learning [23] formation is available via a “definitely negative” function dn ∶ × → {−1, 0}
where ( dn(, ) = 0| = 1) = 1 and ( dn(, ) = −1| = −1) > 0
. This reduces exapplied only to extreme events, where we try to predict the underlying latent state given (, ) . We assume partial labelling is selected completely at random [24] and treat the (constant) negative labelling propensity . (Total Probability) as a hyperparameter. We arrive at our 3-state reward action is taken. Because feedback is assumed conditionally independent of action, the only way for feedback to help predict which action is played is via the (action dependence of the) latent reward.
With 2 latent states, ≠ 0
⟹ = 1 , and we can reduce
to a standard contextual bandit with inferred rewards
1( (| , ) > 2 (|))
. With 3 latent states, ≠ 0
⟹
(
. Under these assumptions we have = 0 ⟹ (| , ) =
( = 0|, ) (|) ∑ ( = 0|, ) (|) ≤ 2 ( = 0|, ) (|) ≤ 2 (|).
(
Note a feedback merely being apriori rare or frequent (i.e., the magnitude of ( |) under the policy (|) ) ⎧ ⎨ ⎩ 0 −1 (| , ) > 2 (|) (| , ) ≤ 2 (|) otherwise and dn(, ) = −1 equivalent to Zhang and Lee [25, Equation 11] scaled by . Note setting = 1 embeds 2-state IGL.
is known but the other probabilities
is estimated online using maxiare estimated. ( |̂ , ) mum likelihood on the problem predicting from (, ) , i.e., on a data stream of tuples ((, ), )
. The current estimates induce ( ,̂ , )
based upon the plug-in
version of equation (
As an additional complication, although (|) is known, it is typically a good policy under which rewards are not rare (e.g., ofline learning with a good historical policy; or acting online according to the policy being learned by the IGL procedure). Therefore we use importance weighting to synthesize a uniform action distribu
from the true action distribution.1 Ultimately we arrive at the procedure of Algorithm 1. 1When the number of actions is changing from round to round, we use importance weighting to synthesize a non-uniform action distribution with low rewards, but we elide this detail for ease of exposition.
Observe context and action set with | | = . determine which feedback are associated with which # # # # # # for evaluation but never revealed to the algorithm.
Simulator Design. Before the start of each experiment,
user profiles with fixed latent rewards for each action are
generated. The users are also assigned predetermined
communication styles, so the probability of emitting a
given signal conditioned on the latent reward remains
static throughout the duration of the experiment. For
the available feedback, users can provide feedback using
ifve signals: (
We now implement Algorithm 1 for 2 latent states as
IGL-P(
Algorithm 1 IGL, Inverse Kinematics and either 2 or 3
Input: Calibrated weighted multiclass classification algorithm MC-Alg.
DN(…) = 0 for 2 state IGL
1: ←
new CB-Alg. 2: IK ← new MC-Alg. 3: for = 1, 2, … ; do if On-policy IGL then (⋅|
) ← . predict( , ).
Play ∼ (⋅| ) and observe feedback . else ( ̂ if ( ̂ ̂ = 0 else
Observe ( , , , (⋅| )). ← 1/( ( | )).
Predict action probability | , ) ← IK.predict(( , ), , ).
| , ) ≤ 2 then . learn( , , , = 0, )
# ̂ ≠ 0 if DN(…) = 0 then . learn( , , , = , ) else
# Definitely negative IK.learn(( , ), , , ).
. learn( , , , = −1, )
Due to the sensitivity around production metrics and
Fig. 2 shows the distribution of rewards for IGL-P(
General Evaluation Setup. At each time step , the con- between satisfied and dissatisfied users. text is provided from either the simulator (Sec. 4.1-4.2) or the logged production data (Sec. 4.3). The learner then selects an action and receives feedback . In these evaluations, each user provides feedback in exactly one interaction and diferent user feedback signals are mutually
exclusive, so that is a one-hot vector. In simulated en- system setting, we now explore the performance of IGLvironments, the ground truth reward is sometimes used
P(
(a) Two latent states
(b) Three latent states
quires direct grounding from the dislike signal. We next
examined how IGL-P(
(a) Ground truth learning curves, P(dislike|r = −1) = 0.2.
(b) Efect of varying P(dislike|r = −1).
Our production setting is a real world image recommendation system that serves hundreds of millions of users.
In our recommendation system interface, users provide
feedback in the form of clicks, likes, dislikes or no
feedback. All four signals are mutually exclusive and the user
IGL-P(
Clicks
Likes
Dislikes
We presented IGL for recommender systems, an approach to producing personalized recommendations that can leverage rich and diverse types of user feedback signals.
In this paper, we showed that IGL can elegantly sidestep complicated manual reward engineering and efectively learn how to maximize user satisfaction with minimal good your recommender system is? a survey on ference on Web search and data mining, 2014, pp. evaluations in recommendation, International Jour- 193–202. nal of Machine Learning and Cybernetics 10 (2019) [16] W. Wang, F. Feng, X. He, L. Nie, T.-S. Chua, De813–831. noising implicit feedback for recommendation, in: [5] K. Hofmann, F. Behr, F. Radlinski, On caption bias Proceedings of the 14th ACM international conin interleaving experiments, in: Proceedings of the ference on web search and data mining, 2021, pp. 21st ACM international conference on Information 373–381.
and knowledge management, 2012, pp. 115–124. [17] Z. Liang, S. Huang, X. Huang, R. Cao, W. Yu, Post[6] K. Hofmann, A. Schuth, A. Bellogin, M. d. Rijke, Ef- click behaviors enhanced recommendation system, fects of position bias on click-based recommender in: 2020 IEEE 21st International Conference on Inevaluation, in: European Conference on Informa- formation Reuse and Integration for Data Science tion Retrieval, Springer, 2014, pp. 624–630. (IRI), IEEE, 2020, pp. 128–135. [7] M. Potthast, S. Köpsel, B. Stein, M. Hagen, Clickbait [18] T. Xie, J. Langford, P. Mineiro, I. Momennejad, detection, in: European conference on information Interaction-grounded learning, in: International retrieval, Springer, 2016, pp. 810–817. Conference on Machine Learning, PMLR, 2021, pp. [8] K. Scott, You won’t believe what’s in this paper! 11414–11423.
clickbait, relevance and the curiosity gap, Journal [19] T. Xie, A. Saran, D. J. Foster, L. Molu, I. Momenneof pragmatics 175 (2021) 53–66. jad, N. Jiang, P. Mineiro, J. Langford, Interaction[9] W. Wang, F. Feng, X. He, H. Zhang, T.-S. Chua, grounded learning with action-inclusive feedback, Clicks can be cheating: Counterfactual recommen- arXiv preprint arXiv:2206.08364 (2022). dation for mitigating clickbait issue, in: Proceed- [20] J. Beel, S. Langer, A. Nürnberger, M. Genzmehr, The ings of the 44th International ACM SIGIR Confer- impact of demographics (age and gender) and other ence on Research and Development in Information user-characteristics on evaluating recommender Retrieval, 2021, pp. 1288–1297. systems, in: International Conference on Theory [10] H. Lu, M. Zhang, S. Ma, Between clicks and satis- and Practice of Digital Libraries, Springer, 2013, pp. faction: Study on multi-phase user preferences and 396–400. satisfaction for online news reading, in: The 41st [21] D. Shin, How do users interact with algorithm International ACM SIGIR Conference on Research recommender systems? the interaction of users, & Development in Information Retrieval, 2018, pp. algorithms, and performance, Computers in Human 435–444. Behavior 109 (2020) 106344. [11] H. Wen, L. Yang, D. Estrin, Leveraging post-click [22] T. T. Nguyen, P.-M. Hui, F. M. Harper, L. Terveen, feedback for content recommendations, in: Pro- J. A. Konstan, Exploring the filter bubble: the efect ceedings of the 13th ACM Conference on Recom- of using recommender systems on content diversity, mender Systems, 2019, pp. 278–286. in: Proceedings of the 23rd international conference [12] M. Ishwarya, G. Swetha, S. Saptha Maaleekaa, on World wide web, 2014, pp. 677–686.
R. Anu Grahaa, Eficient recommender system [23] J. Bekker, J. Davis, Learning from positive and by implicit emotion prediction, in: Advances in unlabeled data: A survey, Machine Learning 109 Big Data and Cloud Computing, Springer, 2019, pp. (2020) 719–760.
173–178. [24] C. Elkan, K. Noto, Learning classifiers from only [13] L. Peška, P. Vojtáš, Estimating importance of im- positive and unlabeled data, in: Proceedings of plicit factors in e-commerce recommender systems, the 14th ACM SIGKDD international conference in: Proceedings of the 2nd International Confer- on Knowledge discovery and data mining, 2008, pp. ence on Web Intelligence, Mining and Semantics, 213–220.
2012, pp. 1–4. [25] D. Zhang, W. S. Lee, A simple probabilistic approach [14] E. Sood, S. Tannert, D. Frassinelli, A. Bulling, N. T. to learning from positive and unlabeled examples, Vu, Interpreting attention models with human vi- in: Proceedings of the 5th annual UK workshop on sual attention in machine reading comprehension, computational intelligence (UKCI), 2005, pp. 83–87. arXiv preprint arXiv:2010.06396 (2020). [26] N. Neophytou, B. Mitra, C. Stinson, Revisiting pop[15] Y. Kim, A. Hassan, R. W. White, I. Zitouni, Mod- ularity and demographic biases in recommender eling dwell time to predict click-level satisfaction, evaluation and efectiveness, in: European Conferin: Proceedings of the 7th ACM international con- ence on Information Retrieval, Springer, 2022, pp. 641–654.