We develop a decision making framework to cast the problem of learning a ranking policy for search or recommendation engines in a two-sided e-Commerce marketplace as an expected reward optimization problem using observational data. As a value allocation mechanism, the ranking policy allocates retrieved items to designated slots to maximize the user utility from the slotted items at any given stage of the shopping journey. The objective of this allocation can in turn be defined with respect to the underlying probabilistic user browsing model as the expected number of interaction events on presented items matching the user intent, given the ranking context. Recognizing the efect of ranking as an intervention action to inform user interactions with slotted items and the corresponding economic value of interaction events for the marketplace, we formulate the expected reward of the marketplace as the collective value from all presented ranking actions. The key element in this formulation is the notion of context value distribution, which signifies not only the attribution of value to ranking interventions within a session but also the distribution of marketplace reward across user sessions. We build empirical estimates for the expected reward of the marketplace from observational data that account for the heterogeneity of economic value across session contexts as well as the distribution shifts in learning from observational user activity data. The ranking policy can then be trained by optimizing the empirical expected reward estimates via standard Bayesian inference techniques. We discuss the connections and distinctions between our proposed perspective and the standard supervised approach to learning to rank via empirical risk minimization with respect to standard information retrieval metrics. The specific focus of this paper is to highlight the significance of the empirical context value distribution in shaping the properties of the corresponding ranking policies by contrasting various empirical importance sampling distributions. We report empirical results from online randomized controlled experiments on a product search ranking task in a major e-commerce platform demonstrating the fundamental trade-ofs governed by ranking polices trained on empirical reward estimates with respect to extreme choices of the context value distribution.
Two-sided e-commerce marketplaces are intermediary economic platforms that connect buyers and sellers, usually providing a wide selection of products for the buyers from a diverse array of sellers. The primary buyer-focused objective of the marketplace is to guide buyers through their search and discovery journeys to identify and purchase items that fulfill their shopping intention. Users’ browsing and purchase journeys in the marketplace are impacted by an ecosystem of decision making systems, most notably via the ranking policies in various stages of their shopping journeys from discovery pages to the Search Engine Result Pages(SERP). An efective ranking policy aims to showcase a set of results that match the intent of the user in any given ranking context along the shopping journey with rewards realized as interaction events on the slotted items on the page. Collectively, user journeys are not equally likely to produce value for the marketplace, and the goal is to expand the set of successful user sessions, optimizing a suitable notion of long-term value for users and the marketplace. It is therefore essential that the ranking policy account for the utility of all stakeholders in this economic setting. In standard formulations of learning to rank in the information retrieval literature, however, there is usually no clear connection between the training objective for the ranking policy, the long-term value for the collective of the users and the key performance metrics of the marketplace. In this paper, focusing primarily on the search ranking policy invoked in response to users’ search queries, we formulate the ranking policy learning as an optimization problem based on a (counterfactual) estimate of the marketplace reward from observational data.
Contribution 1.1. We propose a decision making framework establishing explicit connections between learning a ranking policy for a search/recommendation engines and building efective empirical estimates for a suitable notion of marketplace expected reward.
The problem of developing merit scores for ranking items, post a selection stage from
a large pool of candidates, is widely studied in the context of recommendation systems[
Contribution 1.2. We characterize the key elements in building efective (policy-dependent) expected reward estimates from observational data, controlling for (1) the heterogeneity of the session value distribution, (2) the contribution of interventions within a user journey via the reward attribution scheme, and (3) the distribution shifts incurred by selection biases in observational data.
Reinforcement learning(RL) is a powerful framework to account for sequential interventions
within the session by formulating the problem of recommending new items[
Contribution 1.3. We highlight the significance of the empirical session-context value distribution in building efective marketplace expected reward estimates by demonstrating fundamental performance trade-ofs governed by the search ranking policies trained on extreme choices of the context value distribution via rigorous counterfactual evaluations as well as online randomized controlled experiments in a major e-commerce platform.
The definition of success events and the associated reward to the user events is flexible in our framework and is informed by the strategic choices of the marketplace. Specifically, an early-stage marketplace may focus on maximizing the collective number of engagements, while an acquisition-oriented marketplace targets the collective number of purchases, while a revenue-driven marketplace chooses to maximize the long-term gross merchandise value.
Here is a list of notation adopted throughout the paper. Sets and ordered sets(lists) are represented with upper-case calligraphic symbols; such as . Random quantities are shown in bold such as x with realization . The expected value of random variable x is denoted by E[x] and the conditional expectation of a random variable z = (x, y) given y is denoted by E[z|y] or Ex∼ P()[z]. For a function : X → R, the | | dimensional array [ ()]∈ is denoted by ( ).
The marketplace is interested in maximizing the average total reward across all user session
trajectories over a long time horizon
1 ∑︁ ,
≤
(1)
where is the economic value from a successfully served search session . Our framework is
lfexible in the choice of the reward function and we discuss the fundamental trade-ofs between
multiple strategic marketplace long term reward choices, namely revenue-based, value per
engagement and value per acquisition marketplaces. The reward from a session trajectory is
assumed to be non-negative. Although our framework, can be extended to account for negative
rewards, we ignore it in our formalization. We assume that the reward over search journeys
is a stationary ergodic stochastic process. By invoking Birkhof’s ergodic theorem[
While our formalization can naturally be extended to search journeys with complex goals, we focus on a typical e-commerce purchase decision making scenario of session trajectory a with a single product intent, ignoring sessions with multi-product purchase intent, as well as informational and navigational search sessions. We recognize that users’ decisions are impacted by multiple independently optimized decision making systems, but we are oblivious to potential interactions of the ranking policy with these systems, specifically to the closely related query understanding and candidate retrieval policies. We only focus on policy learning for search result pages with a single layer presentation semantic where the action of the ranking policy is the permutation/ranking of a largely homogeneous set of comparable items for a flat singlelayered presentation of the results, ignoring the multiplicity of user’s search intent and diversity considerations for the result set.
We cast the problem into a Bayesian decision making framework with a user-focused perspective on the definition of success upon a ranking action. The ranking policy aims to increase the expected number of engagements on items that meet the user intent, and the reward is proportional to the likelihood of a success event(non-zero reward) from the user interactions on the search results page produced by the ranking policy. A crucial aspect of this framework is to account for distribution shifts in observational data, i.e. the distinction between the distribution of the logged search activity data that the policy is trained and the inference time distribution of user events.
Given a search query within a session context , the ranking policy : → {1, · · · , } maps a candidate item d from the retrieved set to a slot (). The notion of success with respect to a slotting () of the items on the SERP is defined based on the efectiveness of the policy in driving user interaction events(Click). Specifically, the objective of the ranking policy on a given ranked SERP is to increase the expected number of engagements on desirable items(suitably defined) () given the session context upon issuing the query ≺ ; i.e.
E[c ()|≺ ], where the expectation is with respect to the randomness in user preferences and browsing behaviors in the given query context and possibly the randomness in the ranking policy(if stochastic). Note that ≺ subsumes all the relevant contextual information upon issuing the query ; including all the queries and the corresponding surfaced items, as well as the engaged items prior to the current query context. In the subsequent sections, we will make probabilistic assumptions on users’ browsing and click behaviors and build efective policy dependent empirical estimators.
The notion of success with respect to a user session is defined based on interaction events on desirable items across all interventions by the marketplace within a user journey. Given the per query ranking objective E[c ()|≺ ] (i.e. the expected number of desirable engagements from the SERP given the session context upon issuing the query), the success from the overall user session is shaped by the distribution P(|) that signifies the contribution of the interactions on the ranked SERP to the overall success of the user search session .
Eq∼ P(|≺ )[E[c (q)|≺ q]],
This distribution is usually referred to as the success attribution distribution, which is primarily
studied in the context of online advertising[
The expected reward of the marketplace from the presented ranking is then shaped by the distribution of the value across search contexts, which signifies the economic value of the user-interaction events in the sessions for the marketplace. The random variable vs captures the strategic notion of the value of the session for the marketplace, which corresponds to the value of the interactions events on the item(s) that satisfy the user’s intent. The expected reward of the marketplace can then be written as
Es,v[vsEq∼ P(|≺ )[E[c (q)|s≺ q]]], We can also consider an alternative formulation where we assume that session value distribution P() subsumes both the likelihood of the user session to lead to some reward for the search engine as well as the reward value attributed to the user session :
Es∼ P()[Eq∼ P(|≺ )[E[c (q)|s≺ q]]]. (6) For a value-aware search engine with marketplace revenue objective, economic value is realized only in the event of a transaction as the success event from a search session and the reward is proportional to the price of the sold item(s). For a search engine that aims to optimize for the volume of transactions, it is more suitable to adopt a value per acquisition notion of reward oblivious to the price of the sold items. For a search engine with strategic goal of maximizing (4) (5) user engagements for increased user retention and minimizing abandonment, it is more suitable to adopt a value per click notion of reward oblivious to the post click transaction events.
In the next section, we discuss empirical modeling techniques to build efective empirical reward estimates from observational data, which efectively frame the problem as a standard counterfactual empirical risk minimization with a value-aware context distribution.
We are interested in maximizing E[c ()|≺ ], the expected number of desirable engagements across all the slots on the page, with a suitably parameterized ranking policy . By hypothesizing an explanatory click model based on causal constructs that govern user browsing and engagement behaviors on search result pages, we can build efective likelihood models from which we can estimate the parameters of the ranking policy via maximum likelihood estimation using logged observational data. We instantiate this process with the simple widely adopted click models in information retrieval. Assuming a vanilla Sequential Browsing Model along with the standard Position-Dependent Examination Model, we can write the expected number of desirable engagements as
E[c ()|≺ ] = ∑︁ P(c − 1() = 1|≺ , ≺ ) =1 = ∑︁ P(c − 1() = 1|≺ , ) =1 = ∑︁ P(o − 1() = 1|)× P(R − 1() = 1|≺ ) =1 = ∑︁ P(o − 1() = 1)× P(R − 1() = 1|≺ ) =1 (7) (8) (9) (10) where the first line follows from sequential browsing assumption with c − 1() = 1 representing ≺ representing the slotted items prior to the click event on the item ranked at position and the item ranked in position . The second line follows from assuming that the user interaction event on a given slot is independent of the placed items in the previous slots. The third line follows from the standard examination-based click model that posits that a click event can be expressed as the intersection of a query specific examination event o − 1() = 1 and a presentation-independent contextual relevance event R − 1() = 1; and the last line follows from assuming a query-independent global rank discount function on the examination probabilities. The examination probabilities, a.k.a. propensity scores, are context-specific and can be estimated via explicit online interventions or from observational data. By considering a simple uni-variate model fit on the estimated propensities as a function of rank, one can build data-driven rankdiscount functions to estimate users’ examination efort. However, the standard approach is to adopt vanilla log-based context-oblivious rank discounts as generic estimates of the examination probability P^(o − 1() = 1), i.e.
ℓ() = .
Given an empirical estimate ^ of Bayes contextual relevance probabilities P(R − 1() = 1|≺ ), one can derive the standard discounted cumulative gain (DCG) estimate E^DCG[c ()|≺ ] of the expected reward per query for the policy (· ) as ℓ( ()) ^(). (11)
Upon building all the elements of the expected reward estimates, specifically the policydependent per query expected reward estimates, we can train the ranking policy by maximizing this empirical reward estimate, as elaborated in the next section.
Several techniques can be adopted to estimate the contribution of a ranked SERP and the corresponding observed or potential interactions on that page to the overall success of the user search session . A simple yet popular solution in the context of online advertising is to adopt an attribution distribution that assigns all the probability mass to the immediate query context preceding the post-click conversion event, which is referred to as Last Touch Attribution scheme. In contrast to this tight attribution scheme, one can assume a uniform distribution across all queries in the session in which the item with the attributed interaction event of interest was retrieved as a candidate item, oblivious to whether it was even impressed on the search result page. This approach is referred to as All Touch attribution scheme. Alternatively, One can assume a (Markovian) probabilistic graphical model on user’s touch points within a session journey and infer a probabilistic multi-touch attribution distribution
P^(|≺ ) from observational data. Similarly, one can adopt an attention based sequence modeling approach and infer the contribution weights for interaction events along the user journey with a conversion prediction model. Lose attribution schemes, like the all touch attribution scheme, signify the powerful idea of counterfactual training context generation for ranking policy learning, where in contrast to predictive perspectives, the policy can collect reward from a ranking context where the item of interest was not observed by the user. As discussed in the empirical results section, such attribution schemes are particularly efective for capturing the user behavior in search sessions with longer feedback loops, e.g. sessions with high purchase value user intent.
In order to highlight the importance of the session value distribution,
let us focus on a search engine with a value per acquisition objective. A straightforward
empirical session value distribution is adopt a uniform distribution on sessions that lead to a
transaction event. Such session value selection distribution leads to survivorship bias in training
context selection in that trafic segments where transaction events are rare, e.g. user sessions
with luxury intent, will be under-represented in training. A simple approach is to expand
the definition of success events and estimate the likelihood of session success with a
contentoblivious estimate based on the aggregate conversion likelihood of the richest engagement event
attributed to the element(s) engaged. This perspective on building mixture distributions based
on the richest post-click engagement event was shown to be efective in capturing potential
conversions from browse-heavy user journeys[
For a revenue-focused marketplace, as discussed in Section 2.2, the value of a search session is proportional to the price of the item that matches the user intent. In the presence of an observed success event in the logged data, the purchase price of the item to which the success event is attributed is the realization of the session value; otherwise, in the absence of an interaction event, the value of the session has to be estimated from the content of the asked intent in user queries, or a Canonical set of actual or synthesized items that match the user intent.
One of the main challenges in learning from observational data is the distribution shift between the training data collected from the logging policies and the inference data distribution. We therefore have to introduce another set of techniques, e.g. importance weighting distributions, to account for this mismatch between the (population) expected reward in (6) and the estimated expected reward from the estimated quantities in the previous sections; that is, Es∼ P^()[Eq∼ P^(|≺ )[E^[c (q)|s≺ q]]]. (12) An important source of distribution shift in observational search activity data is the the selection bias due to presentation of the items on the page and the sequential browsing of the users, implying that we only observe relevancy of the items to the user only in the event of an explicit user engagement and it is more likely to observe engagements on SERPs from higher ranking slots.
A key technique to account for this efect is to define a suitable notion of propensity, which is
developed in the context of studying the efect of a treatment(an intervention) on a population
by taking into account attributes of the treatment unit in the way the treatment is assigned. In
the context of ranking, the treatment is defined in correspondence to the examination of a slotted
item by the user, but the key diference with the standard applications of this concept is that
the examination variable is not fully observable. An alternative approach based on potential
outcome modeling, similar to actor-critic networks in the context of ofline reinforcement
learning, is proposed in [
Having discussed an array of importance weighting schemes to build empirical expected reward estimates, it is essential to develop variance reduction techniques to control the generalization behavior of expected reward estimators. For brevity of presentation, we briefly discuss the various reduction techniques adopted and ignore developing generalization bounds on the bias and variance of the estimation error of the proposed empirical reward estimation techniques.
Clipping and truncated importance sampling techniques[
In building empirical session value distributions for a revenue focused marketplace reward, relying on the purchase price of the success items leads to a very high variance estimator, particularly in the presence of high heterogeneity in price intent across user trajectories. Instead, we can use a stratification technique by bucketing user sessions based on value buckets defined according to the empirical revenue distribution. Specifically, we can build a session value distribution based on the empirical revenue share of the bucket corresponding to the price of the purchased item.
One of the the primary challenges of counterfactual learning to rank from logged search activity
data is that the relevancy of the items is observed only in the event of explicit user engagements.
A popular idea in the context of contextual bandits and recommendation systems to circumvent
the challenges in this partial information setting is to use predictive models for reward estimates
as potential outcome models in conjunction with inverse propensity weighting[
Efective stratification is a key technique in the context of importance weighting estimators, e.g. the context value binning idea discussed in sub-section 3.4.1 or training context stratification based on characteristics of logged training contexts[34]. We adopt Self-Normalizing propensity based estimators, recently analyzed in [35], where we use engagement ranks as yet another stratification dimension in our proposed estimators. Yet another standard variance reduction technique that we adopt to control the contribution of the search sessions with many success events in the observational data is to adopt normalization techniques; e.g. the standard Ideal cumulative gain normalization for the per query loss. We note that under this cumulative reward normalization technique, per item propensity weights should be reformulated as context weights.
Having equipped our empirical reward estimates with variance reduction techniques, from this point on, we can assume that the efect of all importance weighting schemes discussed so far are reflected in importance weights ^,.
We consider deterministic policies parameterized by a scoring function , such that = argSort(f), oblivious to the representation of the items and the ranking context. An appealing approach, particularly in the context of online advertising and sponsored search, is to directly estimate the Bayes contextual relevance probabilities P(R = 1|≺ ), or equivalently the counterfactual probability of click had the item been examined P(c = 1|≺ , do(o = 1)) via a standard supervised predictive models, i.e.
∈ ∑︁ ^, ∑︁ D( ()||^), , (13) where ^, is the empirical importance weights based on discussion in section 3.2, is a distance measure, e.g. cross entropy, between the predicted distributions () and the properly debiased empirical label distribution^. For estimating counterfactual probability of click that is Contextually Well-Calibrated and Discriminative for ranking, we need very complex models with rich feature representations, with careful data stratification and selection bias correction. Since absolute merit estimation is usually a harder problem than diference in merit estimation, we resort to alternative techniques for empirical expected reward optimization.
The standard alternative approach is to adopt the LambdaLoss framework[36] and optimize a pairwise upper bound on the (list-wise) empirical estimates for the expected number of engagements, ℓ( , ^), to circumvent the challenges of dealing with highly non-smooth rankdependent policy function, which can be written as ℓ( , ^) =
∑︁ ,′∈ ΔE^ (swap^(, ′)) ( () − (′)), (14) where E^ (swap^(, ′)) is the diference in the estimated expected number of engagements had the ranked slots of the item pairs (, ′) been swapped and (· ) is some inverse link function, e.g. softMax. The approximate surrogate objective, suitably weighted with the empirical reward estimates ^,, expressed as ∑︁ ^,ℓ( , ^) , (15) can then be optimized using iterative optimization techniques, like Expectation-maximization; that is given an estimate () at iteration , in order to build (+1) from the gradient updates from the objective function, the diference in estimated objective E^ () from the swap operation is computed based on ranking order produced by ().
In Section 3, we discussed essential elements of building empirical expected reward estimates for training efective search ranking policies. Since conducting thorough ablation studies for characterizing the efect of each element in building empirical expected reward estimates is not possible given the space constraints, we focus primarily on the rather under-explored element in the literature, which is the efect of context value distribution discussed in section 3.2 in shaping the properties and the generalization performance of the ranking policy.
We focus on a product search ranking scenario in a major E-commerce platform and evaluate candidate policies via online randomized control experiments, as well as rigorous counterfactual evaluations on user session data collected from the online trafic. Since all experiment are performed on proprietary data, we only report lifts compared to a simple clearly-specified baseline, with a focus on the relevant choices for controlling the estimation error with respect to the research question of interest, oblivious to the optimization framework, the feature representations, and the hypothesis class. Specifically, we only discuss the choice of the ranking objective and the relevant importance sampling and attribution techniques for building our estimators of interest, without discussing the details of the models.
Since the main goal of the proposed decision making framework is to build search ranking policies that generalize with respect to a given notion of marketplace expected reward, we primarily evaluate the performance of the candidate policies in online randomized controlled experiments. Specifically, we adopt an experiment design and primary success metric defined with respect to lifts in cumulative reward in treated user sessions. This cumulative reward driven design is in contrast to the standard experiment design practices for incremental ranking changes, where the primary success metric is set to be the standard (immediate) ranking eficiency metrics that measure concentration of success events in Top slots, through simple attribution and aggregation schemes across search result pages. In fact, top slot engagement concentration metrics, e.g. per query DCG with respect to SERP interactions aggregated uniformly across all queries, which are usually tightly correlated with the marketplace reward, should only be treated as secondary metrics in the presence of a measurement of cumulative reward in online experiments. We do recognize, however, that DCG-type metrics are particularly crucial for counterfactual of-policy evaluations, as approximations to the per query expected reward using logged data, because all we can do is to measure concentration of logged success events in top slots upon the shufling action of the new target policy.
We establish the fundamental trade-ofs between ranking policies trained on diferent empirical expected reward objectives primary based on session level cumulative reward metrics, including Number of Engagements, Number of Purchases, and Revenue, as measured in online AB tests. For metrics that attribute the observed efect to search events, we use a simple attribution schemes based on the immediate Search Result Page that precedes the user event of interest.
We adopt the standard supervised counterfactual training and evaluation framework based on logged search activity data collected from the online trafic of a major E-commerce platform. We are oblivious to the logging policy and collect datasets with importance sampling and reward attribution semantics based on the corresponding notions of expected reward of interest. Specifically, given a target notion of expected reward, the context value distribution remains the same for training and evaluation datasets. For candidate item selection per SERP, however, we sample three negative samples at random from impressed unengaged items within each training context, but keep all the candidate items to be re-ranked by the candidate ranker for the evaluation datasets.
For all empirical expected reward metrics, we use the same, suitably debiased and normalized, DCG approximation for the per query expected reward according to (11). Unless explicitly stated otherwise, we use the following vanilla empirical context value distribution for building expected reward estimates as training objectives and the counterfactual metrics.
Expected number of engagements E^[C]: The session value distribution P^() is a uniform distribution across logged sessions with at least one click event. We consider a simple last touch attribution scheme P^ (|≺ ) for the distribution of reward among queries within the session.
Expected number of purchases E^[P]: The session value distribution P^ () is uniform across logged sessions with at least one purchase event. We use a simple multi-touch attribution scheme P^ (|≺ ) with uniform distribution across all queries in the converting session, where the purchased item appeared as a candidate.
Expected revenue E^[Rev]: The session value distribution P^ℛ() is defined on the sessions with a transaction event according to the empirical revenue share of the bucket corresponding to the price of the purchased item. The same multi-touch attribution from above is adopted for this reward estimate as well.
To best highlight the heterogeneity of user behavior with respect to the underlying shopping intent and the associated fundamental trade-ofs between diferent notions of marketplace expected reward, we also stratify our evaluations across trafic segments defined based on purchase price intent of the users, as realized in the price of the purchased item. The price intent bins are defined in such a way so that the empirical revenue distribution is roughly uniform across value buckets.
The primary insight that we would like to highlight in our evaluations is the heterogeneity of users’ browsing and shopping intents, as reflected in diferent notions of marketplace reward from user sessions. These observations signify the crucial importance of the choice of the empirical session value distribution in shaping the properties and the generalization behavior of the search ranking policy.
We do this by contrasting the performance of ranking policies trained on expected reward estimates corresponding to extreme choices of the empirical session value distribution. Specifically, we compare a policy , corresponding to a scoring function , trained on a simple engagement-driven expected reward estimate based on the session value distribution P^() against a policy , corresponding to a scoring function , trained on a simple acquisitionfocused expected reward estimate based on the session value distribution P^ (). We observe meaningfully diferent performance trade-ofs between these extreme policies with respect to the primary notions of marketplace reward in an online randomized controlled experiment. Table 1 summarizes the key observations on the average efect size Δ ( , ) between engagement focused policy and acquisition focused policy , with respect to diferent cumulative metrics , over the global session trafic . The main takeaway from these observations is that the engagement focused policy , on the one hand, drives significantly higher share of search sessions with at least one click(> +3%), and on the other hand, leads to a significant drop in the share of search sessions with at least one purchase( < − 2%).
It is interesting to note, however, that this drop is largely due to a significant loss in the number of bought items in search sessions with lower price intent, which usually take less exploration and browsing to identify and pinpoint the desirable item to purchase. Since the engagement-driven policy is more efective in driving success events with higher economic value in sessions that require more browsing efort, it can compensate for the revenue loss due to lower purchases in lower price intent segments, leading to an overall neutral efect size in total revenue.
In order to explore the fundamental trade-ofs, highlighted in our online experiment, between
diferent notions of marketplace expected reward across heterogeneous price intents in more
depth, we build simple hybrid policies corresponding to a mixture of the engagement-based
and acquisition based objectives. Specifically, we build a simple policy via a simple convex
combination of the extreme polices
= argSort((1 − ) + ),
(16)
where refers to the balanced ranking policy obtained via a linear combination of the scoring
functions of the engagement focused policy, and acquisition focused policy , for some
∈ [
Due to the scarcity of online experimentation trafic, we only conduct counterfactual ofpolicy evaluations for these parameterized policies. While our counterfactual estimates are largely aligned, at least directionally, with the measured efect sizes in online experiments, we point out that all counterfactual of-policy evaluations are fundamentally limited having access only to snapshots of the users’ behavior in the logged sessions. In particular, if the logging policy is substantially diferent from the target policy to be evaluated, the ofline evaluation metrics could be very biased.
Figure 1 highlights the essential trade ofs between diferent expected reward estimates from logged data ^, with the acquisition focused policy as the baseline Δ^ ( , ).
(17) Biasing the training objective heavily on one extreme, leads to significant drops in the estimated reward corresponding to the other extreme. As the contribution of the engagement-focused policy increases, by increasing > 0, we estimate higher expected number of engagements, with a saturation point of diminishing return, after which a sharp drop in the expected number of purchases is observed. Interestingly the estimated expected revenue is convex as a function of , which we will discuss in our subsequent research focused on value-aware objectives.
Next, we explore the observed trade-ofs in the global analysis above across heterogeneous segments ^ corresponding to diferent price intent segments, where the attribution of a session to a value bucket is done with respect to the price of the purchased items. Figure 2 and 3 show the lift in estimated expected number of engagements E^[C] and estimated expected number of engagements E^[P], respectively, for the hybrid policy across value segments ^ with the acquisition focused policy as the baseline.
We clearly see that the extreme acquisition focused policy performs poorly in terms of the expected number of engagements, across all segments, with particularly larger efects sizes in high value price intents that require more exploration. We also observe that, as increases, the lift in expected clicks Δ^ (E^ [C], E^ [C]) increases, with a saturation point in lower price segments(which is in fact an inflation point for low price intent segments). On the contrary, biasing the policy towards the engagement-focused policy, by setting close to 1, leads to a meaningful drop in the expected number of purchases, Δ^ (E^ [P], E^ [P]), particularly in low value price segments, which constitute a high proportion of the overall number of purchases. An interesting observation, however, is that focusing more on an engagement-based objective is helpful for driving even higher expected number of purchases in higher price segments. We leave deeper dives on the observed trade ofs for future work.
In order to highlight the significance of the reward attribution scheme within a user session, we contrast the generalization performance of policies trained with respect to extreme choices of the query contribution distribution P^ (|≺ ). Specifically, we contrast the performance of a policy trained on a session value distribution with a tight attribution of success events to search events, similar to the last touch scheme discussed earlier, to a policy trained with respect to a loose multi-touch attribution of success events to search events, similar to the all touch scheme discussed earlier. While the overall cumulative rewards do not show sizable performance trade-ofs between the two extreme policies, we highlight substantially diferent efect sizes across diferent purchase price intents. Table 2 clearly demonstrates that a loose attribution scheme for the empirical query contribution distribution helps with a significant improved generalization in higher price intent sessions, which tend to be more exploratory and involve multiple ranking intervention touch points.
Finally, in order to highlight the significance of incorporating the purchase price in the session value distribution for a revenue focused marketplace reward, we highlight the results from an online AB test on a simple value-aware policy v in contrast to a value oblivious acquisition driven policy v. The primary diference between the two policies is the empirical session value distribution P^() in the corresponding expected reward estimate for the training objective, which depends also on the price of the sold item in the case of v, and all the other importance weighting distributions and per query reward estimates are the same. In Table 3, we clearly see a significant shift in the distribution of the accumulated revenue across price intent segments, which signifies the importance of taking into account the sparsity of purchase events from higher price intent sessions to avoid the selection bias incurred by the value oblivious policy, which is biased towards lower price intent segments, where sessions with purchase events are abundant.
We established an explicit connection between the training objective for the search ranking policy and the key performance metrics of a two-sided commerce marketplace by building efective empirical estimates of the marketplace reward from observation data. Specifically, we highlighted the significance of the search context value distribution in building efective empirical estimates of the marketplace expected reward to inform the training and evaluation of the search ranking policy. We showcased empirical results from online randomized controlled experiments and counterfactual evaluations in a major e-commerce platform demonstrating the fundamental trade-ofs governed by extreme choices of the context value distribution. [31] X. Wang, R. Zhang, Y. Sun, J. Qi, Doubly robust joint learning for recommendation on data missing not at random, in: International Conference on Machine Learning, PMLR, 2019, pp. 6638–6647. [32] Y. Saito, Doubly robust estimator for ranking metrics with post-click conversions, in:
Proceedings of the 14th ACM Conference on Recommender Systems, 2020, pp. 92–100. [33] A. D. Tucker, T. Joachims, Variance-minimizing augmentation logging for counterfactual evaluation in contextual bandits, in: Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining, 2023, pp. 967–975. [34] E. Ebrahimzadeh, A. Cozzi, A. Bagherjeiran, Intent-aware propensity estimation via click pattern stratification, in: Companion Proceedings of the ACM Web Conference 2023, 2023, pp. 751–755. [35] B. London, A. Buchholz, G. Di Benedetto, J. M. Lichtenberg, Y. Stein, T. Joachims, Selfnormalized of-policy estimators for ranking (2023). [36] X. Wang, C. Li, N. Golbandi, M. Bendersky, M. Najork, The lambdaloss framework for ranking metric optimization, in: Proceedings of the 27th ACM international conference on information and knowledge management, 2018, pp. 1313–1322.