Learning To Rank (LTR) from implicit user feedback is the predominant approach in large scale information retrieval systems for e-commerce. The standard LTR approaches that account for search intent in training ranking models focus primarily either on enriching the feature representation of the model by estimates of the search intent, or adopting an auxiliary intent prediction task in a multi-task learning setting, or debiasing the user feedback using intent-aware propensity models. In this work, we propose intent-aware LTR schemes by stratifying the training data with respect to intent groups identified by empirical labels that correspond to distinct item desirability distributions. Specifically, through importance sampling on training queries based on the richest engagement event attributed to the engaged item on the SRP, we train rankers that align the LTR objective with the corresponding item desirability distributions for the browse and purchase intents. In order to demonstrate the eficacy of the proposed training data stratification technique in generalizing across query segments with diferent underlying intents, we evaluate the rankers trained with diferent importance sampling weights on the trafic segment identified by the Browse search experience in a major e-commerce platform. In particular, we show that the ranker trained only on queries with post click conversion signals is significantly outperformed by a ranker that relies also on non-converting queries in training. We further demonstrate a higher variance in ranking eficiency in the trafic segment identified by the browse intent, due to the more severe distribution shift.
CEUR ceur-ws.org
In e-commerce, a substantial portion of customer purchases are the result of searching and exploring the item inventory of the platform. This behavior is best described as exploratory search, characterized as a combination of exploratory browsing and focused searching [1, 2]. While focused search has been a primary research focus for the information retrieval community, user studies show that roughly 40% to 65% of users’ goals are informational [3], implying that searchers are seeking novel items. It is therefore essential for industry scale information retrieval systems to adjust to the user need depending on the specific search scenario [ 4].
In this work, we study the efect of adjusting the label distribution in the Learning To Rank (LTR) training objective to the underlying intent in exploratory and focused search scenarios in an e-commerce setting. Specifically, we evaluate the generalization performance of the rankers CEUR Workshop Proceedings trained on varying label distributions across trafic segments based on the search experience in an e-commerce search engine. Previous work has accounted for capturing the user intent in LTR mainly by enriching the feature representations of the models. However, there are multiple advantages to dealing with user intent in a feature-agnostic setting, primarily due to the hardness of the intent prediction task in the absence of additional contextual information about the user’s preferences, particularly in the early stages of the search journey.
We take an alternative intent-aware LTR approach, by stratifying the training data with respect to the label distribution that estimates the underlying item desirability distribution for diferent intent groups. Intent estimation policy can be a complex function of the user’s query as well as the prior interactions in the search context, therefore, in our evaluations we identify the browse or focused search intent based on the type of search interaction experience. Specifically, we identify the browse intent based on the user trafic who explicitly adopt the Browse experience and explore the retrieved items in product category groups, in contrast to expressing a shopping intent by issuing a keyword query. We argue that due to the measurable diferences in the post click conversion rates across the two search experiences, an LTR objective with a label distribution based on click engagements is more suitable for users in the exploration phase of their search journey; whereas an objective with an empirical label distribution based on purchases is more suitable for users with a focused search intent. We motivate the relationship between user engagement events (i.e., clicks and purchases) and underlying user intent (i.e., browse and focused search) by identifying distinct user behavior patterns in each segment, as surfaced in logged search data from a major e-commerce platform.
In our search trafic analysis, we observe that users who engage with items in Browse Result Pages (BRP) when browsing product categories are less likely to convert on the clicked items compared to the users who issue a keyword query and engage with the items on Search Result Pages (SRP). Therefore, we argue that browse queries based on a user-selected category drive more engagements for further exploration, and the ranking objective in this segment should be driven primarily by engagement eficiency. On the other hand, for focused queries with an underlying purchase intent, the ranking objective should be driven mostly by the conversion eficiency. Our study makes an initial step towards characterizing the fundamental sale/engagement eficiency trade-of in diferent trafic segments. This is achieved by training ranking policies on optimization objectives tailored to the corresponding intent class and serving requests from each trafic segment with the designated ranker for that class. Our work makes the following contributions: • We propose training data stratification techniques for intent-aware LTR with intent strata identified by the empirical target labels that estimate distinct item desirability distributions in diferent search interaction scenarios. • We identify distinct engagement patterns and measurable diference in post click conversion rates in e-commerce search interaction scenarios corresponding to browsing product category groups (i.e., exploration) versus expressing a purchase-driven intent (i.e., focused search). • We evaluate the efectiveness of our intent-aware LTR methodology in an e-commerce ranking scenario by highlighting the variance in ranking eficiency in the intent segment identified by the browse experience, as the training label distribution of the ranking policy
There is a remarkable body of work that focus on query intent prediction in the context of e-commerce [5, 6, 7, 8, 9, 10, 11]. While exploration has been identified as one of the major user intents in e-commerce settings [12, 13], incorporating exploratory search intent in the e-commerce ranking objective remains an open problem [14].
Once user’s browsing/exploration intent is estimated, much less work has focused on how a search system can efectively support diferent user behavior. For instance, Rahman et al. [15] propose a conceptual recommendation framework to support exploratory search in e-commerce, by combining ranked results from a text based search engine with a recommender system based on user interactions. Ebrahimzadeh et al. [16] propose debiased estimators with intent-aware propensity estimates based on the number of clicks in a search context. Medlar et al. [17] classify users’ search tasks in fine-grained categories, which fall within a behavioral spectrum between focused search and exploratory browsing. Rowley [18] focus on matching a browsing user to the correct product category, while the question of whether the ranking objective should vary depending on the user intent is not addressed. In our work, we proposes to train rankers for users with varying intent, where the preference for diferent user engagements is incorporated in the objective function.
Another line of work focuses on query understanding. For example, Wang et al. [19] parse a query into linguistic concepts that related to specific instances in a knowledge base. Dai and Callan [20] use contextual language models (such as Kenton and Toutanova [21]) in information retrieval to achieve a better understanding of the query context. Another related area of work focuses on a better semantic understanding of queries [22]. Applying these works to E-Commerce is not trivial and needs to address many challenges, such as vocabulary gap and data sparsity [14]. Furthermore, this particular research area does not clearly distinguish users by their intent and these methods are not applicable for category-based queries without any keywords. In contrast, our browse-specific rankers are optimized to support users with browse intent and can perform on keyword-less queries.
We consider a generic supervised LTR setting where we observe user feedback in the form of clicks on the SRP, as well as the subsequent post click transaction event attributed to the clicked items. A search ∈ is characterized by the query ∈ made by some user ∈ , the SRP that is an ordered collection of items (i.e., documents) ∈ retrieved by the engine with respect to as well as the user’s click and post click purchase events, denoted by ( ) ∈ {0, 1} and ( ) ∈ {0, 1} , respectively. It is standard to assume that ground truth relevance labels are produced by some oracle according to an underlying item desirability distribution ℙr |s( |) in the search context . We learn a ranking policy by minimizing the statistical risk, corresponding ( ) = [ℒ ( ( ), ( ))], where ℒ ( (
), ( )) = ℓ( ( )) ( ) is the DCG-based ranking eficiency loss with respect to the ideal ranking defined with respect to ( ), suitably discounted by some function of the rank ℓ( ( )) = [−( ())] ∈
attributed to the items via the scoring function .
Since the underlying joint distribution of the search contexts and relevance labels is not known to the learner, the standard approach is to build an empirical risk estimate based on a sampled set of search contexts with suitably defined empirical labels (̂ ) to approximate the ground truth label distribution ℙr |s( |) such that to a search eficiency loss function, which is meant to approximate the expected number of user behavior events. Specifically, the statistical risk for a ranking function (; ) , which produces a score for each individual document ∈ given the search context , is defined as (1) (2) (3) (4) distribution ℙs()ℙ r |s( |) . In this work, we are primarily interested in estimation error in our Empirical Risk Minimization (ERM) setup, oblivious to feature representation, hypothesis class, and training optimization scheme, which have to do with optimization and approximation error in the ERM setup. ∈ℐ | | ∑ ∈
ℒ ( ( ), ̂( )), where is the logged training data corresponding to the intent segment and ℙ̂ is some stratiifcation distribution across the intent groups. We focus on a simple binary intent segmentation based on the underlying browse or purchase behavior of the user. In this case, our estimate users with the browse intent it is the empirical click engagement distributions ( ̂ ( ) of the underlying intent-based item desirability distribution ℙr |s,i( |, ) is two-fold: for ), whereas for users with focused search intent it is estimated the empirical distribution based on purchase signals (
). In fact, to avoid the complexity in building our empirical estimate, let us assume that we only have a single clicked item within all the training search contexts. Thus, since we
We stratify the LTR training data based on intent groups ∈ ℐ identified by distinct item desirability distributions
ℙr |s,i( |, ) in diferent search interaction scenarios. We can therefore build stratified empirical risk estimates by adjusting the empirical labels ̂ ( ) so that they approximate the underlying desirability distribution in the corresponding intent segment; that is observe post-click purchases only for clicked items in converting queries, our stratified estimate can be written as
1 ∑ ℒ ( ( ), (̂ )), | | ∈ (5) where (̂ ))is the suitably debiased click label(e.g. based on inverse propensity weighting) and the training search contexts are sampled from the search context with post click purchases with an importance sampling parameter and from the search context with no post-click purchases with importance sampling parameter 1 − ; that is = { } ∪ { } (1−) .
Using standard arguments, we can characterize the bias and variance of the proposed stratified empirical risk estimator with respect to the Bayes statistical risk corresponding to the underlying ground truth item desirability distribution in the segment. In fact, the generalization power of the proposed stratified empirical risk estimate can be adjusted to the corresponding query segment by an appropriate choice of the stratification parameter .
While we can clearly highlight the generalization power of the proposed stratified risk estimation technique by focusing on the same strata on the evaluation dataset, we instead rely on evaluation segments that can explicitly be identified at the online serving time. Specifically, we identify the user’s underlying search intent based on the the explicit search engine interaction experience that the user opts into: (A) the user expresses a browse intent by clicking on a category on the homepage, e.g. “Automotive”, and explores the results on a Browse Result Page (BRP); (B) the user expresses a focused search intent by issuing a query by entering keywords in the search bar, e.g. “Audi R8 headlights”, and explores the results on a Search Result Page (SRP).
To motivate the relationship between user intent, as characterized by BRPs and SRPs, with the corresponding intent strata at the training time, we perform an analysis of post click conversion rates (CVR) on user search logs. In fact, we report a measurably lower CVR in BRPs compared to SRPs, with a diference Δ( , ) > 50% . This is a strong indicator that BRP engagements are more likely to be followed by subsequent clicks via engaging in other item recommendation modules or issuing a keyword search query rather than committing to a purchase. This is a key observation as it motivates the connections between the intent classes and the corresponding engagement/conversion driven objectives. To further understand the interaction patterns across query intent segments, we perform a second study to compare click propensities at diferent ranks across browse and focused search query segments (Figure 1). We observe a smoother rank-decay in click propensities in browse queries and more engagement likelihood in lower ranks. This observation confirms the hypothesis that user behavior patterns vary by intent, as users with a browse intent are more likely to explore lower ranks, while users with a purchase intent are more likely to engage with top ranks.
The inherent diferences in the engagement patterns between browse and focused search queries indicate that: (1) Browse queries drive engagements for further exploration, thus the ranking objective in this segment should be driven by the engagement eficiency; and (2) In 20 Rank contrast, focused search queries are more likely to be specific and purchase driven, thus the ranking objective should primarily be driven by eficiency of conversion events.
In this section we design our experiments to measure the empirical performance of the rankers trained on segment-specific training objectives evaluated on query segments corresponding to BRP and SRP search experiences. Given an evaluation segment, we are specifically interested in the relative performance of a ranker that is trained only on queries with post click purchases attributed to the engagement events versus one that can rely also on the non-converting search contexts. We train ranking models , parameterized by the importance sampling parameter , on stratified training datasets , where proportion of the training queries are sampled from queries with a post click purchase event and the remaining 1 − proportion are sampled from the non-converting query set. By training diferent rankers through varying the stratification parameter , and evaluating them over diferent user intent segments, we establish a conversion/engagement trade-of as it relates to the underlying item desirability label distribution of the ranker.
For a logged search context in the evaluation dataset , suppose that the richest engagement event attributed to the engaged item on the SRP is ; which is essentially a purchase event if there is a post-click conversion attributed to that engaged item, otherwise it is just the click event on the SRP. The ranking eficiency of the ranker is measured as the average Mean Reciprocal Rank (MRR) with respect to () , the rank attributed to the engaged item by in search contexts ∈ ; that is ( , ) = | 1 | ∈∑ 1 () .
(6)
Training Data. We build distinct datasets by stratifying the queries based on the richest engagement event observed in the search context ( ) and ( ); Specifically, the proportion of queries with a purchase event in the search context is controlled by importance sampling parameter . Note that during training, models are exposed to queries from all intent segments (both in terms of the training segments based on the underlying item desirability distribution and the evaluation segments based on the BRP/SRP search experience) to take advantage of synergies among all user experiences. To control for the size of the training dataset, for each query we sample three unengaged items from the candidate set at random, as negative examples. For training targets (̂ ), we use debiased labels using the propensities characterized in the previous section. All training datasets contain roughly 1M queries.
Test Data. The test datasets are based on a random sample of queries for which there is at least one click event observed in the search context, suitably separated from training sets to avoid data leakage. The two test datasets are built via stratification based on BRP and SRP intent segments and each contain around 10K to guarantee meaningfully narrow confidence intervals. For the test data, we keep all the candidate items to be re-ranked by the candidate ranker.
Note that datasets are a random sample of e-commerce search trafic in the months of April to June 2022.
Table 1 ofers a fundamental observation on the generalization power of the proposed stratified empirical risk estimate. Essentially, we can achieve a bias/variance trade-of by an appropriate choice of the stratification parameter , which can in fact be adjusted based on the evaluation segment that ranker is served on. As such, the unbiased estimator of the purchase desirability intent =1 , fails to generalize even on the trafic population with purchase intent, primarily due to the selection bias in training search context qualification. Essentially, =1 relies only on converting queries to estimate the more diverse trafic distribution. This is particularly more pronounced in the BRP segment, because as pointed out in the search trafic analysis, most users do not commit to a purchase upon engaging with an item in a BRP. In fact, by relying also on non-converting queries in training the rankers via stratified empirical risk estimates, we can improve the estimation accuracy of the target distribution for engagement eficiency. Performance diference between purchase-driven model =1 and the baseline model with optimal ∗ . All reported lifts are statistically significant and exceed the threshold for a meaningful efect size
Lift w.r.t Evaluation Data Δ ( Δ ( ( =1 , ⋅), ( =1 , ⋅), ( ∗ , ⋅)) ( ∗ , ⋅))
-8.3% -2.1%
-5.7% -0.7%
On the other hand, if we do not over-sample converting queries and only rely on a randomized sample from the trafic population
, both the conversion and engagement rates will be meaningfully impacted, even in population segments with primarily a browse intent, because search contexts with post click purchase usually ofer richer information for training.
Next, by comparing the efect size of the relative performance of the rankers across query segments, we highlight a meaningfully higher variability in engagement eficiency in the browse segment compared to the focused query segment, highlighting strong evidence towards the choice of a click engagement driven objective for the browse segment. In fact, we report a significantly higher variance in the search eficiency metric to the SRP segments as the ranking model varies; that is
in the BRP segment compared Δ (Var[ ( , )],Var[ ( , )]) > 10% (7) The large variance diference is due to a significant performance drop for model close to 1, as they fail to generalize to the population distribution with an underlying browse intent. In fact, the best performing setting for can be tuned optimally for the specific segments that ranking policy serves. We leave the exploration of other query segmentation strategies, beyond BRP and SRPs, as well as the best practice for fine tuning the stratification parameter to when is We presented a training data stratification technique based on intent groups characterized by distinct underlying item desirability distributions. To verify the generalization performance of the proposed scheme, we performed empirical evaluations on search trafic from a major e-commerce platform. We showed the poor generalization of the ranker trained only on queries with rich post click engagement events, and demonstrated that as we change the training target distribution of the ranker, the variance in search eficiency is measurably higher in the browse segment compared to the trafic segment identified by keyword queries. These observations confirm the hypothesis on the fundamental benefits of aligning the ranking objective with the future work.
user’s search intent.
2007. [1] M. J. Bates, What is browsing-really? a model drawing from behavioural science research, [2] G. Marchionini, Exploratory search: from finding to understanding, Communications of the ACM 49 (2006) 41–46. [3] J. Teevan, C. Alvarado, M. S. Ackerman, D. R. Karger, The perfect search engine is not enough: a study of orienteering behavior in directed search, in: Proceedings of the SIGCHI conference on Human factors in computing systems, 2004, pp. 415–422. [4] T. Ruotsalo, J. Peltonen, M. J. Eugster, D. Głowacka, P. Floréen, P. Myllymäki, G. Jacucci, S. Kaski, Interactive intent modeling for exploratory search, ACM Transactions on Information Systems (TOIS) 36 (2018) 1–46. [5] L. Guo, L. Hua, R. Jia, B. Zhao, X. Wang, B. Cui, Buying or browsing?: Predicting realtime purchasing intent using attention-based deep network with multiple behavior, in: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2019, pp. 1984–1992. [6] D. Cirqueira, M. Hofer, D. Nedbal, M. Helfert, M. Bezbradica, Customer purchase behavior prediction in e-commerce: A conceptual framework and research agenda, in: New Frontiers in Mining Complex Patterns: 8th International Workshop, NFMCP 2019, Held in Conjunction with ECML-PKDD 2019, Würzburg, Germany, September 16, 2019, Revised Selected Papers, Springer, 2020, pp. 119–136. [7] B. Zheng, B. Liu, A scalable purchase intention prediction system using extreme gradient boosting machines with browsing content entropy, in: 2018 IEEE International Conference on Consumer Electronics (ICCE), IEEE, 2018, pp. 1–4. [8] P. Minjing, L. Xinglin, L. Ximing, Z. Mingliang, Z. Xianyong, D. Xiangming, W. Mingfen, Recognizing intentions of e-commerce consumers based on ant colony optimization simulation, Journal of Intelligent & Fuzzy Systems 33 (2017) 2687–2697. [9] D. Schellong, J. Kemper, M. Brettel, Generating consumer insights from big data clickstream information and the link with transaction-related shopping behavior (2017) 24. URL: http://aisel.aisnet.org/ecis2017_rp/24. [10] D. Schellong, J. Kemper, M. Brettel, Clickstream data as a source to uncover con-sumer shopping types in a large-scale online setting, in: 24th European Conference on Information Systems, ECIS 2016, Istanbul, Turkey, June 12-15, 2016, 2016, p. Research Paper 1.
URL: http://aisel.aisnet.org/ecis2016_rp/1. [11] M. Zavali, E. Lacka, J. De Smedt, Shopping hard or hardly shopping: Revealing consumer segments using clickstream data, IEEE Transactions on Engineering Management (2021). [12] N. Su, J. He, Y. Liu, M. Zhang, S. Ma, User intent, behaviour, and perceived satisfaction in product search, in: Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, 2018, pp. 547–555. [13] P. Sondhi, M. Sharma, P. Kolari, C. Zhai, A taxonomy of queries for e-commerce search, in: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, 2018, pp. 1245–1248. [14] M. Tsagkias, T. H. King, S. Kallumadi, V. Murdock, M. de Rijke, Challenges and research opportunities in ecommerce search and recommendations, in: ACM SIGIR Forum, volume 54, ACM New York, NY, USA, 2021, pp. 1–23. [15] M. M. Rahman, Z. M. Zaki, N. H. B. M. Alwi, M. M. Islam, A conceptual model for the e-commerce application recommendation framework using exploratory search, Solid State Phenomena (2020). [16] E. Ebrahimzadeh, A. Cozzi, A. Bagherjeiran, Intent-aware propensity estimation via click pattern stratification, in: Companion Proceedings of the ACM Web Conference 2023, 2023, pp. 751–755. [17] A. Medlar, J. Pyykkö, D. Glowacka, Towards fine-grained adaptation of exploration/exploitation in information retrieval, in: Proceedings of the 22nd International Conference on Intelligent User Interfaces, 2017, pp. 623–627. [18] J. Rowley, Product search in e-shopping: a review and research propositions, Journal of consumer marketing (2000). [19] Z. Wang, K. Zhao, H. Wang, X. Meng, J.-R. Wen, Query understanding through knowledgebased conceptualization, in: Twenty-Fourth International Joint Conference on Artificial Intelligence, 2015. [20] Z. Dai, J. Callan, Deeper text understanding for ir with contextual neural language modeling, in: Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, 2019, pp. 985–988. [21] J. D. M.-W. C. Kenton, L. K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, in: Proceedings of NAACL-HLT, 2019, pp. 4171–4186. [22] R. Baeza-Yates, Semantic query understanding, in: Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2017, pp. 1357–1357.