With the rapid growth of online food delivery, ofering personalized recommendations is crucial in reducing the choice paradox for users. This paper solves the problem of deciding the best cross-selling recommendations algorithm by introducing ) that caters to the business objectives with the help of targeted metrics. It further addresses the problem of creating the primary context (required for giving recommendations). A comparison of ofline metrics with online metrics (using three A/B tests on a major food delivery platform) shows that our approach is able to predict the direction-wise (positive/negative) online performance based on the percentage improvement in the ofline metrics. The winning algorithm is already deployed on the platform.
CEUR ceur-ws.org
The global revenue from the online food delivery market is expected to rise from $300 billion in 2022 to $450 billion by 2023 1. Enhancing user experience by personalizing recommendations has been one of the key reasons ([1], [2]). But, evaluating recommender systems is a challenging task. Broadly, there are two ways of evaluation: Online (A/B tests) & Ofline. One can often ofline evaluate multiple ideas in a short amount of time just by changing algorithms, feature sets, hyper-parameters, etc but what we can test online is always limited by time, the number of users on the platform, and the risk of negatively impacting the user experience which could further impact the business metrics. Thus, ofline evaluation can help alleviate this by giving some indication of which algorithms are likely to perform well online.
This paper focuses on creating an ofline evaluation line food delivery platform operating in various markets around the world. The framework is being leveraged to ofline evaluate cross-sell recommender systems on the cart page for diferent business KPIs. Contrary to previous studies ([3], [4], [5]), our logs only have partial information (primary context is absent) stored due to design issues. Hence, the contributions are three-fold and are summarised as follows: nEvelop-O LGOBE 0000-0002-4730-4672 (M. Madan) CEUR htp:/ceur-ws.org ISN1613-073
CEUR
Workshop Proceedings (CEUR-WS.org) https://www.statista.com/statistics/1170631/ online-food-delivery-market-size-worldwide/ • A novel approach to generate an ofline evaluation dataset using users’ interaction with the given recommendations, in the absence of the primary context (items in the cart) • Custom ofline evaluation metrics that comply with the business objectives • Comparison of ofline evaluation with three A/B tests to test the eficacy of our Cart Ofline Evaluation Framework ( )
Instead of either a purely online (or ofline) evaluation, one can reach the middle ground by ofline evaluating using historical data collected online using logs generated from users’ interaction with a recommender system, including “ground truth” inferred from interactions such as emphasized the need to take into account the user’s natural browsing behavior [6]. But it could be challenging to incorporate historical log data for ofline evaluation due to various biases present: “trust bias” which leads to more clicks on links ranked highly by Google, “qualityof-context bias”: clicking decision is also influenced by the overall quality of the other abstracts in the ranking [7], position bias [8], attractiveness bias [9].
There are some studies that have questioned the validity of ofline evaluation and pointed out that error-based, and rank-based metrics computed on ofline datasets do not correlate with online results ([10], [11], [12]). Gruson .
presented a study showing ofline evaluation results can predict online A/B test results under the right an ofline evaluation dataset and metrics that help us predict the direction (positive or negative) of the online framework, using the interactions data for a major on- clicks, add to cart, remove from cart. Gunawardana © 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License approach to de-biasing [4]. In this work, we have curated Attribution 4.0 International (CC BY 4.0). metrics. Secondly, a CatBoost Ranker [18] (gradient
Lastly, we discuss diferent ofline evaluation datasets boosted tree with pair-wise & group-wise losses, wherein & metrics used. Perez . conducted an evaluation grouping was done at level) was built using the study of recommendation models for the traditional historical orders data. The positive training data comtop-N recommendations using impressions data from prised of historical orders, where the items ( 1, 2, 3, open-sourced datasets. The evaluation was done against 4) in an order were split into cart ( 1, 2, 3) & positive the test split [3]. Paraschakis . tried both non- candidate product 4 (last product in an order, based on chronological & chronological split of train-test data [5]. the timestamp of adding it to the cart). Since most of the Agarwal . used the first product (that the user in- orders had up to 4 items, only those orders were considteracted with) as the context & remaining products are ered for training. Orders having fewer than 4 items were assigned to the ground truth set [13]. Talking about the padded by a dummy item. For every , negative canmetrics, Paraschakis . measured precision at the level didates were chosen randomly from the vendor’s menu of recall (R-precision) as it adjusts for the size of the such that they don’t match with the items in the cart & customer’s test set [5]. Other metrics such as Precision, the positive candidate product. Hence, the model Recall, Mean Average Precision, Novelty, Diversity[14] was trained on binary labels. Categorical & numerical and Normalized Cumulative Discounted Gain ([15], [16], features of item, user & vendor were created, including [17]) are commonly used in evaluating recommender pair-wise Lift of candidate item with every item in the systems. While for most of the approaches, either the cart (inspired by Market Basket Analysis [19]), co-bought context was fully available, or the exact context was not frequency of candidate item with the cart (at overall, cusused, our approach aims to create the primary context tomer & vendor level), customer order history (cart size, (items in the cart) for cart cross-selling recommenda- item preferences, etc), price features, hour of the day tions. Also, since the Average Basket Value is one of the features. Hyper-parameter optimization was done on top business objectives in this domain, we’ve introduced the validation split (chronological split: last 1 week of some new ofline evaluation metrics. training data).
Thirdly, this model was used to create that had multi-class labels (instead of binary labels) based 3. Methodology on whether the item was purchased with the cart (1, 2, 3: using price of the candidate item, 3 is the highest The problem statement involves using ofline evaluation price class) & not purchased with the cart (0). Also, the of cart cross-sell recommendations for a major online product names were preprocessed (lowercase, remove food delivery platform to decide the best algorithm for a punctuations, etc) which removed 10% sparsity in the irsneucgcocumessemsrfeusnladodandtliiponrneosdAius/cBttostectsohtma.tOpclnoeetmeoptfhletehmceeapnrutt,rttphhoeeisrreesebxyoifshtceianlrpgt- doanttao.loFginyal(lhsyoa, mvthienegplarasidtodrmitsoitodunedlailesf euastuinrgestwbhaaessmebd:[u2oil0nt]o,tnh[2et1of]po)oo&df items in the cart. From a business point of view, the candidate selection of items (based on food ontologyobjective is to increase the Average Basket Value (ABV): based categories that have been purchased together in (sum of orders’ value/number of orders) and the Order the last 6 months) with aim to improve relevance. Note Share (number of orders with products added from rec- that since there were no latency issues in the previous ommendations/total number of orders). models (as the maximum number of items in a vendor’s menu is less than 200), candidate selection was not done.
In this section, we give an overview of cross-selling al- 3.2. Dataset gorithms that have been experimented on our platform.
The algorithms accept four inputs (items in the cart, user features, vendor features, and time of the day) and return the candidate items (items present in the vendor’s menu) sorted based on the score predicted by the model.
Initially, a heuristics model was built using the historical orders. The candidate items were ranked using a formula that multiplied 1) similarity of past orders with current cart, 2) number of items in the current cart that have been previously purchased by the user, and 3) mean time diference between the current cart & past orders.
Whenever a user lands on the cart page, item recommendations are presented based on the existing items in the cart, user preference, available menu items & time of the day. The dataset consists of impressions (recommendations) & users’ interaction with these impressions.
Figure 1 explains the series of events that are triggered based on the user’s journey. The event ” . ” is ifred when recommendations are loaded. Interaction is defined in terms of Add to Carts ( ) and Remove from Carts ( ). The assumption is that these diferent ways in which a user interacts with an item indicate diferent intents. The aim is to evaluate the performance of the
Our Cart Ofline Evaluation Framework ( divided into three components: ) can be • Create primary context: Items in Cart • Aggregate item interactions • Combine context with interactions 3.3.1. Create primary context: Items in Cart can interact with multiple vendors (restaurants) in a session, 1 is filtered for sessions having maximum vendors (decided based on , distribution of number of vendors interacted at session level) to create 2, thereby removing noise. Example from one market: based on , = 3 is chosen as it covers 90% sessions. ∀ 2, ; ”recos.loaded” events are sorted in the ascending order of their timestamps . To get the primary context (items in the cart) for such that from all −1 , −2 ,...,etc. Then for every , the items in − > 0 (1) where is , , , items in the cart & of . anytime during their session, statistical mode [22] of is assumed at 2, level. Hence, the output of Algorithm 1 are the number of add to carts, remove
Input: • User sessions , gives us 3 (almost 96% of the sessions from 2), from which we create episodes by ranking . events in ascending order of their ∀ , such that: <= <=
+1 gives the positions of items shown to the user for
every . Since cart context is refreshed after every user 2
(2) than
& this, has items with
in episode +1 as well. Then ∀ product interactions are given by equation 1.
In order to estimate the relevance episode level, we assume that
of each item at holds higher weight , which in turn holds higher weight than no interaction. Items without interaction & those with both get 0 weight, whereas items with
hold 3 times more weight than
. Details on these choices of weights are explained in the Results section 4.3.1. Note that the recommendation position is not considered while estimating the relevance. We aim to incorporate it in future works. The output of Algorithm 2 is , , , , item , net
, , recommendation position . 3.3.3. Combine context with interactions
Ofline evaluation data
is finally created by joining the outputs of Algorithm 1 & Algorithm 2 at episode , ”recos.loaded” , user & vendor level to give the sets of impressions (recommendations) & interactions (user’s response). Item’s price information is also added & is used to create by summing over the prices of Given user sessions , user & vendor , the steps for gen- interaction ( or ), . erating the primary context are presented in Algorithm 1. fresh set of recommendations is given. To account for is triggered & a From , 1 is created by filtering for sessions having at least one
from cart recommendations. Since a user , net
Analysis shows that 39% sessions contain just a single episode, & 75% sessions contain <= 3 episodes. Looking at the importance of recommendation position: 35% times items are count for 72% from the 1st position, & top 3 positions ac
, indicating a strong position bias. 37% times only a single item was present in the cart, 2 items were present 30% times, & 3 products 16% times. The price distribution of top 10 recommendation positions show an increasing trend: items added/recommended at 1st position have the lowest price & those at position 10 have the highest price.
Substitution happens whenever a user adds an item from recommendations to the cart & removes an item existing in the cart. It could lead to either an up-sell (if total price of of < >
) or a down-sell (if total price ). Assuming a cart is complete, there are higher chances of up-sell if the recommended items have the same category as the items in the cart (Figure 1 shows an up-sell since ”Burger 1” was substituted by a more expensive ”Burger 2”). Whereas, there are more chances of a down-sell if the recommended items have diferent categories as compared to the items in the cart.
Custom metrics are created to estimate the impact on business objectives (Average Basket Value (ABV) & Order be the sum of prices of items added ; else 0 (4) (3) (5) (6) Share). Let to cart, likewise be for items removed from cart. an increase in the absolute value of relevance weights
The impact on ABV could be estimated by the following metrics (based on top k recommendations):
1. cart value impact (cvi) @k: average percentage improvement in the cart value after relevant items (recommended items present in the ground truth) are added, removed from cart. It is given by∑=1 ( new cart value - cart value )
cart value where new cart value = cart value + - 2. upsell @k: average number of up-sells, where upsell is a binary value based on the following equation
wap @k = ∑( − +1 ) ∗ ∗ & is the relevance given to ℎ item. where , are the precision, recall at the ℎ threshold
5. ndcg @k: Normalized Discounted Cumulative Gain, a standard information retrieval measure used for evaluating the goodness of ranking a set of items.
Using the Cart Ofline Evaluation Framework tion 3.3), the ofline evaluation dataset is prepared (example for 1 market, there are 186K episodes (samples), 134k users, 19k vendors. Using the context (items in the cart, user, vendor, time of the day) from the episodes, available items in the vendor’s menu are ranked (in descending order of their probability of being ordered) using the Cart Recommender. All available items in the vendor’s menu are ranked to mimic the actual online scenario. Then, for each episode, the recommendations are compared with ground truth interactions to come up with the ofline
(Secevaluation metrics set {
} (Section 4.2). 4.3.1. Impact of Relevance Weights In this section, we discuss the impact of the relevance weights of user interactions ℎ threshold, given by =1 3. downsell @k: average number of down-sells at < ∑
; else 0 The impact on Order Share could be estimated by:
4. mean weighted avg. precision @k: mean of weighted average precision (wap), given by =1 −1 =0 > ∑ =1 =1
Online Metrics ABV Order Share 1% * 8% * 0.5% * 0.1% -0.1% 0.3% Libraries: 19th International Conference on Theory and Practice of Digital Libraries, TPDL 2015, Poz[1] Y. Wang, L. Tao, X. Zhang, Recommending for a nań, Poland, September 14-18, 2015, Proceedings three-sided food delivery marketplace: A multi- 19, Springer, 2015, pp. 153–168. objective hierarchical approach, Technical Report, [12] G. G. Gebremeskel, A. P. de Vries, Recommender Working paper, Google Brain, Mountain View, CA, systems evaluations: Ofline, online, time and a/a 2022. test (2016). [2] C. N. Sánchez, J. Domínguez-Soberanes, A. Arreola, [13] P. Agarwal, S. Vempati, S. Borar, Personalizing M. Graf, Recommendation system for a delivery similar product recommendations in fashion efood application based on number of orders, Ap- commerce, arXiv preprint arXiv:1806.11371 (2018). plied Sciences 13 (2023) 2299. [14] M. D. Ekstrand, F. M. Harper, M. C. Willemsen, [3] F. B. Perez Maurera, M. Ferrari Dacrema, P. Cre- J. A. Konstan, User perception of diferences in monesi, Towards the evaluation of recommender recommender algorithms, in: Proceedings of the systems with impressions, in: Proceedings of the 8th ACM Conference on Recommender systems, 16th ACM Conference on Recommender Systems, 2014, pp. 161–168.
2022, pp. 610–615. [15] K. Järvelin, J. Kekäläinen, Cumulated gain-based [4] A. Gruson, P. Chandar, C. Charbuillet, J. McInerney, evaluation of ir techniques, ACM Transactions on S. Hansen, D. Tardieu, B. Carterette, Ofline eval- Information Systems (TOIS) 20 (2002) 422–446. uation to make decisions about playlistrecommen- [16] P. Cremonesi, Y. Koren, R. Turrin, Performance of dation algorithms, in: Proceedings of the Twelfth recommender algorithms on top-n recommendaACM International Conference on Web Search and tion tasks, in: Proceedings of the fourth ACM conData Mining, 2019, pp. 420–428. ference on Recommender systems, 2010, pp. 39–46. [5] D. Paraschakis, B. J. Nilsson, J. Holländer, Com- [17] M. Madan, A. Chouragade, S. Vempati, The joy parative evaluation of top-n recommenders in e- of dressing is an art: Outfit generation using selfcommerce: An industrial perspective, in: 2015 attention bi-lstm, in: Joint European Conference IEEE 14th International Conference on Machine on Machine Learning and Knowledge Discovery in Learning and Applications (ICMLA), IEEE, 2015, pp. Databases, Springer, 2021, pp. 218–233. 1024–1031. [18] A. V. Dorogush, V. Ershov, A. Gulin, Catboost: [6] A. Gunawardana, G. Shani, S. Yogev, Evaluating gradient boosting with categorical features support, recommender systems, in: Recommender systems arXiv preprint arXiv:1810.11363 (2018). handbook, Springer, 2012, pp. 547–601. [19] M. Kaur, S. Kang, Market basket analysis: Identify [7] T. Joachims, L. Granka, B. Pan, H. Hembrooke, the changing trends of market data using associF. Radlinski, G. Gay, Evaluating the accuracy of ation rule mining, Procedia computer science 85 implicit feedback from clicks and query reformula- (2016) 78–85. tions in web search, ACM Transactions on Infor- [20] M. El-Dosuky, M. Z. Rashad, T. Hamza, A. Elmation Systems (TOIS) 25 (2007) 7–es. Bassiouny, Food recommendation using ontology [8] N. Craswell, O. Zoeter, M. Taylor, B. Ramsey, An ex- and heuristics, in: Advanced Machine Learning perimental comparison of click position-bias mod- Technologies and Applications: First International els, in: Proceedings of the 2008 international con- Conference, AMLTA 2012, Cairo, Egypt, Decemference on web search and data mining, 2008, pp. ber 8-10, 2012. Proceedings 1, Springer, 2012, pp. 87–94. 423–429. [9] Y. Yue, R. Patel, H. Roehrig, Beyond position bias: [21] T. N. Trang Tran, M. Atas, A. Felfernig, M. StetExamining result attractiveness as a source of pre- tinger, An overview of recommender systems in sentation bias in clickthrough data, in: Proceedings the healthy food domain, Journal of Intelligent of the 19th international conference on World wide Information Systems 50 (2018) 501–526. web, 2010, pp. 1011–1018. [22] T. Dalenius, The mode–a neglected statistical pa[10] M. Rossetti, F. Stella, M. Zanker, Contrasting of- rameter, Journal of the Royal Statistical Society. lfine and online results when evaluating recom- Series A (General) (1965) 110–117. mendation algorithms, in: Proceedings of the 10th [23] R. Kohavi, R. Longbotham, D. Sommerfield, R. M. ACM conference on recommender systems, 2016, Henne, Controlled experiments on the web: survey pp. 31–34. and practical guide, Data mining and knowledge [11] J. Beel, S. Langer, A comparison of ofline evalu- discovery 18 (2009) 140–181. ations, online evaluations, and user studies in the context of research-paper recommender systems, in: Research and Advanced Technology for Digital