Workshop on Learning and Evaluating Recommendations with Impressions (LERI)⋆

Abstract

This volume contains the papers presented at the Workshop on Learning and Evaluating Recommendations with Impressions (LERI), held in conjunction with the 17th ACM Conference on Recommender Systems (RecSys 2023). Recommender systems typically rely on past user interactions as the primary source of information for making predictions. However, although highly informative, past user interactions are strongly biased. Impressions, on the other hand, are a new source of information that indicate the items displayed on screen when the user interacted (or not) with them, and have the potential to impact the ifeld of recommender systems in several ways. Early research on impressions was constrained by the limited availability of public datasets, but this is rapidly changing and, as a consequence, interest in impressions has increased. Impressions present new research questions and opportunities, but also bring new challenges. Several works propose to use impressions as part of recommender models in various ways and discuss their information content. Others explore their potential in of-policy-estimation and reinforcement learning. Overall, the interest of the community is growing, but eforts in this direction remain disconnected. Therefore, one of the aims of the LERI workshop is to bring the community together.

Keywords

• eol>Recommender Systems
• Impressions
• Evaluation
• User Modeling

1. Motivation

ous ways: impressions. Impressions [ 2, 3, 4, 5, 6 ] refer to the items displayed on the screen when a user interacts In the early days of research on recommender systems, (or not) with them and are the product of the whole recpredictions were primarily based on past user interac- ommendation engine [ 7, 5, 8 ]. Impressions constitute a tions and user or item features. However, with advance- nuanced and intricate data source that raises novel rements in technology, the scope and complexity of recom- search questions, opportunities, and challenges. These mender systems have increased and new sources of data may have profound implications for how recommender (such as context, knowledge-bases, and sequence struc- systems are conceptualized, trained, and evaluated. ture) have emerged, driving the field forward and creating Impressions took longer than ratings and interactions thriving sub-fields. Nevertheless, past user interactions to cross the corporate boundary towards wider research remain the most potent and comprehensive source of pre- availability [9, 10, 11, 12]. This started to happen eventudictive power. Despite this, observed interactions are a ally: early examples include the ACM RecSys Challenge sparse and strongly biased source of information, which in 2016, 2017 and 2019 [13, 14, 15], where the released has significant implications for both learning from user datasets included impression data. Until recently, reactions and evaluating the quality of recommendations search was still limited by the lack of datasets, this was ofline [ 1 ]. because the datasets released as part of the RecSys chal

Recently, a source of information that was previously lenges are usually non redistributable and focus on very almost unavailable to the wider research community has specific and narrow applications, while only very few emerged with the potential to impact the field in numer- other datasets were publicly available. This is rapidly Workshop on Learning and Evaluating Recommendations with Impres- changing and most of the available datasets including sions (LERI) @ RecSys 2023, September 18-22 2023, Singapore impressions have been published in the last few years: ⋆Workshop held in conjunction with the 17th ACM Conference on e.g., ContentWise Impressions [7], MIND [16], FINN.no Recommender Systems (RecSys 2023), in Singapore. Slates [ 5 ], Pandor [17], Ali-CCP [18], Alimama [19], $ [e-mail] (M. Ferrari Dacrema); Cross-shop Combo [20], In-Shop Combo [20], Kwai FAIR (pJa.bBlaos.cilaisctoe)l;[e-mail](oPn.eCsai@steplolsl)im;jbi.aits([e-mail])m System [21], Kwai FAIR Experiment [21]. With the emer0000-0001-7103-2788 (M. Ferrari Dacrema); 0000-0003-0668-6317 gence of these new datasets, studying the use of impres(P. Castells); 0000-0002-3005-5200 (J. Basilico); 0000-0002-1253-8081 sions has become a more accessible topic for research. (P. Cremonesi) However, despite the increasing research interest, the ef© 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License forts devoted to studying the use of impressions are still CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g ACttEribUutRion W4.0oInrtekrnsahtioonpal (PCCroBYce4.0e).dings (CEUR-WS.org) Some works have already tried to use impressions to build better recommendation models in various ways: [22, 23, 24, 25, 26, 27] use impression data to compute features, re-ranking, sampling and to learn biases. Furthermore [28, 29, 30, 31, 32, 33, 34] apply neural or deep-learning models including impressions. Most of these papers have been published in the last two years in conferences such as SIGIR, KDD, WWW and RecSys.

Among the new research opportunities opened by impressions, being able to distinguish between the items that the user observed and did not observe could allow to provide better assumptions on how to label missing interactions. Some studies consider impressions to be a positive interaction signal, while others view them as negative signals [8].

Impressions also provide a direction for research that could help to bridge the gap between algorithms and user experience, two sides of recommender systems that are often studied independently of each other. For instance, continuously recommending the same item may lead to user fatigue [25], resulting in reduced user satisfaction with the system and wasted recommendations.

By using impressions, recommender systems can better understand how users interact with the system and, thus, provide recommendations that improve user experience and engagement.

A further direction of research is in the evaluation of recommendation models. It is known that the past user interactions are a highly biased data source [ 1 ] and impressions, which represent the real behavior of the recommendation engine that acts as the intermediary between the user and the available catalogue, could allow to better identify those biases. The community is also exploring new methods for the evaluation of recommender systems, such as of-policy estimation (OPE) [ 35, 36, 37] and simulation environments [38] some of which already use impressions [39].

1.2. Workshop Description

The Workshop on Learning and Evaluating Recommendations with Impressions focuses on all aspects related to leveraging impression data to build and evaluate a recommendation engine. The goal is to both help to coalesce researchers exploring the use of impressions from diferent perspectives, as well as foster increased interest from the community for this new and still largely underexplored topic that has the potential of impacting the limited and fragmented. Therefore, one of the core aims field in several ways. The workshop aims to provide a of the LERI workshop is to bring together and consolidate venue for researchers and practitioners to come together the community working on this topic. in order to: (i) share experience and lessons learned; (ii) identify key challenges in the area; (iii) build a common 1.1. Status of Research, Challenges and mental model and conceptual framework for thinking and researching on the use of impressions; (iv) identify Opportunities emerging topics and new opportunities. The workshop also aims to lay bridges between practitioners and academics, encourage a wider availability of impression data sources and leverage industry’s experience to guide and inform academic research.

1.3. Workshop Topics

Conceptual framework: definition of “impression”, role of impressions in the recommendation task definition, user action attribution to impressions, prediction and causation, closed vs. open loops;

Recommendation models: new learning approaches taking advantage of impression data, impressions in label data, loss functions, model topologies;

Model training: data preprocessing, sampling, partitioning, hyperparameter tuning with impressions;

Evaluation: evaluation methodology and metrics, impact on ofline evaluation bias;

User modeling: new models considering user behavior in face of impressed items;

Reinforcement learning and of-policy estimation: ofline vs. online setting, impressions in RL and OPE;

Datasets: collection of new datasets with impressions from diferent domains, user interfaces, applications;

User Studies: how the user behavior is impacted by the composition of impressions, impact of user fatigue, etc.;

Theory: theoretical aspects in the use of impressions for recommender systems, both in the development of new and improved recommender systems and in their evaluation;

Perspectives: new perspectives on existing problems that could benefit or just change by adding impressions as a new variable, as well as old challenges that can be now tackled from new angles, and new challenges that derive from the use of impressions.

1.4. Workshop Organization

The workshop has been organized by:

Maurizio Ferrari Dacrema: Professor at Politecnico di Milano. His research interests include recommender systems evaluation and quantum computing. He has been local organization chair at the 12th Italian Information Retrieval Workshop.1

Pablo Castells: Professor at Universidad Autónoma de

Madrid (UAM) and Amazon scholar. His research interests include recommender systems evaluation, algorithmic and experimental bias, and beyond-accuracy perspectives. He has organized six RecSys workshops in areas such as evaluation and experimentation, novelty and diversity, and industry applications; as well as workshops and tutorials at SIGIR, WSDM and The Web Conference. He has served in the RecSys organizing committee in diferent roles including PC co-chair in 2016, and served as PC co-chair and general co-chair of SIGIR in 2021 and 2022 respectively.

Justin Basilico: Netflix. He has been an Industry co

chair at RecSys 2022 and 2023, he has coorganized the 2020 and 2021 International Workshop on Industrial Recommendation Systems at KDD, and the 2022 REVEAL workshop at RecSys. He also coorganizes the annual Netflix Personalization, Recommendation, and Search (PRS) workshop.

impressions has been connected to the following other workshops: Causality, Counterfactuals, Sequential Decision–

Making & Reinforcement Learning for Recommender Systems (RecSys 2022) the workshop did not discuss primarily impressions but the topic of of-policy estimation is connected to the availability of information on the real user preferences and on the bias introduced by the recommendation engine which could be estimated using impressions.

ACM RecSys Challenge Workshop (RecSys 2019, 2017 and 2016) the workshop did not discuss primarily impressions but the data available during the challenge included impressions and therefore some of the papers described how the teams used them.

RecSys workshops on recommender systems evaluation: Evaluation has been a recurring workshop topic at RecSys: workshops such as RUE 2012, RepSys 2013, REDD 2014, SimuRec 2021 have focused on ofline evaluation methodology, metrics, reproducibility, bias, and datasets, among many other important elements and issues in recommender system evaluation. As far as the proposers are aware (as co-organizers of these past workshops), impressions were not addressed or discussed in that scope so far.

Paolo Cremonesi: Professor at Politecnico di Milano and co-Founder of ContentWise. His research in- 3. Program Committee terests include recommender systems and quantum computing. He has served in the organi- The following is a list of the program committee: Anzation of scientific meetings in diferent roles, tonio Ferrara (Politecnico di Bari), Claudio Pomo including program chair of ACM iTVX in 2013, (Politecnico di Bari), Daniele Malitesta (Politecnico di and general co-chair of ACM RecSys in 2016. He Bari), David Massimo (Free University of Bolzano), Ferserves in the RecSys steering committee since nando Benjamín Pérez Maurera (Politecnico di Mi2017. lano), Marco de Gemmis (Università degli Studi di Bari Aldo Moro), Marco Polignano (Università degli Studi di The proceedings have been curated by: Bari Aldo Moro), Maurizio Ferrari Dacrema (Politecnico di Milano), Nicolò Felicioni (Politecnico di Milano), Olivier Jeunen (ShareChat), Pengjie Ren (Shandong University), Vito Walter Anelli (Politecnico di Bari).

Fernando Benjamín Pérez Maurera: Ph.D. candidate in Information Technology at Politecnico di Milano. His research interests include recommender systems evaluation and algorithmic design; specifically, impression-aware recommender systems.

2. Related Prior Workshops

We are not aware of any prior workshop that focused

on the topic of impressions itself. However, the use of

1https://recsyspolimi.github.io/iir2022/

4. Timeline

The following is the timeline of LERI: • Paper submission deadline: August 10th, 2023 • Author notification: August 28th, 2023 • Camera-ready version deadline: September 10th, 2023

5. Workshop Program

Panel Discussion: Moderated by Paolo Cremonesi

with panelists Jiangwei Pan (Netflix), Arnab Bhadury The workshop program was articulated as follows.2 (YouTube) and Srijan Saket (Sharechat). The discussion focused on the challenges to conduct research on the use Keynote by Jiangwei Pan: “Recommendation Mod- of impressions outside companies, due to the highly coneling with Impression Data at Netflix”. The keynote dis- textual nature of impression data and the strong conneccussed how impression data is used to build recommender tion of their information content to the specific domain models at Netflix, focusing in particular on which issues and user interface. The panelists also discussed the usearise when using impressions to train a recommendation fulness of impressions in multimedia recommendation model at scale in a two-pass architecture, where impres- where it is dificult to obtain good metadata and content sions can be useful to perform a more fine-grained rank- can have a short shelf-life, meaning impressions can be ing. Furthermore, the keynote discussed how repeated useful to provide more information in a shorter amount impressions afect the user behavior, their correspond- of time. ing long-term value and the issue of performing enough item exploration without negatively afecting the user satisfaction.3 References Accepted Papers: • [40] Impression-Informed Multi-Behavior Recommender System: A Hierarchical Graph Attention Approach; Dong Li; Divya Bhargavi; Vidya Sagar

Ravipati • [41] Characterizing Impression-Aware Recommender Systems; Fernando Benjamín Pérez Maurera; Maurizio Ferrari Dacrema; Pablo Castells;

Paolo Cremonesi • [42] Efects of Human-curated Content on Diversity in PSM: ARD-M Dataset; Marcel Hauck; Ahtsham

Manzoor; Sven Pagel • [43] Formulating Video Watch Success Signals for

Recommendations on Short Video Platforms; Srijan Saket; Sai Baba Reddy Velugoti; Rishabh Mehrotra • [44] Ofline Evaluation using Interactions to Decide Cross-selling Recommendations Algorithm for

Online Food Delivery; Manchit Madan • [45] Contextual Position Bias Estimation Using a Single Stochastic Logging Policy; Giuseppe Di Benedetto; Alexander Buchholz; Ben London; Matej Jakimov; Yannik Stein; Jan Malte Lichtenberg; Vito Bellini; Matteo Rufini; Thorsten

Joachims • [46] Incorporating Impressions to Graph-Based Recommenders; Fernando Benjamín Pérez Maurera; Maurizio Ferrari Dacrema; Pablo Castells; Paolo

Cremonesi

2https://recsyspolimi.github.io/leri2023/

3https://www.slideshare.net/JiangweiPan/ recommendation-modeling-with-impression-data-at-netflixpptx and thompson sampling, CoRR abs/2104.15046 Y. Li, H. Jin, X. Xie, Show me the whole world: (2021). URL: https://arxiv.org/abs/2104.15046. Towards entire item space exploration for inarXiv:2104.15046. teractive personalized recommendations, in: [6] X. Xin, J. Yang, H. Wang, J. Ma, P. Ren, H. Luo, X. Shi, K. S. Candan, H. Liu, L. Akoglu, X. L. Dong, Z. Chen, Z. Ren, On the user behavior leakage from J. Tang (Eds.), WSDM ’22: The Fifteenth ACM recommender system exposure, ACM Trans. Inf. International Conference on Web Search and Syst. (2022). URL: https://doi.org/10.1145/3568954. Data Mining, Virtual Event / Tempe, AZ, USA, doi:10.1145/3568954, just Accepted. February 21 - 25, 2022, ACM, 2022, pp. 947–956. [7] F. B. Pérez Maurera, M. Ferrari Dacrema, L. Saule, URL: https://doi.org/10.1145/3488560.3498459.

M. Scriminaci, P. Cremonesi, Contentwise im- doi:10.1145/3488560.3498459. pressions: An industrial dataset with impressions [13] F. Abel, A. A. Benczúr, D. Kohlsdorf, M. A. Larincluded, in: M. d’Aquin, S. Dietze, C. Hauf, son, R. Pálovics, Recsys challenge 2016: Job recE. Curry, P. Cudré-Mauroux (Eds.), CIKM ’20: The ommendations, in: S. Sen, W. Geyer, J. Freyne, 29th ACM International Conference on Informa- P. Castells (Eds.), Proceedings of the 10th ACM tion and Knowledge Management, Virtual Event, Conference on Recommender Systems, Boston, MA, Ireland, October 19-23, 2020, ACM, 2020, pp. 3093– USA, September 15-19, 2016, ACM, 2016, pp. 425– 3100. URL: https://doi.org/10.1145/3340531.3412774. 426. URL: https://doi.org/10.1145/2959100.2959207. doi:10.1145/3340531.3412774. doi:10.1145/2959100.2959207. [8] F. B. Pérez Maurera, M. Ferrari Dacrema, P. Cre- [14] F. Abel, Y. Deldjoo, M. Elahi, D. Kohlsdorf, Recmonesi, Towards the evaluation of recommender sys challenge 2017: Ofline and online evaluasystems with impressions, in: J. Golbeck, F. M. tion, in: P. Cremonesi, F. Ricci, S. Berkovsky, Harper, V. Murdock, M. D. Ekstrand, B. Shapira, A. Tuzhilin (Eds.), Proceedings of the Eleventh ACM J. Basilico, K. T. Lundgaard, E. Oldridge (Eds.), Rec- Conference on Recommender Systems, RecSys Sys ’22: Sixteenth ACM Conference on Recom- 2017, Como, Italy, August 27-31, 2017, ACM, 2017, mender Systems, Seattle, WA, USA, September 18 pp. 372–373. URL: https://doi.org/10.1145/3109859. - 23, 2022, ACM, 2022, pp. 610–615. URL: https: 3109954. doi:10.1145/3109859.3109954. //doi.org/10.1145/3523227.3551483. doi:10.1145/ [15] P. Knees, Y. Deldjoo, F. B. Moghaddam, J. Adam3523227.3551483. czak, G. P. Leyson, P. Monreal, Recsys challenge [9] D. Agarwal, B. Chen, P. Elango, Spatio-temporal 2019: session-based hotel recommendations, in: models for estimating click-through rate, in: Pro- T. Bogers, A. Said, P. Brusilovsky, D. Tikk (Eds.), Proceedings of the 18th International Conference on ceedings of the 13th ACM Conference on RecomWorld Wide Web, WWW 2009, Madrid, Spain, April mender Systems, RecSys 2019, Copenhagen, Den20-24, 2009, ACM, 2009, pp. 21–30. URL: https: mark, September 16-20, 2019, ACM, 2019, pp. 570– //doi.org/10.1145/1526709.1526713. doi:10.1145/ 571. URL: https://doi.org/10.1145/3298689.3346974. 1526709.1526713. doi:10.1145/3298689.3346974. [10] N. Gupta, A. Das, S. Pandey, V. K. Narayanan, Fac- [16] F. Wu, Y. Qiao, J. Chen, C. Wu, T. Qi, J. Lian, toring past exposure in display advertising target- D. Liu, X. Xie, J. Gao, W. Wu, M. Zhou, MIND: ing, in: Q. Yang, D. Agarwal, J. Pei (Eds.), The 18th A large-scale dataset for news recommendation, ACM SIGKDD International Conference on Knowl- in: D. Jurafsky, J. Chai, N. Schluter, J. R. Tetreault edge Discovery and Data Mining, KDD ’12, Beijing, (Eds.), Proceedings of the 58th Annual Meeting China, August 12-16, 2012, ACM, 2012, pp. 1204– of the Association for Computational Linguistics, 1212. URL: https://doi.org/10.1145/2339530.2339719. ACL 2020, Online, July 5-10, 2020, Association for doi:10.1145/2339530.2339719. Computational Linguistics, 2020, pp. 3597–3606. [11] C. Wu, F. Wu, T. Qi, C. Li, Y. Huang, Is news rec- URL: https://doi.org/10.18653/v1/2020.acl-main.331. ommendation a sequential recommendation task?, doi:10.18653/v1/2020.acl-main.331. in: E. Amigó, P. Castells, J. Gonzalo, B. Carterette, [17] S. Sidana, C. Laclau, M. Amini, Learning to recJ. S. Culpepper, G. Kazai (Eds.), SIGIR ’22: The 45th ommend diverse items over implicit feedback on International ACM SIGIR Conference on Research PANDOR, in: S. Pera, M. D. Ekstrand, X. Amand Development in Information Retrieval, Madrid, atriain, J. O’Donovan (Eds.), Proceedings of the Spain, July 11 - 15, 2022, ACM, 2022, pp. 2382– 12th ACM Conference on Recommender Systems, 2386. URL: https://doi.org/10.1145/3477495.3531862. RecSys 2018, Vancouver, BC, Canada, October 2-7, doi:10.1145/3477495.3531862. 2018, ACM, 2018, pp. 427–431. URL: https://doi.org/ [12] Y. Song, S. Sun, J. L. and10.1145/3543846/non- 10.1145/3240323.3240400. doi:10.1145/3240323. impressions/reinforcement-learning-based- 3240400. recommender-systems-a-survey Hong Huang, [18] X. Ma, L. Zhao, G. Huang, Z. Wang, Z. Hu, X. Zhu, K. Gai, Entire space multi-task model: An efec- E. Gabrilovich (Eds.), Proceedings of the 26th Intertive approach for estimating post-click conversion national Conference on World Wide Web Companrate, in: K. Collins-Thompson, Q. Mei, B. D. Davi- ion, Perth, Australia, April 3-7, 2017, ACM, 2017, son, Y. Liu, E. Yilmaz (Eds.), The 41st International pp. 583–592. URL: https://doi.org/10.1145/3041021. ACM SIGIR Conference on Research & Develop- 3054202. doi:10.1145/3041021.3054202. ment in Information Retrieval, SIGIR 2018, Ann Ar- [25] C. Wu, C. V. Alvino, A. J. Smola, J. Basilico, Using bor, MI, USA, July 08-12, 2018, ACM, 2018, pp. 1137– navigation to improve recommendations in real1140. URL: https://doi.org/10.1145/3209978.3210104. time, in: S. Sen, W. Geyer, J. Freyne, P. Castells doi:10.1145/3209978.3210104. (Eds.), Proceedings of the 10th ACM Conference on [19] Q. Shen, H. Wen, W. Tao, J. Zhang, F. Lv, Z. Chen, Recommender Systems, Boston, MA, USA, SeptemZ. Li, Deep interest highlight network for click- ber 15-19, 2016, ACM, 2016, pp. 341–348. URL: https: through rate prediction in trigger-induced recom- //doi.org/10.1145/2959100.2959174. doi:10.1145/ mendation, in: F. Laforest, R. Troncy, E. Simperl, 2959100.2959174.

D. Agarwal, A. Gionis, I. Herman, L. Médini (Eds.), [26] X. Zhang, Y. Zhou, Y. Ma, B. Chen, L. Zhang, WWW ’22: The ACM Web Conference 2022, Virtual D. Agarwal, Glmix: Generalized linear mixed modEvent, Lyon, France, April 25 - 29, 2022, ACM, 2022, els for large-scale response prediction, in: B. Krpp. 422–430. URL: https://doi.org/10.1145/3485447. ishnapuram, M. Shah, A. J. Smola, C. C. Aggarwal, 3511970. doi:10.1145/3485447.3511970. D. Shen, R. Rastogi (Eds.), Proceedings of the 22nd [20] C. Zhu, P. Du, W. Zhang, Y. Yu, Y. Cao, Combo- ACM SIGKDD International Conference on Knowlfashion: Fashion clothes matching CTR prediction edge Discovery and Data Mining, San Francisco, with item history, in: A. Zhang, H. Rangwala (Eds.), CA, USA, August 13-17, 2016, ACM, 2016, pp. 363– KDD ’22: The 28th ACM SIGKDD Conference on 372. URL: https://doi.org/10.1145/2939672.2939684. Knowledge Discovery and Data Mining, Washing- doi:10.1145/2939672.2939684. ton, DC, USA, August 14 - 18, 2022, ACM, 2022, pp. [27] M. Aharon, Y. Kaplan, R. Levy, O. Somekh, A. Blanc, 4621–4629. URL: https://doi.org/10.1145/3534678. N. Eshel, A. Shahar, A. Singer, A. Zlotnik, Soft fre3539101. doi:10.1145/3534678.3539101. quency capping for improved ad click prediction in [21] J. Wang, W. Ma, J. Li, H. Lu, M. Zhang, B. Li, yahoo gemini native, in: W. Zhu, D. Tao, X. Cheng, Y. Liu, P. Jiang, S. Ma, Make fairness more P. Cui, E. A. Rundensteiner, D. Carmel, Q. He, J. X. fair: Fair item utility estimation and exposure re- Yu (Eds.), Proceedings of the 28th ACM Internadistribution, in: A. Zhang, H. Rangwala (Eds.), KDD tional Conference on Information and Knowledge ’22: The 28th ACM SIGKDD Conference on Knowl- Management, CIKM 2019, Beijing, China, Novemedge Discovery and Data Mining, Washington, DC, ber 3-7, 2019, ACM, 2019, pp. 2793–2801. URL: https: USA, August 14 - 18, 2022, ACM, 2022, pp. 1868– //doi.org/10.1145/3357384.3357801. doi:10.1145/ 1877. URL: https://doi.org/10.1145/3534678.3539354. 3357384.3357801.

doi:10.1145/3534678.3539354. [28] Z. Chen, J. Wu, C. Li, J. Chen, R. Xiao, B. Zhao, Co[22] N. Buchbinder, M. Feldman, A. Ghosh, training disentangled domain adaptation network J. Naor, Frequency capping in online ad- for leveraging popularity bias in recommenders, vertising, J. Sched. 17 (2014) 385–398. URL: in: E. Amigó, P. Castells, J. Gonzalo, B. Carterette, https://doi.org/10.1007/s10951-014-0367-z. J. S. Culpepper, G. Kazai (Eds.), SIGIR ’22: The doi:10.1007/s10951-014-0367-z. 45th International ACM SIGIR Conference on Re[23] Q. Zhao, G. Adomavicius, F. M. Harper, M. C. search and Development in Information Retrieval, Willemsen, J. A. Konstan, Toward better interac- Madrid, Spain, July 11 - 15, 2022, ACM, 2022, pp. 60– tions in recommender systems: Cycling and serpen- 69. URL: https://doi.org/10.1145/3477495.3531952. tining approaches for top-n item lists, in: C. P. Lee, doi:10.1145/3477495.3531952. S. E. Poltrock, L. Barkhuus, M. Borges, W. A. Kellogg [29] P. Li, R. Chen, Q. Liu, J. Xu, B. Zheng, Trans(Eds.), Proceedings of the 2017 ACM Conference form cold-start users into warm via fused behavon Computer Supported Cooperative Work and So- iors in large-scale recommendation, in: E. Amigó, cial Computing, CSCW 2017, Portland, OR, USA, P. Castells, J. Gonzalo, B. Carterette, J. S. CulpepFebruary 25 - March 1, 2017, ACM, 2017, pp. 1444– per, G. Kazai (Eds.), SIGIR ’22: The 45th Interna1453. URL: https://doi.org/10.1145/2998181.2998211. tional ACM SIGIR Conference on Research and doi:10.1145/2998181.2998211. Development in Information Retrieval, Madrid, [24] D. C. Liu, S. K. Rogers, R. Shiau, D. Kislyuk, K. C. Spain, July 11 - 15, 2022, ACM, 2022, pp. 2013– Ma, Z. Zhong, J. Liu, Y. Jing, Related pins at pin- 2017. URL: https://doi.org/10.1145/3477495.3531797. terest: The evolution of a real-world recommender doi:10.1145/3477495.3531797. system, in: R. Barrett, R. Cummings, E. Agichtein, [30] J. Lou, H. Wen, F. Lv, J. Zhang, T. Yuan, Z. Li, Re-weighting negative samples for model-agnostic port using side information, in: Advances in Neumatching, in: E. Amigó, P. Castells, J. Gonzalo, ral Information Processing Systems (NeurIPS 2022), B. Carterette, J. S. Culpepper, G. Kazai (Eds.), SI- 2022, p. 15 pages.

GIR ’22: The 45th International ACM SIGIR Con- [37] A. H. Jadidinejad, C. Macdonald, I. Ounis, The ference on Research and Development in Informa- simpson’s paradox in the ofline evaluation of rection Retrieval, Madrid, Spain, July 11 - 15, 2022, ommendation systems, ACM Trans. Inf. Syst. ACM, 2022, pp. 1823–1827. URL: https://doi.org/ 40 (2022) 4:1–4:22. URL: https://doi.org/10.1145/ 10.1145/3477495.3532053. doi:10.1145/3477495. 3458509. doi:10.1145/3458509.

3532053. [38] D. Rohde, S. Bonner, T. Dunlop, F. Vasile, A. Karat[31] S. Gong, K. Q. Zhu, Positive, negative and neu- zoglou, Recogym: A reinforcement learning tral: Modeling implicit feedback in session-based environment for the problem of product recnews recommendation, in: E. Amigó, P. Castells, ommendation in online advertising, CoRR J. Gonzalo, B. Carterette, J. S. Culpepper, G. Kazai abs/1808.00720 (2018). URL: http://arxiv.org/abs/ (Eds.), SIGIR ’22: The 45th International ACM SI- 1808.00720. arXiv:1808.00720. GIR Conference on Research and Development [39] J. McInerney, E. Elahi, J. Basilico, Y. Raimond, T. Jein Information Retrieval, Madrid, Spain, July 11 bara, Accordion: A trainable simulator forlong- 15, 2022, ACM, 2022, pp. 1185–1195. URL: https: term interactive systems, in: H. J. C. Pampín, //doi.org/10.1145/3477495.3532040. doi:10.1145/ M. A. Larson, M. C. Willemsen, J. A. Konstan, 3477495.3532040. J. J. McAuley, J. Garcia-Gathright, B. Huurnink, [32] H. Wang, T. Chang, T. Liu, J. Huang, Z. Chen, C. Yu, E. Oldridge (Eds.), RecSys ’21: Fifteenth ACM ConR. Li, W. Chu, ESCM2: entire space counterfac- ference on Recommender Systems, Amsterdam, tual multi-task model for post-click conversion rate The Netherlands, 27 September 2021 - 1 October estimation, in: E. Amigó, P. Castells, J. Gonzalo, 2021, ACM, 2021, pp. 102–113. URL: https://doi.org/ B. Carterette, J. S. Culpepper, G. Kazai (Eds.), SIGIR 10.1145/3460231.3474259. doi:10.1145/3460231. ’22: The 45th International ACM SIGIR Conference 3474259. on Research and Development in Information Re- [40] D. Li, D. Bhargavi, V. S. Ravipati, Impressiontrieval, Madrid, Spain, July 11 - 15, 2022, ACM, 2022, informed multi-behavior recommender system: A pp. 363–372. URL: https://doi.org/10.1145/3477495. hierarchical graph attention approach, in: Pro3531972. doi:10.1145/3477495.3531972. ceedings of the Workshop on Learning and Evalu[33] N. Ma, M. Ispir, Y. Li, Y. Yang, Z. Chen, D. Z. Cheng, ating Recommendations with Impressions (LERI) L. Nie, K. Barman, An online multi-task learn- co-located with the 17th ACM Conference on Recing framework for google feed ads auction mod- ommender Systems (RecSys 2023), Singapore, 18thels, in: A. Zhang, H. Rangwala (Eds.), KDD ’22: 22nd September 2023, CEUR Workshop ProceedThe 28th ACM SIGKDD Conference on Knowl- ings, CEUR-WS.org, 2023, pp. 9–21. edge Discovery and Data Mining, Washington, DC, [41] F. B. Pérez Maurera, M. Ferrari Dacrema, P. Castells, USA, August 14 - 18, 2022, ACM, 2022, pp. 3477– P. Cremonesi, Characterizing impression-aware rec3485. URL: https://doi.org/10.1145/3534678.3539055. ommender systems, in: Proceedings of the Workdoi:10.1145/3534678.3539055. shop on Learning and Evaluating Recommenda[34] R. Xie, S. Zhang, R. Wang, F. Xia, L. Lin, A peep tions with Impressions (LERI) co-located with the into the future: Adversarial future encoding in rec- 17th ACM Conference on Recommender Systems ommendation, in: K. S. Candan, H. Liu, L. Akoglu, (RecSys 2023), Singapore, 18th-22nd September X. L. Dong, J. Tang (Eds.), WSDM ’22: The Fifteenth 2023, CEUR Workshop Proceedings, CEUR-WS.org, ACM International Conference on Web Search and 2023, pp. 22–33.

Data Mining, Virtual Event / Tempe, AZ, USA, [42] M. Hauck, A. Manzoor, S. Pagel, Efects of humanFebruary 21 - 25, 2022, ACM, 2022, pp. 1177– curated content on diversity in PSM: ARD-M 1185. URL: https://doi.org/10.1145/3488560.3498476. dataset, in: Proceedings of the Workshop on Learndoi:10.1145/3488560.3498476. ing and Evaluating Recommendations with Impres[35] D. Precup, R. S. Sutton, S. Singh, Eligibility traces sions (LERI) co-located with the 17th ACM Confor of-policy policy evaluation, in: P. Langley (Ed.), ference on Recommender Systems (RecSys 2023), Proceedings of the Seventeenth International Con- Singapore, 18th-22nd September 2023, CEUR Workference on Machine Learning (ICML 2000), Stanford shop Proceedings, CEUR-WS.org, 2023, pp. 34–40. University, Stanford, CA, USA, June 29 - July 2, 2000, [43] S. Saket, S. B. R. Velugoti, R. Mehrotra, Formulating Morgan Kaufmann, 2000, pp. 759–766. video watch success signals for recommendations [36] N. Felicioni, M. Ferrari Dacrema, M. Restelli, P. Cre- on short video platforms, in: Proceedings of the monesi, Of-policy evaluation with deficient sup- Workshop on Learning and Evaluating Recommendations with Impressions (LERI) co-located with the 17th ACM Conference on Recommender Systems (RecSys 2023), Singapore, 18th-22nd September 2023, CEUR Workshop Proceedings, CEUR

WS.org, 2023, pp. 41–48. [44] M. Madan, Ofline evaluation using interactions to decide cross-selling recommendations algorithm for online food delivery, in: Proceedings of the Workshop on Learning and Evaluating Recommendations with Impressions (LERI) co-located with the 17th ACM Conference on Recommender Systems (RecSys 2023), Singapore, 18th-22nd September 2023, CEUR Workshop Proceedings, CEUR

WS.org, 2023, pp. 49–54. [45] G. Di Benedetto, A. Buchholz, B. London, M. Jakimov, Y. Stein, J. M. Lichtenberg, V. Bellini, M. Rufini, T. Joachims, Contextual position bias estimation using a single stochastic logging policy, in: Proceedings of the Workshop on Learning and Evaluating Recommendations with Impressions (LERI) co-located with the 17th ACM Conference on Recommender Systems (RecSys 2023), Singapore, 18th22nd September 2023, CEUR Workshop Proceedings, CEUR-WS.org, 2023, pp. 55–61. [46] F. B. Pérez Maurera, M. Ferrari Dacrema, P. Castells,

P. Cremonesi, Incorporating impressions to graphbased recommenders, in: Proceedings of the Workshop on Learning and Evaluating Recommendations with Impressions (LERI) co-located with the 17th ACM Conference on Recommender Systems (RecSys 2023), Singapore, 18th-22nd September 2023, CEUR Workshop Proceedings, CEUR-WS.org, 2023, pp. 62–67.

References

[1] T. Schnabel , A. Swaminathan , A. Singh , N. Chandak , T. Joachims, Recommendations as treatments: Debiasing learning and evaluation , in: M. Balcan , K. Q. Weinberger (Eds.), Proceedings of the 33nd International Conference on Machine Learning , ICML 2016 , New York City, NY, USA, June 19-24, 2016 , volume 48 of JMLR Workshop and Conference Proceedings, JMLR.org , 2016 , pp. 1670 - 1679 . URL: http://proceedings.mlr.press/v48/schnabel16.html.

[2] D. Liang , L. Charlin , J. McInerney , D. M. Blei , Modeling user exposure in recommendation , in: J. Bourdeau , J. Hendler , R. Nkambou , I. Horrocks , B. Y. Zhao (Eds.), Proceedings of the 25th International Conference on World Wide Web, WWW 2016 , Montreal, Canada, April 11 - 15 , 2016 , ACM, 2016 , pp. 951 - 961 . URL: https://doi.org/10.1145/2872427. 2883090. doi: 10 .1145/2872427.2883090.

[3] T. Shih , T. Hou , J. Jiang , Y. Lien , C. Lin , P. Cheng, Dynamically integrating item exposure with rating prediction in collaborative filtering , in: R. Perego , F. Sebastiani , J. A. Aslam , I. Ruthven , J. Zobel (Eds.), Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval , SIGIR 2016 , Pisa, Italy, July 17-21 , 2016 , ACM, 2016 , pp. 813 - 816 . URL: https: //doi.org/10.1145/2911451.2914769. doi: 10 .1145/ 2911451.2914769.

[4] M. Sato , J. Singh , S. Takemori , T. Sonoda , Q. Zhang , T. Ohkuma, Uplift-based evaluation and optimization of recommenders , in: T. Bogers , A. Said , P. Brusilovsky , D. Tikk (Eds.), Proceedings of the 13th ACM Conference on Recommender Systems, RecSys 2019 , Copenhagen, Denmark, September 16-20 , 2019 , ACM, 2019 , pp. 296 - 304 . URL: https: //doi.org/10.1145/3298689.3347018. doi: 10 .1145/ 3298689.3347018.

[5] S. Eide , D. S. Leslie , A. Frigessi , Dynamic slate recommendation with gated recurrent units