Impressions are a novel data source providing researchers and practitioners with more details about user interactions and their context. In particular, an impression contain the items shown on screen to users, alongside users' interactions toward such items. In recent years, interest in impressions has thrived, and more papers use impressions in recommender systems. Despite this, the literature does not contain a comprehensive review of the current topics and future directions. This work summarizes impressions in recommender systems under three perspectives: recommendation models, datasets with impressions, and evaluation methodologies. Then, we propose several future directions with an emphasis on novel approaches. This work is part of an ongoing review of impressions in recommender systems.
Recommender systems aim to generate user engagement in the short and long term. They achieve this by creating personalized recommendations, i.e., a curation of the catalog tailored to users based on their preferences. When such recommendations are relevant, they generate the desired engagement of users toward the recommender system. However, this is not an easy task or goal. A recommender system must be able to predict future user preferences in various conditions, e.g., when new users arrive, when users change their tastes, or when handling anonymous users.
The research community has devised recommenders using distinct data sources to learn users’ preferences. Interactions are one of the most relevant and commonly used data sources; most research works in recommender systems use interactions. Interactions are those actions users perform toward items of a recommender system, e.g., product purchases or movie ratings.
Using additional data sources is an efective approach to improve the recommendation quality. This work focuses on one specific data source: impressions, also known as slates, past recommendations, exposure. An impression contain information about items shown on-screen to users, alongside the possible interactions with such items. In some cases, an impression also contains layout information, i.e., the collection of items shown on-screen, the position of items, their placement on-screen, and labels indicating which items were interacted with. Impressions are not exclusive to recommender systems; they can be generated by diferent entities, e.g., a search engine or editors. This work, however, focuses on impressions in recommender systems.
With impressions, researchers are able to dissect whether a given item has been shown to a user and whether the user preferred it. Impressions contain mixed signals of users’ preferences, e.g., for a given user and impression, the user may like all, some, or none of the items in the impression. This juxtaposition enables researchers to investigate novel areas or complex attributes of recommender systems using impressions. However, using impressions in recommender systems does come with additional challenges, e.g., the number of impressions is usually higher than the number of interactions, in some cases, by several orders of magnitude.
This work is part of an in-progress systematic literature review on impressions in recommender systems. This work summarizes our review’s relevant findings, highlighting the current state of the art, open research questions, and future directions. Throughout this work, we structure the discussion under three perspectives: recommendation models, datasets with impressions, and evaluation methodologies.
We start this section by providing the definition of the most common and relevant terms used in this work. Then, we proceed with the discussion of the state of the art. We structure the discussion around three fundamental angles in recommender systems: recommendation models, datasets with impressions, and evaluation methodologies.
The reviewed literature consists of regular conference or journal papers published in highlevel venues describing recommenders using impressions. We discovered papers using relevant academic search engines and a query to match papers containing the keywords impression (or its synonyms) and recommender systems in their content. Most papers were discarded due to the broad definition of impression, e.g., some papers use the word impression as how users perceive the recommender system.
We define an impression as a selection of N items to be served to the user by a recommender system or another entity (e.g., a search engine). Upon arrangement of an impression on-screen, users decide whether to interact with items in the impression or not, i.e., a user’s action over an item. Traditionally, the community handles user feedback using user-item pairs, i.e., for a given user and item, the pair indicates whether the user interacted with the item. We refine the definition of user feedback based on users’ interactions with impressions: interacted impression, non-interacted impression, and non-impressed. An interacted impression is a user-item pair where the user interacts with the item. A non-interacted impression is a user-item pair where the user does not interact with a served item. Lastly, a non-impressed is a user-item pair where the item has never been served to the user.
Depending on the recommender’s system data collection strategy, impressions can be categorized into two classes: contextual and global impressions. A contextual impression holds the items shown on-screen and the interactions such items receive. On the contrary, a global impression holds only one of the two: the items shown on-screen or the interactions one item receives.
This section describes recommendation models: the module in a recommender system in charge
of learning users’ preferences and predicting the relevance of items. Recommendation models
can be classified according to the design of the recommender. 1 From the literature, we identify
ifve classes of recommenders:
Heuristics Use ad-hoc functions, techniques, or rules to learn users’ preferences.
Recommenders of this type were published between 2014 and 2017. See [
Deep learning Use deep learning architectures to learn users’ preferences. This is the most
popular recommender type in the literature. The most common deep learning architecture
is the two-tower framework [
Reinforcement learning Model the recommendation task using reinforcement learning, i.e.,
as a Markov decision process [
This section describes datasets with impressions from the reviewed literature. We identify three types of datasets: public, expired, and private datasets. Public datasets are those researchers and practitioners can access via the Internet and use in future works as long as the license agreements are respected. Expired datasets have been used in competitions (e.g., the ACM RecSys Challenge) and are not accessible anymore. Private datasets have never been published nor made available to the community. The downside of expired and private datasets is they cannot be used in future research. 1Due to space limitations, we leave out reviewed papers’ descriptions and more taxonomies looking at other recommenders’ properties.
Inside public datasets, we categorize datasets by the type of impressions they contain:
contextual or global impressions. Three public datasets contain contextual impressions: ContentWise
Impressions [
The last topic addresses the current research goals and the importance of using sound evaluation methodologies to measure progress in the community. Most research papers use ofline evaluations. However, many papers by industrial actors are also performing online evaluations via A/B testing. One paper [27] performs user studies.
The most common research goal in the literature is to improve recommendations quality, where
researchers devise one or several recommendation models to improve a particular evaluation
metric, e.g., precision or diversity. Authors may need to alter existing evaluation methodologies
or create ad-hoc ones when using impressions. Nonetheless, they must be cautious to avoid data
leakages, thus, invalidating their findings. Despite the suggestions made by several previous
works [28, 29, 30], we find some papers use improper evaluation methodologies, e.g., creating
artificial sessions to evaluate session-based recommenders [
This section describes existing open research questions or research needs in the same three perspectives of this work: recommendation models, datasets with impressions, and evaluation methodologies. Then, we identify future work directions addressing such questions or needs.
On recommendation models, we identify the lack of recommenders handling side information in the literature, i.e., those recommenders designed to leverage interactions and other data sources, e.g., factorization machines [32, 33]. In principle, this type of recommender is suitable for using impressions as side information.
On datasets with impressions, we identify the need for more datasets containing contextual impressions. Contextual impressions are more informative than global impressions: researchers may know all items shown at a particular moment and which are interacted and non-interacted impressions.
On evaluations, an open question how to use impressions in the evaluation of recommenders? Felicioni [34] states one future direction is to use impressions to debias evaluation methodologies, e.g., by computing propensity scores. Applying inverse propensity weighting [35] produces an unbiased estimator by adjusting the relevance of items by their propensity score, i.e., the probability of a given user being exposed to a given item. Using impressions to model propensity may be beneficial, as impressions contain the system’s exposed and non-exposed items. 2https://www.kaggle.com/competitions/kddcup2012-track2 ’21: Fifteenth ACM Conference on Recommender Systems, Amsterdam, The Netherlands, 27 September 2021 - 1 October 2021, ACM, 2021, pp. 556–558. URL: https://doi.org/10.1145/ 3460231.3474607. doi:10.1145/3460231.3474607. [18] S. Eide, D. S. Leslie, A. Frigessi, Dynamic slate recommendation with gated recurrent units and thompson sampling, Data Min. Knowl. Discov. 36 (2022) 1756–1786. URL: https://doi.org/10.1007/s10618-022-00849-w. doi:10.1007/s10618-022-00849-w. [19] L. Li, W. Chu, J. Langford, X. Wang, Unbiased ofline evaluation of contextual-bandit-based news article recommendation algorithms, in: I. King, W. Nejdl, H. Li (Eds.), Proceedings of the Forth International Conference on Web Search and Web Data Mining, WSDM 2011, Hong Kong, China, February 9-12, 2011, ACM, 2011, pp. 297–306. URL: https://doi.org/10. 1145/1935826.1935878. doi:10.1145/1935826.1935878. [20] W. Chu, S. Park, T. Beaupre, N. Motgi, A. Phadke, S. Chakraborty, J. Zachariah, A case study of behavior-driven conjoint analysis on yahoo!: front page today module, in: J. F. E. IV, F. Fogelman-Soulié, P. A. Flach, M. J. Zaki (Eds.), Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Paris, France, June 28 - July 1, 2009, ACM, 2009, pp. 1097–1104. URL: https://doi.org/10.1145/1557019.1557138. doi:10.1145/1557019.1557138. [21] C. Gentile, S. Li, G. Zappella, Online clustering of bandits, in: Proceedings of the 31th International Conference on Machine Learning, ICML 2014, Beijing, China, 21-26 June 2014, volume 32 of JMLR Workshop and Conference Proceedings, JMLR.org, 2014, pp. 757–765.
URL: http://proceedings.mlr.press/v32/gentile14.html. [22] S. Sidana, C. Laclau, M. Amini, Learning to recommend diverse items over implicit feedback on PANDOR, in: S. Pera, M. D. Ekstrand, X. Amatriain, J. O’Donovan (Eds.), Proceedings of the 12th ACM Conference on Recommender Systems, RecSys 2018, Vancouver, BC, Canada, October 2-7, 2018, ACM, 2018, pp. 427–431. URL: https://doi.org/10.1145/3240323.3240400. doi:10.1145/3240323.3240400. [23] X. Ma, L. Zhao, G. Huang, Z. Wang, Z. Hu, X. Zhu, K. Gai, Entire space multi-task model: An efective approach for estimating post-click conversion rate, in: K. Collins-Thompson, Q. Mei, B. D. Davison, Y. Liu, E. Yilmaz (Eds.), The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR 2018, Ann Arbor, MI, USA, July 08-12, 2018, ACM, 2018, pp. 1137–1140. URL: https://doi.org/10.1145/3209978.3210104. doi:10.1145/3209978.3210104. [24] Q. Shen, H. Wen, W. Tao, J. Zhang, F. Lv, Z. Chen, Z. Li, Deep interest highlight network for click-through rate prediction in trigger-induced recommendation, in: F. Laforest, R. Troncy, E. Simperl, D. Agarwal, A. Gionis, I. Herman, L. Médini (Eds.), WWW ’22: The ACM Web Conference 2022, Virtual Event, Lyon, France, April 25 - 29, 2022, ACM, 2022, pp. 422–430.
URL: https://doi.org/10.1145/3485447.3511970. doi:10.1145/3485447.3511970. [25] C. Zhu, P. Du, W. Zhang, Y. Yu, Y. Cao, Combo-fashion: Fashion clothes matching CTR prediction with item history, in: A. Zhang, H. Rangwala (Eds.), KDD ’22: The 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Washington, DC, USA, August 14 - 18, 2022, ACM, 2022, pp. 4621–4629. URL: https://doi.org/10.1145/3534678. 3539101. doi:10.1145/3534678.3539101. [26] J. Wang, W. Ma, J. Li, H. Lu, M. Zhang, B. Li, Y. Liu, P. Jiang, S. Ma, Make fairness more fair: Fair item utility estimation and exposure re-distribution, in: A. Zhang, H. Rangwala