Many video-on-demand and music streaming services provide the user with a page consisting of several recommendation lists, i.e., widgets or swipeable carousels, each built with a specific criteria (e.g., most recent, TV series, etc.). Finding eficient strategies to select which carousels to display is an active research topic of great industrial interest. In this setting the overall quality of the recommendations of a new algorithm cannot be assessed by measuring solely its individual recommendation quality. Rather, it should be evaluated in a context where other recommendation lists are already available, to account for how they complement each other. The traditional ofline evaluation protocol however does not take this into account. To address this limitation, we propose an ofline evaluation protocol for a carousel setting in which the recommendation quality of a model is measured by how much it improves upon that of an already available set of carousels. Our results indicate that under a carousel setting the ranking of the algorithms changes sometimes significantly. This work is an extended abstract of [1].
The general goal of a recommendation system is to help the users navigate the large number
of options at their disposal by suggesting a limited number of relevant results. Traditionally,
the focus of newly developed recommendation systems is to generate the best possible ranked
list of results, see [
The carousel interface layout and the way it is usually generated by video-on-demand and music streaming platforms has important characteristics that distinguish it from a single-list setup, see [11]. While a carousel layout may seem similar to a traditional merge-list ensemble, where multiple recommendation lists are combined into one, this is not the case. In a real scenario multiple constraints play a role and must be taken into account: Layout Structure: The two dimensional user interface of almost all devices is organized with multiple horizontal carousels, where each carousel is generated according to a certain (often explainable) criteria e.g., most recent, most popular, because you watched, editorially curated. Recommendation Lists: The lists shown to the users within each carousel are generated with diferent algorithms or by diferent providers and independently from each other (i.e., each algorithm or provider is not aware of the existence of the other lists or of their content). Consider for example content aggregators, which combine carousels from diferent providers, e.g., Sky, Youtube, Netflix, Prime Video, etc. Due to either business constraints or the strict real-time requirements, no single post-processing step is applied, e.g., to remove items duplicated across diferent carousels. Due to this, while each individual recommendation list does not contain duplicates, the same item may be recommended in multiple carousels.
User Behavior: The users will focus on the top-left triangle of the screen rather than exploring the carousels sequentially. This is usually called golden triangle. Furthermore, they will explore the recommendations in diferent ways according to which actions they need to perform in order to reveal them. Usually users tend to navigate more easily with simple swipes rather than repeated mouse clicks, hence their behavior, as it is known, will change according to the device (e.g., personal computer, smartphone, tablet, Smart TV).
While the traditional evaluation assesses the recommendation quality of a single recommendation model, in a carousel scenario the goal is to assess the recommendation quality of a certain layout composed of recommendation lists. Once it is possible to evaluate the overall recommendation quality of a single layout it is possible to compare diferent layouts in order to select the best one. For example, one may wish to select the optimal carousel ranking or to choose which recommendation model should generate a specific carousel.
Layout generation: The layout will contain a fixed number of carousels, or recommendation lists, of a given length. If some of the carousels are generated with recommendation models, the ifrst step is to ensure that all models are adequately optimized. Since the specific layout structure that will be shown is, in general, not known in advance each recommendation model should be optimized independently. The recommendations that will be shown to the users are the sequence of all the recommendation lists in the layout, without any centralized postprocessing. Evaluation metrics: The recommendations provided to the user will be displayed in a twodimensional pattern. A frequent simplification is to concatenate all lists in a single one and remove duplicate recommendations. While this allows to use traditional metrics (e.g., NDCG, MAP), it makes assumptions that are not consistent with a carousel layout: (i) the user explores the lists sequentially; (ii) the recommendation lists are centrally collected and postprocessed. In a real scenario we must ensure that a correct recommendation is only counted once and with the correct ranking, if it appeared multiple times in diferent positions. The correct recommendation should be counted where the user would see it first , according to the user’s navigation pattern, which requires to define a new ranking discount function to compute ranking metrics. 1
In a realistic carousel scenario, several recommendation lists (or carousels) are available
generated with diferent algorithms or editorial rules. In order to mimic this setting we include
in the evaluation 16 algorithms that are both simple, well-known and competitive [12]. The
algorithsm are: TopPopular, Global Efects, SLIM ElasticNet and SLIM BPR [ 13], EASE [14],
P3 [15] and RP3 [16], PureSVD [
Due to space reasons, we will describe in particular one experiment comparing the model
ranking for Movielens20M. The goal is to choose which model to add as the last carousel in an
interface that contains an increasing number of carousels: TopPopular, ItemKNN CF and, for
the Movielens 20M dataset, ItemKNN CBF. The models are first evaluated individually with
the traditional evaluation protocol and then with the proposed carousel evaluation protocol.
1See the full paper for a detailed description on how to compute evaluation metrics [
All recommendation lists have a length of 10 and are evaluated with NDCG. As a general trend we can see that the relative efectiveness of the models difer, resulting in changes to the ranking of the algorithms in the two evaluation modes, see Figure 1. Some models such as GlobalEfects and PureSVD are always ranked in the same position. Others, in this case all other matrix factorization algorithms, gain 2 or 3 positions. On the other hand item-based machine learning models tend to consistently lose some positions, with EASE being the worst afected and losing 4 positions. As a result, while in the individual evaluation the best algorithms are SLIM ElasticNet, UserKNN CF, SLIM BPR and EASE, in the carousel evaluation the best algorithms are UserKNN CF, SLIM ElasticNet, IALS and FunkSVD. Since the recommendation lists generated by all algorithms are identical for both evaluation procedures, the diference in the ranking lies in how those recommendations intersect. Algorithms which will tend to recommend popular items will be penalized in this carousel evaluation because popular items will already be present in the TopPopular carousel, whereas algorithms providing accurate recommendation but involving less popular items will be advantaged. These results are similar for the Netflix Prize and ContentWise Impressions datasets, although the afected models change. For example, on the Netflix Prize dataset a carousel layout with TopPopular and ItemKNN CF causes two sharp changes in ranking: EASE falls by 6 positions while FunkSVD gains 7.
The results indicate that accounting for how multiple recommendation lists complement each other can produce substantially diferent results compared to evaluating each model independently and therefore it is an aspect that should be taken into account when developing recommendation models aimed to domains that use carousel interfaces. The carousel evaluation protocol also opens new research directions, such as how to combine the strength of various models to provide the user with ever more accurate and interesting recommendations. constraints, in: A. Teredesai, V. Kumar, Y. Li, R. Rosales, E. Terzi, G. Karypis (Eds.), Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2019, Anchorage, AK, USA, August 4-8, 2019, ACM, 2019, pp. 3153– 3161. URL: https://doi.org/10.1145/3292500.3330675. doi:10.1145/3292500.3330675. [11] N. Felicioni, M. Ferrari Dacrema, P. Cremonesi, A methodology for the ofline evaluation of recommender systems in a user interface with multiple carousels, in: J. Masthof, E. Herder, N. Tintarev, M. Tkalcic (Eds.), Adjunct Publication of the 29th ACM Conference on User Modeling, Adaptation and Personalization, UMAP 2021, Utrecht, The Netherlands, June 21-25, 2021, ACM, 2021, pp. 10–15. URL: https://doi.org/10.1145/3450614.3461680. doi:10.1145/3450614.3461680. [12] M. Ferrari Dacrema, S. Boglio, P. Cremonesi, D. Jannach, A troubling analysis of reproducibility and progress in recommender systems research, ACM Trans. Inf. Syst. 39 (2021).
URL: https://doi.org/10.1145/3434185. doi:10.1145/3434185. [13] X. Ning, G. Karypis, SLIM: Sparse linear methods for top-n recommender systems, in: Proceedings of the 11th IEEE International Conference on Data Mining (ICDM ’11), 2011, pp. 497–506. [14] H. Steck, Embarrassingly shallow autoencoders for sparse data, in: L. Liu, R. W. White, A. Mantrach, F. Silvestri, J. J. McAuley, R. Baeza-Yates, L. Zia (Eds.), The World Wide Web Conference, WWW 2019, San Francisco, CA, USA, May 13-17, 2019, ACM, 2019, pp. 3251– 3257. URL: https://doi.org/10.1145/3308558.3313710. doi:10.1145/3308558.3313710. [15] C. Cooper, S. Lee, T. Radzik, Y. Siantos, Random walks in recommender systems: exact computation and simulations, in: C. Chung, A. Z. Broder, K. Shim, T. Suel (Eds.), 23rd International World Wide Web Conference, WWW ’14, Seoul, Republic of Korea, April 7-11, 2014, Companion Volume, ACM, 2014, pp. 811–816. URL: https://doi.org/10.1145/ 2567948.2579244. doi:10.1145/2567948.2579244. [16] B. Paudel, F. Christofel, C. Newell, A. Bernstein, Updatable, accurate, diverse, and scalable recommendations for interactive applications, ACM Trans. Interact. Intell. Syst. 7 (2017) 1:1–1:34. URL: https://doi.org/10.1145/2955101. doi:10.1145/2955101. [17] A. Cichocki, A. H. Phan, Fast local algorithms for large scale nonnegative matrix and tensor factorizations, IEICE Trans. Fundam. Electron. Commun. Comput. Sci. 92-A (2009) 708–721.
URL: https://doi.org/10.1587/transfun.E92.A.708. doi:10.1587/transfun.E92.A.708. [18] S. Rendle, C. Freudenthaler, Z. Gantner, L. Schmidt-Thieme, BPR: bayesian personalized ranking from implicit feedback, in: J. A. Bilmes, A. Y. Ng (Eds.), UAI 2009, Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence, Montreal, QC, Canada, June 18-21, 2009, AUAI Press, 2009, pp. 452–461. URL: https://dslpitt.org/uai/ displayArticleDetails.jsp?mmnu=1&smnu=2&article_id=1630&proceeding_id=25. [19] Y. Hu, Y. Koren, C. Volinsky, Collaborative filtering for implicit feedback datasets, in: Proceedings of the 8th IEEE International Conference on Data Mining (ICDM 2008), December 15-19, 2008, Pisa, Italy, IEEE Computer Society, 2008, pp. 263–272. URL: https: //doi.org/10.1109/ICDM.2008.22. doi:10.1109/ICDM.2008.22. [20] B. Mobasher, X. Jin, Y. Zhou, Semantically enhanced collaborative filtering on the web, in: B. Berendt, A. Hotho, D. Mladenic, M. van Someren, M. Spiliopoulou, G. Stumme (Eds.), Web Mining: From Web to Semantic Web, First European Web Mining Forum, EMWF 2003, Cavtat-Dubrovnik, Croatia, September 22, 2003, Revised Selected and Invited Papers, volume 3209 of Lecture Notes in Computer Science, Springer, 2003, pp. 57–76. URL: https://doi.org/10.1007/978-3-540-30123-3_4. doi:10.1007/978-3-540-30123-3\_4. [21] F. M. Harper, J. A. Konstan, The movielens datasets: History and context, ACM Trans.
Interact. Intell. Syst. 5 (2016) 19:1–19:19. URL: https://doi.org/10.1145/2827872. doi:10. 1145/2827872. [22] J. Bennett, S. Lanning, et al., The netflix prize, in: Proceedings of KDD cup and workshop, volume 2007, New York, 2007, p. 35.