Ofline data-driven evaluation is considered a low-cost and more accessible alternative for the online empirical method of assessing the quality of recommender systems. Despite their popularity and efectiveness, most data-driven approaches are unsuitable for evaluating interactive recommender systems. In this paper, we attempt to address this issue by simulating the user interactions with the system as a part of the evaluation process. Particularly, we demonstrate that simulated users find their desired item more eficiently when recommendations presented as a list of carousels compared to a simple ranked list.
For many years, empirical evaluation based on various kinds of user studies was the key approach
for evaluating all kinds of user-adaptive systems, i.e., interactive systems that can adapt their
behavior to individual users [
The establishment of data-driven evaluation approaches was very important for the
recommender system field. Promoted by the Netflix prize, these approaches helped to engage a large
number of researchers in the work on recommender systems and stimulated rapid progress
in the development and evaluation of recommendation algorithms. Data-driven evaluation
quickly became a gold standard in the field overshadowing the empirical evaluation approaches.
Numerous papers discussed the comparative benefits of data-driven vs. empirical evaluation
and pointed out that these studies frequently deliver diferent results [
However, the choice between data-driven and empirical evaluation approaches is currently
not available for researchers working on various interactive recommender systems [
This paper attempts to make a case for simulation-based evaluation of an interactive
recommender system. The idea of this approach is a continuous simulation of user behavior in a
target system while computing various performance metrics “on the go”. It could be applied to
relatively complex interaction scenarios as long as user behavior in these scenarios could be
modeled suficiently well. While this approach enables the application of diverse metrics that
are typically used in empirical studies, it is based on simulated rather than real users and can be
performed ofline. Simulation-based evaluation is a recognized approach for evaluating various
kinds of interactive systems [
To demonstrate the power of the simulation-based approach and its potential value for datadriven evaluation of interactive recommender systems, the paper presents a simulation-based study that compares user behavior and performance in two types of recommendations interfaces - an interface with multiple carousels (sometimes referred to as a multilist) and a traditional ranked list. This simulation-based study was enabled by the availability of user behavior models for both, the ranked list and the multilist interfaces. While the main goal of this paper was to demonstrate how a simulation-based ofline study could be organized in a recommender system context, we also pay attention to the study results, which help to explain the increasing popularity of carousel-based interfaces.
The paper is structured as follows. In Section 2, we review past research on simulation-based evaluation. In Section 3, we present an empirically grounded model of user behavior in ranked list-based and carousel-based interfaces. In Section 4, we present in detail the setup for the simulation-based study. Finally, we discuss our results in Section 5.
As mentioned in the introduction, simulation-based evaluations have been used in a number
of fields where suficiently detailed models of user behavior could be built. Simulation-based
evaluation is a recognized approach for evaluating various kinds of personalized interactive
systems from adaptive learning systems [
Early attempts to use simulations for exploring information filtering and recommender
systems were made in the first decade of 2000 [
The main obstacle in using a simulation-based approach for evaluating interactive recommender systems is the need for suficiently reliable behavior models for the realistic simulation of user behavior. While these models do not yet exist for all kinds of interactive recommender systems, user behavior in several types of interactive recommender systems is explored suficiently well to build these models. In our paper, we want to demonstrate the use of simulation-based evaluation for assessing the eficiency of carousel-based recommendation interfaces and comparing it with the traditional recommender list approach.
The goal of our simulation-based study is to compare user performance with two types of recommendations interfaces - a carousel-based multi-list and a traditional ranked list - in a typical modern recommendation context where items could be associated with multiple “interests” and users could favor several of these interests in parallel (although probably to a diferent extent and at a diferent time). Depending on the domain, these interests could have diferent semantic natures. For example, it could be movie genres (such as action movies) or research topics (such as context-aware recommendation). For uniformity, we refer to these interests as topics. Note that some recommender systems could model interests as latent categories rather than explicit semantic topics. In this paper, we focus on domains with explicitly represented interests, to separate the problem of latent interest discovery from the problems of user modeling and item ranking.
Extensive studies of user information access behavior started in the field of information retrieval
and were originally motivated by the need to improve Web search engine performance. While
“old school” information retrieval considered item relevance as the only factor determining user
decision to click on a specific result, it became increasingly evident that the position of items in
a ranked list has to be considered as well [
We define our ranked list interaction model on the basis of the cascade model. This means that we assume the user starts by examining the first item on the list and continue the examination one by one until finding the desired item or when there is no item left to examine. However, to make this interaction model consistent with the carousel interaction model, in our simulations we adapted the ranked list interaction model to a 2D context. We define the ranked list as a matrix of × recommended items, which is examined row by row (Figure 1a). This isolates the impact of 2D presentation from the impact of topic labels.
The ranked list interaction model in a 2D context defines as follows: The user starts at position (1, 1). If that item is not desired, the user proceeds to the next item (1, 2). The user examines row 1, from left to right, until the desired item is found or the end of the row is reached. If the end of the row is reached, the user moves to the item (2, 1), the first item in the next row. Then the user examines this row, from left to right, and this process continues until the desired item is found.
A recommender interface with multiple carousels ofers its users several ranked lists, each marked with a topic, in place of a single ranked list. This interface leaves the choice of the current topic of interest to the user making the recommendation process more interactive. While this interface is not very complex, evaluating it in a traditional ofline way using static metrics for its [37]. As a result, carousel-based interfaces are predominantly explored through user studies [38, 39]. On the other hand, the studies of multi-list and other 2D presentation interfaces [38, 39, 40] provided useful information for developing models of user behavior in carousel-based interfaces.
The key work on examining user behavior in 2D presentation interfaces was performed not in the area of recommender systems, but in the area of Web search as an extension of the work on click models. As the presentation of results was becoming increasingly two-dimensional, understanding user behavior when exploring 2D presentation was important to optimize novel ways to present information in 2D [41, 42, 40]. A range of user studies that frequently engaged eye-tracking brought consistent results. There was compelling evidence that the users examine the 2D presentation top-down, row-by-row, similarly to the case of a 1D ranked list. In the horizontal dimension, each row is examined as far as to ascertain whether the information is relevant or not. Given the user’s known perception of higher-ranked rows as more relevant, upper rows are usually examined more extensively (Figure 1b).
(a) The user item navigation in a single ranked list - presented in 2D.
(b) The user topic-item navigation in the carousel interaction model
The combination of these two factors created a characteristic “golden triangle” pattern in eye-tracking studies [41]. The results of studies of user behavior in search context helped to create the first 2D click models of user behavior [ 42, 40]. While no rigorous studies specifically focused on user work with carousel-based interfaces have been performed yet, the analysis of user behavior data obtained in recent studies of carousel-based interfaces [38, 39] demonstrated the same general golden triangle pattern. Based on this empirical data, we developed a model of user behavior in carousel-based recommender interfaces. This model is presented in a formal way in [43]. Below we present this model less formally to the extent it is necessary to understand how our simulation-based study was performed.
To quantify the benefit of carousels, we formalize the problem of carousel recommendation using a mathematical model, which we call a carousel interaction model. We have a matrix of × recommended items, where is the number of rows (carousels) and is the number of columns (items per carousel). Each carousel is associated with some topic, such as a movie genre. To simplify exposition, we assume that each item belongs to a single topic. We refer to the item at row ∈ [] and column ∈ [] as (, ).
The user preferences are defined by two sets of probabilities. The first is topic preferences. Specifically, ≥ 0 is the probability that the user is interested in topic , for any ∈ []. The second set is topic-conditioned item preferences. Specifically, | ≥ 0 is the conditional probability that the user is interested in item given that they desire topic , for any ∈ [] and ∈ []. We assume that ∑︀ =1 | = 1 for any topic ∈ [].
=1 = 1, and that ∑︀
Based on the above assumptions, we developed three increasingly more complex behavior models. The simplest of these models assumes a patient and focused user who continues to examine topics and items until finds the desirable item, i.e., an item that matches user preferences and interests in the given moment. This user interacts in the carousel model as follows. First, we assume that the user starts with some understanding what kind of topic (the desired topic) and item in that topic (a desired item) she wants to consume before she starts using the recommender system interface to locate such an item. To simulate this process in a personalized way, the desired topic is sampled as ∼ Cat(()=1) and the desired item is sampled as ∼ Cat((| )=1), where Cat( ) is a categorical distribution with outcome probabilities . In plain English, exactly one topic is chosen with probability , and exactly one item is chosen with probability | conditioned on that topic. An equivalent way of thinking of this process is that exactly one (, ) is chosen with probability , = |. The user seeks item (, ) as follows. They start by examining the first carousel. If its topic does not match that of , they proceed to the next carousel. The user examines all carousels, from top to bottom, until they stop at carousel . After that, the user examines the items in carousel , from left to right, until they find the desired item, in column .
There are two diferent scenarios under which the user might leave the system. The session may end after the user successfully find the desired item or because none of the items are desirable for the user and there is no more item and topic to examine. We are aware that this browsing behavior is unlikely to occur in a realistic situation due to the position bias efect [ 30]. However, we start with this simple model to highlight the diference between this and other more realistic behavioral models.
The simple model assume that the user is patiently scanning presented items until the desirable item is found - even if it requires to go through thousands of items. In the majority of cases, this assumption is not realistic. To better model a browsing behavior of an actual user, we assume that the user has limited patience for finding the desirable item. We implemented this behavior as follows. The user starts by examining the first topic or item at position (1, 1). The user exits with a probability of = 0.02 after examining either a carousel or item. Generally, users are likely to abandon the session after 50 interactions on average, when no items or topics are desirable. This is the same as the ideal setting except for exiting with probability = 0.02 upon each examination, of either a carousel or an item.
We initially assumed that the user always knew which carousel (with a genre as a topic) includes the desirable movie. However, in reality, the user might get distracted and as a result, begin browsing the wrong carousel or pass the correct carousel and miss out on finding the desired item. We consider this assumption to be an extension of the previous assumption described in Section 3.2.3.
In both ideal and distracted user settings, when the user examines an undesirable carousel, they will move to the next carousel with a probability 1. We define = 0.05 as the distraction probability. Here user moves to the next carousel with probability 1 − and starts examining items in the undesirable carousel with probability . Similarly, when the user examines a desirable carousel, they move to the next carousel with probability and start examining items in the desirable carousel with probability 1 − . Considering a user as distracted only applies in carousel interaction model. Including this assumption in the carousel interaction model allows us to capture the complexity that comes with providing additional information to the user in the form of carousel topics.
Because of lacking a large enough data set that can accurately estimate the parameters of our proposed settings, we set the values of and intuitively based on how we presume the user would behave under those settings.
We conduct a series of ofline simulation-based experiments to evaluate how our proposed carousel interaction model performs against a standard baseline (single ranked list). In this section, we discuss the details of our simulation-based study.
For our experiments, we choose the domain of movie recommendation. The choice of the domain was motivated by two reasons. First, movie recommendation is a good example of a modern context where users can have multiple interests and favor diferent interests at diferent times. Second, it is the context where carousels are currently very popular, which makes it easier to simulate realistic carousel-based recommendations. We use the MovieLens 100K Dataset [44] which consists of 100,836 ratings applied to 9,724 movies in 19 genres by 610 users. In our experiments, we only utilize the information about the user ratings and movie genres. We apply a pre-processing step to remove movies with no genres. A total number of 34 movies was removed from the dataset through this process.
We assume that the user follows three distinct browsing behavior models when seeking a movie to watch depending on whether the results are presented as a single ranked list or a set of carousels. These three models are explained in Section 3.2.2, 3.2.3 and 3.2.4 respectively.
To generate the recommendations, we consider two sets of probabilities. The topic preferences and the topic-conditioned item preferences. The preferences are computed as follows. The dataset of ratings is a set of tuples = {(, , )}=1, where is the index of the user in data point , is the index of the rated movie in data point , and is the corresponding rating. The topic-conditioned item preference reflects how representative the movie is of a genre. We compute it as the sum of all ratings of the movie over the sum of all ratings in its genre. Formally, let be the set of all movies in genre . Then for any movie ∈ , the topic-conditioned item preference of movie in genre is
∑︀=1 1{ = } .
| = ∑︀=1 1{ ∈ } We set | = 0 for any ∈/ . For any user , the topic preference reflects how much the user prefers a genre. We compute it as the sum of all ratings of the user in a given genre over all ratings by that user. Formally, the topic preference of user for genre is =
∑︀=∑1︀1={11={,=∈} } .
Having the user profile assigned to each user, we generate two sets of recommendations as follows: For the first set of recommendations for carousels, we use the topic preferences to sort them and then populate each one with movies using the topic-conditioned item preferences. This approach generates a set of carousels each representing a genre (19 carousels for 19 genres in the dataset). Each carousel contains all the movies within the representative genre. With an average of more than 475 movies in each genre, we assume that is a realistic enough scenario for the user to be able to scroll down or right and examine all items and find the desirable movie. The movie is sorted by their scores, where the score of movie is ∑︀ =1 , . Due to the sheer number of movies in the dataset, we assume that users will be able to scroll down the list to find what they are looking for. In this evaluation, the user profile and the recommendations were not afected by further user interactions and remained unchanged throughout all sessions.
We define a session as a single instance of evaluation in which the user seeks a movie (, ) from the set of recommended results, which can be displayed as a single ranked list or carousel interaction model.
The process of simulation is as follows: For each setting, we first generate two sets of recommendations (one using single ranked list and another using carousel interaction model) for every user in the dataset. Next, we ran 100 independent sessions for every user that includes selecting a genre, selecting a movie within that genre, and calculating the number of interactions required to reach that movie in both models. We consider the average value of these 100 sessions as the outcome of the experiment for a given user in a given setting.
To simulate user navigation in each session, we assume that the desired genre and a movie in that genre are realized in the mind of the user. The desired genre is sampled as ∼ Cat(()=1) and the desired movie is sampled as ∼ Cat((| )=1). This process is described in detail in Section 3.2. In each session, the user is only interested in a single genre and a single movie within that genre.
There are many ways of measuring the complexity of interacting with the recommended items in single ranked list and carousel interaction model. We employ two metrics to evaluate our proposed approach.
First, we define navigation efort as the number of examinations by users until the desired item is found. These examinations include browsing genres as carousel topics and movies as items. A lower navigation efort means less efort to find a desirable item.
Second, we define the exiting Probability which determines on average what proportion of users left the session after a certain number of interactions. For example, in Figure 3a on average, just under 50% of users in carousel interaction model exited the session with fewer than 30 interactions. The total number of interactions includes all examinations done by the user to find the desirable movie. It is important to state that the exiting Probability only can be considered as a positive metric under the ideal setting where the user continues the examination until finding the desirable items.
Unlike the ideal setting, in distracted and inpatient settings, the exiting Probability could be an indication of either satisfactory, due to finding the desirable items or unsatisfactory, due to impatience or distraction and without necessarily finding the desirable items. In our experiments, we only compare the exiting Probability under the comparable settings.
In this section, we present the results of our simulation-based evaluation. In Section 5.1 we compare the distribution of navigation efort for carousel interaction model and single ranked list models with samples from a diferent number of top movies in the dataset. In Section 5.2 we demonstrate how diferent browsing behavior afects the exiting pattern among users.
Figure 2a shows the distribution of navigation efort values for all the movies (9708) and users (610) in our dataset. The experiment was conducted under the ideal setting described in Section 3.2.2. Similarly, Figures 2b to 2d display the distribution of navigation efort values for the top 1000, 500 and 100 movies respectively. These results indicate that the carousel interaction model significantly reduced the number of required interactions to find a desirable movie. It is evident that even though the slope of the lines remains relatively steady, the higher number of (a) Samples drawn from all movies.
(b) Samples drawn from top 1000 movies. (c) Samples drawn from top 500 movies. (d) Samples drawn from top 100 movies. movies resulted in a slightly more prominent improvement from single ranked list to a carousel interaction model.
To compare the behavior of our model under more realistic settings, we visualize the average exiting probability of users after a certain number of interactions with the recommendations in Figure 3a and Figure 3b. In Figure 3a we observe a significant diference between the carousel interaction model and the single ranked list under the ideal settings. In the ideal setting, the user continues the examination until reaches the desirable item. We limit the number of interactions to 50 meaning the user would exit unsatisfied if they could not find the desired item in the ifrst 50 interactions. The higher exiting probability in carousel interaction model (blue line) shows that more users exit the system satisfied by finding their desired item. A larger spike in exiting probability on single ranked list at the end indicates a larger number of users that left without finding their desired item. It is worth noting that based on the result of this experiment, a significantly larger portion of users (just under 75%) exit the system after finding their desirable. This number drops to close to 30% when recommendations are presented in the form of a (a) Cumulative Exiting Probability under ideal condition.
(b) Cumulative Exiting Probability for distracted and impatient user. ranked list. The exiting behaviour of the simulated impatient and distracted users is displayed in Figure 3b.
Although the gap between the probability of exiting the session in carousel interaction model and single ranked list models is less significant, the former still performs better. Comparing the Impatient and distracted exiting behavior indicates a non-significant diference between the two settings but shows a slight decrease in performance in carousel interaction model. Unlike Figure 3a where the exiting probability promote a positive event (satisfaction of finding the desirable item), in Figure 3b there can be also adverse reasons for exiting a session, such as "impatience" and "distraction". Therefore, the improvement of this metric compared to the ideal setting is not necessarily a positive sign. Despite this, since we compare carousel interaction model and single ranked list in Figure 3b under the same setting where the probability of "impatience" is the same, an improvement in the metric likely signal a positive event.
This paper makes a case for using simulation-based approaches for ofline data driven evaluation of interactive recommender systems. We believe that the ability to use ofline evaluation will benefit the research on interactive recommender systems in the save way as data-driven ofline studies boosted the work on recommendation algorithms. To demonstrate the value and the opportunities for simulation-based evaluation we presented and example of a simulation-based study, which compared user behavior in a ranked list and multi-list interfaces. This study was enabled by empirically based models of user interaction with these two kinds of interfaces. The results of our comparative evaluation demonstrate the navigational superiority of the carousel-based interface and uncovers the reasons for its increasing popularity. While this result is important by itself, in the context of our paper it serves as an illustrations of interesting ifndings that could be made by using simulation-based evaluation.
The case presented in the paper could be considered as relatively simple - the benefits of carousel-based interfaces over simple ranked lists might be intuitively evident. Yet even in this simple case only with a thorough study these benefits can be measured and quantified. A simulation-based approach enables us to quantify diferences through a fair comparative study and without engaging expensive human subjects. In a similar way, this approach could be applied to more complex scenarios where simple intuition will not sufice. We hope that this paper will help to promote simulation-based evaluation of all kinds of interactive recommenders. In our future work, we plan to continue exploration of carousel-based interfaces from the prospect of human-AI collaboration, explore more powerful approaches to ranking items within each topic-based carousel, and compare these approaches through simulation-based and empirical studies.
We would like to thank Branislav Kveton, for his great help with formalizing the ranked list and carousel interactions using mathematical models. His willingness to give his time so generously has been very much appreciated. web page viewing behavior: an eye-tracking study, in: ETRA’2004: Proceedings of the symposium on Eye tracking research and applications, ACM, 2004, pp. 147–154. [26] Z. Guan, E. Cutrell, What are you looking for?: An eye-tracking study of information usage in web search, in: CHI ’07: ACM SIGCHI conference on human factors in computing systems, 2007, pp. 407–416. [27] M. Keane, M. O’Brien, B. Smyth, Are people biased in their use of search engines?, Commun.
ACM 51 (2008) 49–52. [28] E. Agichtein, E. Brill, S. Dumais, R. Ragno, Learning user interaction models for predicting web search result preferences, in: 29th Annual International ACM SIGIR Conference, 2006, pp. 3–10. [29] M. Richardson, E. Dominowska, R. Ragno, Predicting clicks: Estimating the click-through rate for new ads, in: 16th International Conference on World Wide Web, 2007, pp. 521–530. [30] N. Craswell, O. Zoeter, M. Taylor, B. Ramsey, An experimental comparison of click positionbias models, in: Proceedings of the 1st ACM International Conference on Web Search and Data Mining, 2008, pp. 87–94. [31] F. Guo, C. Liu, A. Kannan, T. Minka, M. Taylor, Y. M. Wang, C. Faloutsos, Click chain model in web search, in: Proceedings of the 18th International Conference on World Wide Web, 2009, pp. 11–20. [32] O. Chapelle, Y. Zhang, A dynamic Bayesian network click model for web search ranking, in: Proceedings of the 18th International Conference on World Wide Web, 2009, pp. 1–10. [33] S. Li, Y. Abbasi-Yadkori, B. Kveton, S. Muthukrishnan, V. Vinay, Z. Wen, Ofline evaluation of ranking policies with click models, in: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2018, pp. 1685–1694. [34] A. Chuklin, I. Markov, M. de Rijke, Click Models for Web Search, Morgan & Claypool
Publishers, 2015. [35] B. Kveton, C. Szepesvari, Z. Wen, A. Ashkan, Cascading bandits: Learning to rank in the cascade model, in: International Conference on Machine Learning, 2015. [36] R. Combes, S. Magureanu, A. Proutiere, C. Laroche, Learning to rank: Regret lower bounds and eficient algorithms, in: Proceedings of the 2015 ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems, 2015. [37] N. Felicioni, M. F. Dacrema, P. Cremonesi, A methodology for the ofline evaluation of recommender systems in a user interface with multiple carousels, in: Adjunct Proceedings of the 29th ACM Conference on User Modeling, Adaptation and Personalization, ACM, 2021, pp. 10–15. [38] D. Jannach, M. Jesse, M. Jugovac, C. Trattner, Exploring multi-list user interfaces for similar-item recommendations, in: Proceedings of the 29th ACM Conference on User Modeling, Adaptation and Personalization, ACM, 2021, pp. 224–228. [39] A. Starke, E. Asotic, C. Trattner, “Serving Each User”: Supporting diferent eating goals through a multi-list recommender interface, in: Fifteenth ACM Conference on Recommender Systems, ACM, 2021, pp. 124–132. [40] L. Wu, M. Grbovic, J. Li, Toward user engagement optimization in 2d presentation, in: Proceedings of the 14th ACM International Conference on Web Search and Data Mining, ACM, 2021, pp. 1047–1055. [41] F. Chierichetti, R. Kumar, P. Raghavan, Optimizing two-dimensional search results presentation, in: Proceedings of the 4th ACM International Conference on Web Search and Data Mining - WSDM '11, ACM Press, 2011, pp. 257–266. [42] Y. Wang, D. Yin, L. Jie, P. Wang, M. Yamada, Y. Chang, Q. Mei, Beyond ranking: Optimizing whole-page presentation, in: Proceedings of the Ninth ACM International Conference on Web Search and Data Mining, ACM, 2016, pp. 103–112. [43] B. Rahdari, B. Kveton, P. Brusilovsky, The magic of carousels: Single vs. multi-list recommender systems, in: 33rd ACM Conference on Hypertext and Social Media, 2022, pp. 166–174. [44] F. M. Harper, J. A. Konstan, The movielens datasets: History and context, ACM Transactions on Interactive Intelligent Systems 5 (2015) 1–19.