Thus far, in most of the user experiments conducted in the area of recommender systems, the respective system is considered as an isolated component, i.e., participants can only interact with the recommender that is under investigation. This fails to recognize the situation of users in real-world settings, where the recommender usually represents only one part of a greater system, with many other options for users to ifnd suitable items than using the mechanisms that are part of the recommender, e.g., liking, rating, or critiquing. For example, in current web applications, users can often choose from a wide range of decision aids, from text-based search over faceted filtering to intelligent conversational agents. This variety of methods, which may equally support users in their decision making, raises the question of whether the current practice in recommender evaluation is suficient to fully capture the user experience. In this position paper, we discuss the need to take a broader perspective in future evaluations of recommender systems, and raise awareness for evaluation methods which we think may help to achieve this goal, but have not yet gained the attention they deserve.
are supported in a holistic fashion: Even if multiple decision aids are available, they are often isolated, i.e., the input provided to one component hardly afects the results generated by another. For instance, when a user constrains the set of items by selecting several filter criteria, a recommender that is part of the same system will not necessarily reflect this selection in the generated recommendations, but only take the user’s implicit or explicit item feedback into account (and vice versa). Similarly, the interaction with a chatbot will usually start from nothing, ignoring any other interests or needs that have been expressed before.
This separation is dificult to understand in light of the fact that it is well known that diferent decision aids contribute diferently to the user’s progress in accomplishing typical choice and decision-making tasks [8, 9]. Recent studies on systems that provide multiple support components have confirmed that users use diferent mechanisms before settling on an item, and that this is (partly) due to personal and situational characteristics [10, 11, 12, 13]. For these reasons, there have, of course, been several calls over the years to bring the methods and their fields closer together [ 14, 15, 16, 17]. These calls, however, have largely been focused on methodological aspects, whereas it should not be overseen, that this narrow perspective is also reflected in the evaluation of the methods—regardless of whether they are tightly integrated with other decision aids, or appear in a decoupled fashion, as it is currently common.
The work presented in [18] is one of the few exceptions from information retrieval research that argues for a more holistic evaluation approach, in particular, “to think outside the (search) box.” In general, however, this aspect has, to the best of our knowledge, not yet received much attention, neither in the area of information retrieval, nor conversational user interfaces, nor recommender systems [cf. 19, 20, 1, 21]. This means that whenever user experience of recommender systems is studied in empirical experiments, participants can typically only interact with the recommendation component, i.e., the subject of the evaluation, which is often designed specifically for the purpose of the study, especially in academia. Switching to a diferent method, which participants may find more appealing depending on their personal preferences and progress in the decision-making process, is, however, not possible. Hence, there is a lack of data to indicate which method works best for which user. As a consequence, we argue that it is necessary for all kinds of decision aids “to think outside the box.”
In the following, as another motivating example, we present qualitative feedback from a user experiment on multiple decision aids that we recently conducted. Afterwards, we provide an overview of methods that may help to apply a broader perspective when evaluating recommender systems, and thus, to obtain a more accurate picture of real-world scenarios, in which these systems usually represent only one out of many available decision aids.
Because of the reasons described above, we argue that a broader perspective is required when evaluating recommender systems. Otherwise, there is the danger of continuing to follow certain paths in the development of interactive and conversational recommendation approaches (cf. the surveys in [22, 23]), without knowing whether these approaches are really what users want. To highlight that this is an issue already, we refer to a recent experiment, in which we asked participants ( = 100, 47 females, 2 non-binary, age: M = 35.36, SD = 12.60) to use diferent decision aids. For this purpose, we confronted them with one of two tasks, either related to a goal-driven or an explorative scenario. In both scenarios, participants had to find at least two suitable laptops. To accomplish the respective task, they were allowed to choose (and to switch) between: a faceted filtering component, a content-based recommender with the option to like and dislike items, a product advisor with a dialog showing a limited number of guiding questions, and a natural language chatbot implemented using Google Dialogflow .
While the study is described in more detail in [13], we here want to provide additional comments made by participants when they were asked why they did not use one of the components. For example, this was the case because they were reluctant to use chatbots. One participant stated: “I generally do not like interacting with chatbots. I feel whatever input I give to select a product, I might as well use a filter component.” In contrast to the increasing popularity of conversational agents for recommendation purposes, another participant indicated that he or she uses chatbots only when there is “a problem with the product or a payment issue, [but that a] chatbot is not required [for] browsing.” Others were even more direct in their criticism, writing: “I hate chatbots. I feel I spend a lot of time typing, and I hate the fake cheeriness of them,” or that they turn “something really simple like searching for a new laptop into a dark comedy.”
Other participants cited personal characteristics as reasons for their concerns, such as domain knowledge (“I assume the chatbot would be more useful for someone who does not know what to look for.”) or need for control (“I prefer doing things myself, I can chat to the bot when I cannot seem to find what I am looking for using other [. . . ] options.”) For the other options, however, we obtained similar feedback. For example, with respect to the advisor, one participant stated that he or she “should have tried this component, but [likes] to shop for these products using iflters.” Although it was clearly an academic study, another participant refrained from using the recommender because he or she suspected “that this component uses sponsored companies which pay money to have their products included in the recommendation section.” Also in this case, others were generally reluctant, writing: “I never use recommendations as they are never in line with what I need. It may be the bestseller for the store in question, but not for my needs.” Domain knowledge again played a role, notably in both directions, with participants stating to be “knowledgeable enough [that they] do not need recommendations,” but also that they “do not know enough about computers to give a thumbs up or thumbs down.” A lack of knowledge was also a major reason to stay away from faceted filtering , of which participants “thought you need technical understanding of laptops,” or mentioned that they “often feel overwhelmed using iflters, and generally do not know where to start in regard to buying laptops.”
All these comments suggest that users often have an idea of which decision aid to use. However, this is not necessarily the one that is ofered by a system—or is the subject of the experiment they participate in. For a holistic user-centered evaluation, this means, that the perspective is too narrow, as only the user experience with the specific recommender for a given task is addressed, ignoring the interdependencies with other methods that may exist, and may be more suitable depending on the current circumstances. Some participants explicitly indicated that they would like to “specify certain filters and let the recommendations pick only from the result set of the filters,” or to “combine components [so that] the advisor/chatbot works within the result set created by the must-have filters.” However, this is exactly what participants usually cannot do, since most experiments are strongly focused on individual decision aids. To evaluate recommender systems from a broader perspective, we therefore suggest to improve the current practice by applying the following evaluation methods, which, to date, are used only for single methods, or not at all.
Ofline experiments are well established in recommender research [24]. However, they are
increasingly criticized as they do not allow obtaining insights into the quality dimensions
that are relevant from a user perspective [
The same applies to simulation studies, which have only recently gained more attention in recommender research [27]. By simulating typical user behavior, e.g., with respect to critiquing mechanisms [28] or interaction with items over time [29], this type of experiment has shown strong potential in economic terms. With multiple decision aids, and thus, a larger design space, this will become even more important, also on a global level, e.g., to study long-term user behavior with respect to the question of when a recommender is used, or another component is perceived as more suitable and may contribute more to the user’s progress. In addition, domain and other factors such as product type (search vs. experience), product category (cheap streaming content vs. expensive high-risk products), and the given task (goal-oriented or explorative) may afect which method works best. Thus, simulation studies may be the only way to investigate how preferences evolve over time when using diferent decision aids, and to understand which interaction efects can occur between a recommender and other components—something that would never be possible with actual users.
Well-known qualitative methods from human-computer interaction research, which are
frequently used at the beginning of user-centered design processes, e.g., focus groups, interviews,
or contextual inquiries [cf. 30], are rarely used in the area of recommender systems. We argue,
however, that these techniques could be useful for obtaining insights into users’ actual needs
with respect to the interaction with such a system, in particular, when it comes to the relation to
other decision aids. In this context, it is worth noting that it has been found only recently, that
users’ mental models do not necessarily correspond to the implementations of recommender
systems, and are subject to large inter-individual diferences [ 31]. However, identifying the
understanding users have of the system behavior is considered highly important for evaluating
the impact of a recommender and improving it [32]. Accordingly, to better inform the design
of applications that embed multiple support components, it will be inevitable to explore these
models in more depth, in particular, with a focus on the users’ comprehension of possible
interactions between the components—by means of both qualitative methods such as grounded
theory [33] and quantitative approaches such as proposed in [
Once a (prototypical) recommender is implemented, questionnaire-based assessment is the
most common way of measuring the diferent qualities related to user experience. For this
purpose, well-established frameworks and questionnaires exist [e.g., 35, 36, 37], which, however,
are strongly focused on dimensions that are specific to recommender systems. On the other hand,
general usability questionnaires, e.g., SUS [
Finally, we would like to highlight two further aspects that do not receive much attention
in current recommender research: in-situ and long-term evaluation. The former is important,
among others, because questionnaires sufer from the problem that they typically require self
reflection disconnected from actual system usage, and, worse, often from consumption or
experience of the items, which has shown to influence the assessment of recommendations [
Overall, it seems important for future research in the recommender area, but also for other communities, to face the challenge of evaluating the respective methods in a context that is more similar to real-world settings, where decision aids rarely stand on their own. In this position paper, we explained why we think this way, and outlined how this challenge may be addressed. By this means, we hope to raise awareness that contemporary decision aids not only need to be brought together from a methodological perspective, but that a broader perspective is required when evaluating the methods. Of course, this may open up new issues. For instance, the more holistic the evaluation, the higher efort and costs for running an experiment, which are factors that already limit many studies in academia. Moreover, the dificulty of designing an experiment and analyzing its results, but also, the number of application-specific parameters and possible confounding factors, increase with the consideration of more than one decision aid. For these reasons, it will remain important to keep in mind the specific circumstances in which most experiments take place, and not to think that a broader perspective in the evaluation of the methods automatically allows more general conclusions to be drawn.
Nevertheless, we are certain that “thinking outside the (recommendations) box” will help gain a better understanding not only of the degree to which a recommender can satisfy a user in a given situation, but, in particular, of how the interplay with other decision aids can afect the assessment of the system. In the end, this may shift the focus away from further improving decision aids that are less efective or users do not want to use for specific tasks, to those that exhibit the greatest potential for providing support at the respective stage of the decision-making process. For now, however, we hope to encourage at least a discussion about using the mentioned evaluation methods more extensively to gain more in-depth insights into the users’ understanding of and preference for recommendation components in relation to other decision aids. Of course, other methods may equally well be used, but we leave it for future work to provide more concrete suggestions on which methods to use and in which order.
Thanks to Timm Kleemann, who implemented the system for the study that is mentioned here, and contributed to this study to the same extent as the author of the present paper. The study was partially supported by the Eurostars project ACODA (grant no. 01QE1946C). [2] C. A. Gomez-Uribe, N. Hunt, The Netflix recommender system: Algorithms, business value, and innovation, ACM Transactions on Management Information Systems 6 (2015) 13:1–13:19. [3] B. Smith, G. Linden, Two decades of recommender systems at Amazon.com, IEEE Internet
Computing 21 (2017) 12–18. [4] R. Baeza-Yates, B. Ribeiro-Neto, Modern Information Retrieval, ACM, New York, NY, USA, 1999. [5] M. A. Hearst, Search User Interfaces, Cambridge University Press, Cambridge, UK, 2009. [6] K. Ramesh, S. Ravishankaran, A. Joshi, K. Chandrasekaran, A survey of design techniques for conversational agents, in: ICICCT ’17: Proceedings of the 2nd International Conference on Information, Communication and Computing Technology, Springer Singapore, Singapore, 2017, pp. 336–350. [7] A. Anand, L. Cavedon, H. Joho, M. Sanderson, B. Stein, Conversational search (Dagstuhl
Seminar 19461), Dagstuhl Reports 9 (2020) 34–83. [8] G. Häubl, V. Trifts, Consumer decision making in online shopping environments: The efects of interactive decision aids, Marketing Science 19 (2000) 4–21. [9] S. Castagnos, N. Jones, P. Pu, Recommenders’ influence on buyers’ decision process, in: RecSys ’09: Proceedings of the 3rd ACM Conference on Recommender Systems, ACM, New York, NY, USA, 2009, pp. 361–364. [10] J. Schafer, J. Humann, J. O’Donovan, T. Höllerer, Quantitative modeling of dynamic human-agent cognition, in: M. D. McNeese, E. Salas, M. R. Endsley (Eds.), Contemporary Research: Models, Methodologies, and Measures in Distributed Team Cognition, CRC Press, Boca Raton, FL, USA, 2020, pp. 137–186. [11] P. Virdi, A. D. Kalro, D. Sharma, Online decision aids: The role of decision-making styles and decision-making stages, International Journal of Retail & Distribution Management 48 (2020) 555–574. [12] T. Kleemann, M. Wagner, B. Loepp, J. Ziegler, Modeling user interaction at the convergence of filtering mechanisms, recommender algorithms and advisory components, in: Mensch & Computer 2021 – Tagungsband, ACM, New York, NY, USA, 2021, pp. 531–543. [13] T. Kleemann, B. Loepp, J. Ziegler, Towards multi-method support for product search and recommending, in: UMAP ’22: Adjunct Proceedings of the 30th ACM Conference on User Modeling, Adaptation and Personalization, ACM, New York, NY, USA, 2022, pp. 74–79. [14] H. Garcia-Molina, G. Koutrika, A. Parameswaran, Information seeking: Convergence of search, recommendations, and advertising, Communications of the ACM 54 (2011) 121–130. [15] E. H. Chi, Blurring of the boundary between interactive search and recommendation, in: IUI ’15: Proceedings of the 20th International Conference on Intelligent User Interfaces, ACM, New York, NY, USA, 2015, p. 2. [16] B. Loepp, On the convergence of intelligent decision aids, in: UCAI ’21: Proceedings of the 2nd Workshop on User-Centered Artificial Intelligence, 2021. [17] A. D. Starke, M. Lee, Unifying recommender systems and conversational user interfaces, in: CUI ’22: Proceedings of the 4th International Conference on Conversational User Interfaces, ACM, New York, NY, USA, 2022. [18] P. Clough, Evaluation: Thinking outside the (search) box, in: FIRE ’14: Proceedings of the
Forum for Information Retrieval Evaluation, ACM, New York, NY, USA, 2015, pp. 1–9. [19] D. Kelly, Methods for evaluating interactive information retrieval systems with users,
Foundations and Trends in Information Retrieval 3 (2009) 1–224. [20] C. Mulwa, S. Lawless, M. Sharp, V. Wade, The evaluation of adaptive and personalised information retrieval systems: A review, International Journal of Knowledge and Web Intelligence 2 (2011) 138–156. [21] A. B. Kocaballi, L. Laranjo, E. Coiera, Understanding and measuring user experience in conversational interfaces, Interacting with Computers 31 (2019) 192–207. [22] M. Jugovac, D. Jannach, Interacting with recommenders – Overview and research directions, ACM Transactions on Interactive Intelligent Systems 7 (2017) 10:1–10:46. [23] D. Jannach, A. Manzoor, W. Cai, L. Chen, A survey on conversational recommender systems, ACM Computing Surveys 54 (2022) 105:1–105:36. [24] A. Gunawardana, G. Shani, S. Yogev, Evaluating recommender systems, in: F. Ricci, L. Rokach, B. Shapira (Eds.), Recommender Systems Handbook, Springer, New York, NY, USA, 2022, pp. 547–601. [25] M. Rossetti, F. Stella, M. Zanker, Contrasting ofline and online results when evaluating recommendation algorithms, in: RecSys ’16: Proceedings of the 10th ACM Conference on Recommender Systems, ACM, New York, NY, USA, 2016, pp. 31–34. [26] T. Rehorek, O. Biza, R. Bartyzal, P. Kordik, I. Povalyev, O. Podstavek, Comparing ofline and online evaluation results of recommender systems, in: REVEAL ’18: Proceedings of the Workshop on Ofline Evaluation for Recommender Systems, 2018. [27] N. Hazrati, F. Ricci, Simulating users’ interactions with recommender systems, in: Adjunct Proceedings of the 30th ACM Conference on User Modeling, Adaptation and Personalization, ACM, New York, NY, USA, 2022, pp. 95–98. [28] H. Xie, D. D. Wang, Y. Rao, T.-L. Wong, L. Y. K. Raymond, L. Chen, F. L. Wang, Incorporating user experience into critiquing-based recommender systems: A collaborative approach based on compound critiquing, International Journal of Machine Learning and Cybernetics 9 (2018) 837–852. [29] J. McInerney, E. Elahi, J. Basilico, Y. Raimond, T. Jebara, Accordion: A trainable simulator for long-term interactive systems, in: RecSys ’21: Proceedings of the 15th ACM Conference on Recommender Systems, ACM, New York, NY, USA, 2021, pp. 102–113. [30] G. Cockton, Usability evaluation, The Encyclopedia of Human-Computer Interaction (2nd
Edition) (2013). [31] T. Ngo, J. Kunkel, J. Ziegler, Exploring mental models for transparent and controllable recommender systems: A qualitative study, in: UMAP ’20: Proceedings of the 28th ACM Conference on User Modeling, Adaptation and Personalization, ACM, New York, NY, USA, 2020, pp. 183–191. [32] M. M. Ghori, A. Dehpanah, J. Gemmell, H. Qahri-Saremi, B. Mobasher, Does the user have a theory of the recommender? A grounded theory study, in: Adjunct Proceedings of the 30th ACM Conference on User Modeling, Adaptation and Personalization, ACM, New York, NY, USA, 2022, pp. 167–174. [33] J. Corbin, A. Strauss, Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory, 3 ed., Sage Publications, Inc., Thousand Oaks, CA, USA, 2008. USA, 2020, pp. 173–182. [47] Y. Zhong, T. L. S. Menezes, V. Kumar, Q. Zhao, F. M. Harper, A field study of related video recommendations: Newest, most similar, or most relevant?, in: Proceedings of the 12th ACM Conference on Recommender Systems, ACM, New York, NY, USA, 2018, pp. 274–278. [48] Y. Liang, M. C. Willemsen, Exploring the longitudinal efects of nudging on users’ music genre exploration behavior and listening preferences, in: RecSys ’22: Proceedings of the 16th ACM Conference on Recommender Systems, ACM, New York, NY, USA, to appear.