What if? Interaction with Recommendations Martin Zürn Daniel Buschek Malin Eiband daniel.buschek@uni-bayreuth.de martin.zuern@campus.lmu.de Research Group HCI + AI, Department of Computer malin.eiband@ifi.lmu.de Science, University of Bayreuth LMU Munich Bayreuth, Germany Munich, Germany ABSTRACT To support users, the HCI research community has worked on Showing users recommended content has become a prominent way several approaches. One of them are so-called What if? questions, of integrating algorithmic decision-making in everyday intelligent which allow users to explore, investigate and question algorithmic applications (e.g. recommendations of films, music, news, routes). decision-making. However, in comparison to work that focuses on In this context, the research community has identified What if? deriving explicit explanations for system decisions (e.g. [14]), con- questions as an approach for users to investigate and question such crete design and interaction solutions for this exploratory approach recommendations – yet many current applications seem limited in in everyday use are still in their infancy. Notable exceptions include practically supporting this. We present a set of example GUIs and work by Lim and Dey [13] who built a What if? experimentation UI interaction techniques currently used in everyday recommendation that allows users to set specific input sensor values and observe the systems in practice (e.g. Grammarly, Apple Music, Google Maps). respective system prediction for those values. Another example has Based on these example cases, we discuss possible UI extensions been presented by Nguyen et al. [15] who introduced interactive to explicitly support What if? interactions. From our analysis and sliders to adjust model parameters and observe the resulting change reflection emerges the general approach of treating decision vari- in the system output. ables as a “first-class citizen” in UIs: We propose to 1) represent a In this paper, we argue that What if? interaction with recom- recommended item’s decision variables in the user interface (and mender systems offers rich – and till now underexplored – oppor- not just the item itself), and 2) to enable direct manipulation of tunities for user support: What if? exploration may 1) allow users these decision variables for What if? explorations. to understand the system decision-making in an implicit way with- out the need for specifically crafted explanations, as argued in [5], CCS CONCEPTS and in this way exert their “right to an explanation” as part of the European Union General Data Protection Regulation (GDPR) [17]. • Computer systems organization → Embedded systems; Re- Moreover, it may 2) let them become aware of system errors and, dundancy; Robotics; • Networks → Network reliability. given options for feedback and correction, improve future recom- mendation, and thus 3) foster overall control of the system. KEYWORDS This is in line with calls for more expressive feedback and cor- intelligent systems, recommender systems, explainability, intelli- rection and fine-grained control options [6] in intelligent systems gibility, interpretability, end-user debugging, interactive machine as well as more interactive system explanation [1, 6]. learning To provide inspiration for concrete design solutions, we present ACM Reference Format: a set of example GUIs and interaction techniques currently used in Martin Zürn, Malin Eiband, and Daniel Buschek. 2020. What if? Interaction everyday recommendation systems in practice, such as Grammarly, with Recommendations. In Proceedings of the IUI workshop on Explainable Apple Music, Google Maps, and the like. Based on these example Smart Systems and Algorithmic Transparency in Emerging Technologies (ExSS- cases, we discuss possible UI extensions to explicitly support What ATEC’20). Cagliari, Italy, 4 pages. if? interactions. From our analysis and reflection emerges the gen- eral approach of treating decision variables as a “first-class citizen” 1 INTRODUCTION in UIs: We propose to 1) represent a recommended item’s decision Artificial intelligence has been integrated into many everyday end- variables in the user interface (and not just the item itself ), and 2) user products and services to improve the user experience and to to enable direct manipulation of these decision variables for What help users navigate an ever-increasing amount of data. To this aim, if? explorations. many of these systems show users recommended content like films, music, routes and news based on complex algorithmic inferences. 2 APPLICATION EXAMPLES To date, the algorithmic decision-making process and the decision In the following section we first present for inspiration some se- variables leading to a recommendation are often not accessible to lected, popular examples of interactions that can be interpreted users or hidden in the user interface, making it difficult for users to as What if? interactions and that are well established and already navigate this “inferred world” [4, 16]. familiar to users. For this purpose, we identified prominent use cases of recommender systems from different domains, such as ExSS-ATEC’20, March 2020, Cagliari, Italy shopping, entertainment, news, location-based services, social me- Copyright ©2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). dia, fitness and health, as well as typical decision variables used in these applications. ExSS-ATEC’20, March 2020, Cagliari, Italy Zürn, Eiband and Buschek We then describe other existing interactions outside the What if? 2.2 Interactions applied to What if? context, which were not originally designed for this purpose nor are In this section, we consider interactions that are not currently used currently used in this context – but which we argue are promising specifically for What if? exploration, but which we argue appear candidates for use in a What if? setting for recommender systems. to be promising candidates in this respect. The examples that we deem most relevant will now be presented Specifically, we present feature weighting because it is applicable briefly. Please note that this is not a comprehensive overview, but a to any kind of recommender system, a temporal component in form collection meant to inspire reflections towards concrete UI elements of a time axis since many recommender decisions change over time that support What if? exploration. (both through new content and changing user profiles), as well as virtual personas and emulation, as these allow for interaction with 2.1 What if? interactions in-use as of today and manipulation of physical objects. Here we present three example interactions, which already support We first look at how these interactions are currently used and What if? exploration. These examples cover “classic” item recom- then transfer them to the context of intelligent user interfaces by mendation (e.g. restaurants), recommendation for productivity (e.g. describing a scenario in which they could be used for What if? routing) as well as recommendation for creativity (e.g. text). exploration. 2.1.1 Location: Foursquare. The service Foursquare provides rec- 2.2.1 Feature weighting: Apple Music. To alleviate the cold start ommendations for restaurants and other places based on users’ problem, users are asked about their personal preferences when location and their search and check-in history. they first set up Apple Music (see [8]). Figure 1 illustrates this While the app aims to find relevant venues at the current location, onboardig process: Users are given a number of items (first a set it also allows users to change their search location, for example in a of genres, then artists) to interact with. Each item is displayed as map view. This gives them the option to discover recommendations a bubble, and users can click on it to increase its size (and thus they would receive if they were at a location different from their its relevance for system decision-making) or remove it altogether. current one (i.e. What if I would be at a different location?). This allows for a playful interaction with the weighting of different items of a given set. 2.1.2 Routing: Google Maps. Google Maps not only allows users to In the context of What if?, this interaction could be used to view maps and search for addresses and places, but also to generate remove or weight certain features more or less to observe the effect a route from A to B using different means of transport. For car on the model result. More concretely, in recommender systems that routes, Google Maps offers two different dimensions that can be recommend further products based on characteristics of a product, explored with What if?, namely time and path of the route. certain characteristics could be weighted more strongly by means By default, Google Maps assumes that users start at the current of this interaction, while others that users do not consider relevant time, and includes current traffic information about the route (such can be removed (i.e. What if I would change the importance of factor as traffic jams, closures, etc.) and estimated travel time. However, X?). users can also specify an individual time for departure or arrival, and Google Maps will then provide an estimated travel time for 2.2.2 Time axis: MacOS. Time Machine is a built-in backup solution that time (i.e. What if I would start at a different time?). in MacOS and allows the user to view a specific document in all Another dimension is the path of the route. Google Maps pro- available backup states. As soon as Time Machine is started for a vides a route suggestion and usually displays alternative routes document, the states are displayed piled up like a stack of cards, greyed out on the map. In addition to the suggestions, the user can allowing the user to move through the versions along the time also customise the route by clicking on a point on the route and axis [2] (see Figure 2). then freely moving it on the map (i.e. What if I would take a different Since user preferences may change over time, a time axis-like path?). interaction could enable users to apply the status of a recommenda- The route length and the travel time are always displayed, so tion model from the past to today’s inventory. In this way, a “time that users can easily compare different routing suggestions. travel” would reveal system learning over time (i.e. What if today’s inventory is recommended using my profile from 10 years ago?). 2.1.3 Text: Grammarly. Grammarly is an advanced spell checker Moreover, this interaction might be applicable for content which that attempts to improve the quality of writing. In addition to inherently changes over time, such as news or social media posts. providing traditional spell checking, it suggests alternative words The time dimension of the model could also be fixed here, and and phrases that may better match a particular context and target instead the time dimension of the inventory could be altered (i.e. audience. What if music from 10+ years ago would be recommended today?). For this, users can choose different options for the variables audience, formality and domain. Based on this choice, the system 2.2.3 Virtual persona: The Sims 4 & Memojis. In the game The provides different suggestions for correction and improvement Sims 4 Electronic Arts Inc. [7] players create virtual characters of the text. While a user might write for a specific audience and which lives they direct. During this process, players have various domain, the settings can be changed with one click. Thus, with possibilities to edit character traits and appearance in great detail this type of What if? exploration, the user can see the influence – from the width of the nostrils to the distance between the eyes. of these variables on the system recommendations (i.e. What if I A similar concept can be found in the Memoji feature on iOS [3], would change the audience?). where users can create their lookalike as an emoji. What if? Interaction with Recommendations ExSS-ATEC’20, March 2020, Cagliari, Italy Figure 3: Car Emulator allows to create and manipulate an object’s state [10]. any state of a vehicle for software testing (e.g. open/close doors, lights on/off, etc.). Such an emulator-based UI approach might be used also to let users manipulate system decision variables in recommendation systems. For example, users could change product characteristics Figure 1: As part of Apple Music’s onboarding, users weight such as colour or brand – and see to what extent system recommen- different artists and genres so that the system gets to know dations change. This could not only be applied to the currently rec- their personal preferences. ommended product, but also to past recommendations (i.e. What if this product would have different characteristics?). 3 DISCUSSION The examples presented in the last section are meant to provide inspiration for our community to work towards a richer UI design vocabulary for What if? exploration in recommender systems – in particular one based on direct manipulation of decision vari- ables. In this section, we summarise general desiderata for such a design vocabulary. Moreover, we discuss overarching implications of a What if? approach to supporting users in interaction with recommendations. 3.1 Conceptual view: “Recommendation item” Figure 2: Time Machine allows users to navigate through the includes its key decision variables history of a data item. Designing for What if? raises the basic question of how we define a “recommendation”. In this paper, we suggest a new comprehensive notion: A recommendation comprises not only the recommended Transferred to the context of What if? interaction, the appear- content itself (e.g. a film, product, etc.) but also the key decision ance of such a virtual persona could be used to visualise a specific variables. This view highlights that design solutions for What if? user model. Allowing users to change their persona (e.g. adapt could – and should – not just engage users in exploration with the gender, age or interests) could give them the option to explore recommended content itself, but also with the decision variables system recommendations for other user models. This could be par- based on which this recommendation was given. Casting these ticularly insightful in recommendation systems where the physical variables as an inherent part of the recommendation highlights appearance is an important factor, e.g. in dating applications such their need to be presented in the UI. More concretely, this would as Tinder or personalised fitness apps such as Freeletics (i.e. What if enable new direct manipulation interactions for users to influence I would change my physical shape?). the decision-making process in different dimensions or at different 2.2.4 Emulation: Car Emulator. Car Emulator [9, 10], shown in stages, not only by providing feedback via star ratings or giving a Figure 3, is an application which lets developers reproduce almost “thumbs up” or “thumbs down”. ExSS-ATEC’20, March 2020, Cagliari, Italy Zürn, Eiband and Buschek 3.2 Design solution principle: Rich direct Decennial Aarhus Conference on Critical Alternatives (Aarhus, Denmark) (CA ’15). Aarhus University Press, 169–180. https://doi.org/10.7146/aahcc.v1i1.21197 manipulation of recommendations [5] Malin Eiband, Charlotte Anlauff, Tim Ordenewitz, Martin Zürn, and Heinrich Most notably, the presented examples foster direct manipulation Hussmann. 2019. Understanding Algorithms Through Exploration: Supporting Knowledge Acquisition in Primary Tasks. In Proceedings of Mensch Und Computer – and through that a sense of immediateness: For example, when 2019 (Hamburg, Germany) (MuC’19). ACM, New York, NY, USA, 127–136. https: assigning weights to artists and genres in Apple Music, users do //doi.org/10.1145/3340764.3340772 [6] Malin Eiband, Sarah Theres Völkel, Daniel Buschek, Sophia Cook, and Heinrich not have to look through a list of filter criteria but instead can im- Hussmann. 2019. When People and Algorithms Meet: User-reported Problems mediately grasp and experiment with the system decision-making in Intelligent Everyday Applications. In Proceedings of the 24th International in a playful way. Similarly, in Google Maps, customising a route is Conference on Intelligent User Interfaces (Marina del Ray, California) (IUI ’19). ACM, New York, NY, USA, 96–106. https://doi.org/10.1145/3301275.3302262 done via dragging the suggested route on the map. Moreover, Time [7] Electronic Arts Inc. [n.d.]. Create Your Sim Demo - The Sims 4 Official Site. Machine and Car Emulator support direct manipulation through Retrieved 2019-12-18 from https://www.ea.com/games/the-sims/the-sims-4/pc/ affordances (i.e. through a card stack and a real-life object). create-a-sim-demo [8] GottaBeMobile . 2015. Apple Music Setup - YouTube. Retrieved 2019-12-18 from We argue that direct manipulation of recommendations and https://youtu.be/ydYah_BXt9A?t=30 decision variables in combination with affordances form promis- [9] High-Mobility GmbH. 2017. How To Use HIGH MOBILITY’s Car Emulator. HIGH MOBILITY Magazine (2017). Retrieved 2019-12-18 from https://medium.com/ ing research avenues towards finding ways to integrate What if? high-mobility/how-to-use-high-mobilitys-car-emulator-244200558a8a exploration into everyday applications. [10] High-Mobility GmbH. 2018. Porsche NEXT OI Competition: Car Emulator - YouTube. Retrieved 2019-12-18 from https://youtu.be/y_-Wp7uETBA [11] Todd Kulesza, Margaret Burnett, Weng-Keen Wong, and Simone Stumpf. 2015. 3.3 Challenges Principles of Explanatory Debugging to Personalize Interactive Machine Learning. While many expert systems for interactive machine learning such In Proceedings of the 20th International Conference on Intelligent User Interfaces (Atlanta, Georgia, USA) (IUI ’15). ACM, New York, NY, USA, 126–137. https: as Tensorboard1 already support What if? interactions, designing //doi.org/10.1145/2678025.2701399 them for laypersons in everyday use is challenging. These chal- [12] Brian Y. Lim and Anind K. Dey. 2011. Design of an Intelligible Mobile Context- aware Application. In Proceedings of the 13th International Conference on Human lenges include designing exploration in a way which does not dis- Computer Interaction with Mobile Devices and Services (Stockholm, Sweden) (Mo- tract from the actual task users want to do [5], and which does bileHCI ’11). ACM, New York, NY, USA, 157–166. https://doi.org/10.1145/2037373. not overwhelm users [11, 12]. Moreover, one fundamental premise 2037399 [13] Brian Y. Lim, Anind K. Dey, and Daniel Avrahami. 2009. Why and Why Not for designing for What if? in this context should be that users are Explanations Improve the Intelligibility of Context-aware Intelligent Systems. not always interested in exploration – the HCI research community Proceedings of the 27th International Conference on Human factors in Computing should therefore think about how What if? can be seamlessly inte- Systems, 2119–2128. https://doi.org/10.1145/1518701.1519023 [14] Tim Miller. 2019. Explanation in Artificial Intelligence: Insights from the Social grated into the interface, for example via on-demand approaches. Sciences. Artificial Intelligence 267 (2019), 1 – 38. https://doi.org/10.1016/j.artint. Finally, while we argue that What if? might foster overall (feeling 2018.07.007 [15] An T. Nguyen, Matthew Lease, and Byron C. Wallace. 2019. Mash: Software of) control of the system, future work should explore if and how Tools for Developing Interactive and Transparent Machine Learning Systems. often people actually make use of this possibility. In Explainable Smart Systems Workshop at the 24th International Conference on Intelligent User Interfaces (IUI ’19). [16] Advait Sarkar. 2015. Confidence, Command, Complexity: Metamodels for Struc- 4 CONCLUSION tured Interaction with Machine Intelligence. In Proceedings of the 26th Annual We presented a set of example UIs from various current applica- Conference of the Psychology of Programming Interest Group (PPIG 2015). 23–36. [17] The European Parliament and Council. 2016. Regulation (EU) 2016/679 of the tions to inspire working towards a richer UI design vocabulary for European Parliament and of the Council of 27 April 2016 on the protection interactions with recommendations, in particular supporting What of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data if? exploration. Our suggestion for a promising design solution Protection Regulation). Official Journal of the European Union (2016). principle here has two key takeaways: First, we propose to conceptually view key decision variables as an integral part of any “recommendation item” shown to the user. In other words, such decision variables need to have a representation in the UI. Second, we propose to then design direct manipulation interactions for these representations of the decision variables. Here, our set of examples provides ideas for possible starting points to work towards concrete UI designs. REFERENCES [1] Ashraf Abdul, Jo Vermeulen, Danding Wang, Brian Y. Lim, and Mohan Kankan- halli. 2018. Trends and Trajectories for Explainable, Accountable and Intelligible Systems: An HCI Research Agenda. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18). ACM, New York, NY, USA, Article 582, 18 pages. https://doi.org/10.1145/3173574.3174156 [2] Apple, Inc. 2018. Use Time Machine to restore deleted files or older files - Apple Support. Retrieved 2019-12-18 from https://support.apple.com/en-us/HT209152 [3] Apple, Inc. 2019. How to use Memoji on your iPhone or iPad Pro - Apple Support. Retrieved 2019-12-18 from https://support.apple.com/en-us/HT208986 [4] Alan F. Blackwell. 2015. Interacting with an Inferred World: The Challenge of Machine Learning for Humane Computer Interaction. In Proceedings of The Fifth 1 https://www.tensorflow.org/tensorboard, accessed 17 December 2019.