=Paper=
{{Paper
|id=Vol-2582/paper8
|storemode=property
|title=What if? Interaction with Recommendations
|pdfUrl=https://ceur-ws.org/Vol-2582/paper8.pdf
|volume=Vol-2582
|authors=Martin Zürn,Malin Eiband,Daniel Buschek
|dblpUrl=https://dblp.org/rec/conf/iui/ZurnEB20
}}
==What if? Interaction with Recommendations==
https://ceur-ws.org/Vol-2582/paper8.pdf
What if? Interaction with Recommendations
Martin Zürn Daniel Buschek
Malin Eiband daniel.buschek@uni-bayreuth.de
martin.zuern@campus.lmu.de Research Group HCI + AI, Department of Computer
malin.eiband@ifi.lmu.de Science, University of Bayreuth
LMU Munich Bayreuth, Germany
Munich, Germany
ABSTRACT To support users, the HCI research community has worked on
Showing users recommended content has become a prominent way several approaches. One of them are so-called What if? questions,
of integrating algorithmic decision-making in everyday intelligent which allow users to explore, investigate and question algorithmic
applications (e.g. recommendations of films, music, news, routes). decision-making. However, in comparison to work that focuses on
In this context, the research community has identified What if? deriving explicit explanations for system decisions (e.g. [14]), con-
questions as an approach for users to investigate and question such crete design and interaction solutions for this exploratory approach
recommendations – yet many current applications seem limited in in everyday use are still in their infancy. Notable exceptions include
practically supporting this. We present a set of example GUIs and work by Lim and Dey [13] who built a What if? experimentation UI
interaction techniques currently used in everyday recommendation that allows users to set specific input sensor values and observe the
systems in practice (e.g. Grammarly, Apple Music, Google Maps). respective system prediction for those values. Another example has
Based on these example cases, we discuss possible UI extensions been presented by Nguyen et al. [15] who introduced interactive
to explicitly support What if? interactions. From our analysis and sliders to adjust model parameters and observe the resulting change
reflection emerges the general approach of treating decision vari- in the system output.
ables as a “first-class citizen” in UIs: We propose to 1) represent a In this paper, we argue that What if? interaction with recom-
recommended item’s decision variables in the user interface (and mender systems offers rich – and till now underexplored – oppor-
not just the item itself), and 2) to enable direct manipulation of tunities for user support: What if? exploration may 1) allow users
these decision variables for What if? explorations. to understand the system decision-making in an implicit way with-
out the need for specifically crafted explanations, as argued in [5],
CCS CONCEPTS and in this way exert their “right to an explanation” as part of the
European Union General Data Protection Regulation (GDPR) [17].
• Computer systems organization → Embedded systems; Re-
Moreover, it may 2) let them become aware of system errors and,
dundancy; Robotics; • Networks → Network reliability.
given options for feedback and correction, improve future recom-
mendation, and thus 3) foster overall control of the system.
KEYWORDS This is in line with calls for more expressive feedback and cor-
intelligent systems, recommender systems, explainability, intelli- rection and fine-grained control options [6] in intelligent systems
gibility, interpretability, end-user debugging, interactive machine as well as more interactive system explanation [1, 6].
learning To provide inspiration for concrete design solutions, we present
ACM Reference Format: a set of example GUIs and interaction techniques currently used in
Martin Zürn, Malin Eiband, and Daniel Buschek. 2020. What if? Interaction everyday recommendation systems in practice, such as Grammarly,
with Recommendations. In Proceedings of the IUI workshop on Explainable Apple Music, Google Maps, and the like. Based on these example
Smart Systems and Algorithmic Transparency in Emerging Technologies (ExSS- cases, we discuss possible UI extensions to explicitly support What
ATEC’20). Cagliari, Italy, 4 pages. if? interactions. From our analysis and reflection emerges the gen-
eral approach of treating decision variables as a “first-class citizen”
1 INTRODUCTION in UIs: We propose to 1) represent a recommended item’s decision
Artificial intelligence has been integrated into many everyday end- variables in the user interface (and not just the item itself ), and 2)
user products and services to improve the user experience and to to enable direct manipulation of these decision variables for What
help users navigate an ever-increasing amount of data. To this aim, if? explorations.
many of these systems show users recommended content like films,
music, routes and news based on complex algorithmic inferences. 2 APPLICATION EXAMPLES
To date, the algorithmic decision-making process and the decision In the following section we first present for inspiration some se-
variables leading to a recommendation are often not accessible to lected, popular examples of interactions that can be interpreted
users or hidden in the user interface, making it difficult for users to as What if? interactions and that are well established and already
navigate this “inferred world” [4, 16]. familiar to users. For this purpose, we identified prominent use
cases of recommender systems from different domains, such as
ExSS-ATEC’20, March 2020, Cagliari, Italy shopping, entertainment, news, location-based services, social me-
Copyright ©2020 for this paper by its authors. Use permitted under Creative Commons
License Attribution 4.0 International (CC BY 4.0). dia, fitness and health, as well as typical decision variables used in
these applications.
ExSS-ATEC’20, March 2020, Cagliari, Italy Zürn, Eiband and Buschek
We then describe other existing interactions outside the What if? 2.2 Interactions applied to What if?
context, which were not originally designed for this purpose nor are In this section, we consider interactions that are not currently used
currently used in this context – but which we argue are promising specifically for What if? exploration, but which we argue appear
candidates for use in a What if? setting for recommender systems. to be promising candidates in this respect.
The examples that we deem most relevant will now be presented Specifically, we present feature weighting because it is applicable
briefly. Please note that this is not a comprehensive overview, but a to any kind of recommender system, a temporal component in form
collection meant to inspire reflections towards concrete UI elements of a time axis since many recommender decisions change over time
that support What if? exploration. (both through new content and changing user profiles), as well as
virtual personas and emulation, as these allow for interaction with
2.1 What if? interactions in-use as of today and manipulation of physical objects.
Here we present three example interactions, which already support We first look at how these interactions are currently used and
What if? exploration. These examples cover “classic” item recom- then transfer them to the context of intelligent user interfaces by
mendation (e.g. restaurants), recommendation for productivity (e.g. describing a scenario in which they could be used for What if?
routing) as well as recommendation for creativity (e.g. text). exploration.
2.1.1 Location: Foursquare. The service Foursquare provides rec- 2.2.1 Feature weighting: Apple Music. To alleviate the cold start
ommendations for restaurants and other places based on users’ problem, users are asked about their personal preferences when
location and their search and check-in history. they first set up Apple Music (see [8]). Figure 1 illustrates this
While the app aims to find relevant venues at the current location, onboardig process: Users are given a number of items (first a set
it also allows users to change their search location, for example in a of genres, then artists) to interact with. Each item is displayed as
map view. This gives them the option to discover recommendations a bubble, and users can click on it to increase its size (and thus
they would receive if they were at a location different from their its relevance for system decision-making) or remove it altogether.
current one (i.e. What if I would be at a different location?). This allows for a playful interaction with the weighting of different
items of a given set.
2.1.2 Routing: Google Maps. Google Maps not only allows users to In the context of What if?, this interaction could be used to
view maps and search for addresses and places, but also to generate remove or weight certain features more or less to observe the effect
a route from A to B using different means of transport. For car on the model result. More concretely, in recommender systems that
routes, Google Maps offers two different dimensions that can be recommend further products based on characteristics of a product,
explored with What if?, namely time and path of the route. certain characteristics could be weighted more strongly by means
By default, Google Maps assumes that users start at the current of this interaction, while others that users do not consider relevant
time, and includes current traffic information about the route (such can be removed (i.e. What if I would change the importance of factor
as traffic jams, closures, etc.) and estimated travel time. However, X?).
users can also specify an individual time for departure or arrival,
and Google Maps will then provide an estimated travel time for 2.2.2 Time axis: MacOS. Time Machine is a built-in backup solution
that time (i.e. What if I would start at a different time?). in MacOS and allows the user to view a specific document in all
Another dimension is the path of the route. Google Maps pro- available backup states. As soon as Time Machine is started for a
vides a route suggestion and usually displays alternative routes document, the states are displayed piled up like a stack of cards,
greyed out on the map. In addition to the suggestions, the user can allowing the user to move through the versions along the time
also customise the route by clicking on a point on the route and axis [2] (see Figure 2).
then freely moving it on the map (i.e. What if I would take a different Since user preferences may change over time, a time axis-like
path?). interaction could enable users to apply the status of a recommenda-
The route length and the travel time are always displayed, so tion model from the past to today’s inventory. In this way, a “time
that users can easily compare different routing suggestions. travel” would reveal system learning over time (i.e. What if today’s
inventory is recommended using my profile from 10 years ago?).
2.1.3 Text: Grammarly. Grammarly is an advanced spell checker Moreover, this interaction might be applicable for content which
that attempts to improve the quality of writing. In addition to inherently changes over time, such as news or social media posts.
providing traditional spell checking, it suggests alternative words The time dimension of the model could also be fixed here, and
and phrases that may better match a particular context and target instead the time dimension of the inventory could be altered (i.e.
audience. What if music from 10+ years ago would be recommended today?).
For this, users can choose different options for the variables
audience, formality and domain. Based on this choice, the system 2.2.3 Virtual persona: The Sims 4 & Memojis. In the game The
provides different suggestions for correction and improvement Sims 4 Electronic Arts Inc. [7] players create virtual characters
of the text. While a user might write for a specific audience and which lives they direct. During this process, players have various
domain, the settings can be changed with one click. Thus, with possibilities to edit character traits and appearance in great detail
this type of What if? exploration, the user can see the influence – from the width of the nostrils to the distance between the eyes.
of these variables on the system recommendations (i.e. What if I A similar concept can be found in the Memoji feature on iOS [3],
would change the audience?). where users can create their lookalike as an emoji.
What if? Interaction with Recommendations ExSS-ATEC’20, March 2020, Cagliari, Italy
Figure 3: Car Emulator allows to create and manipulate an
object’s state [10].
any state of a vehicle for software testing (e.g. open/close doors,
lights on/off, etc.).
Such an emulator-based UI approach might be used also to let
users manipulate system decision variables in recommendation
systems. For example, users could change product characteristics
Figure 1: As part of Apple Music’s onboarding, users weight such as colour or brand – and see to what extent system recommen-
different artists and genres so that the system gets to know dations change. This could not only be applied to the currently rec-
their personal preferences. ommended product, but also to past recommendations (i.e. What if
this product would have different characteristics?).
3 DISCUSSION
The examples presented in the last section are meant to provide
inspiration for our community to work towards a richer UI design
vocabulary for What if? exploration in recommender systems –
in particular one based on direct manipulation of decision vari-
ables. In this section, we summarise general desiderata for such a
design vocabulary. Moreover, we discuss overarching implications
of a What if? approach to supporting users in interaction with
recommendations.
3.1 Conceptual view: “Recommendation item”
Figure 2: Time Machine allows users to navigate through the includes its key decision variables
history of a data item. Designing for What if? raises the basic question of how we define a
“recommendation”. In this paper, we suggest a new comprehensive
notion: A recommendation comprises not only the recommended
Transferred to the context of What if? interaction, the appear-
content itself (e.g. a film, product, etc.) but also the key decision
ance of such a virtual persona could be used to visualise a specific
variables. This view highlights that design solutions for What if?
user model. Allowing users to change their persona (e.g. adapt
could – and should – not just engage users in exploration with the
gender, age or interests) could give them the option to explore
recommended content itself, but also with the decision variables
system recommendations for other user models. This could be par-
based on which this recommendation was given. Casting these
ticularly insightful in recommendation systems where the physical
variables as an inherent part of the recommendation highlights
appearance is an important factor, e.g. in dating applications such
their need to be presented in the UI. More concretely, this would
as Tinder or personalised fitness apps such as Freeletics (i.e. What if
enable new direct manipulation interactions for users to influence
I would change my physical shape?).
the decision-making process in different dimensions or at different
2.2.4 Emulation: Car Emulator. Car Emulator [9, 10], shown in stages, not only by providing feedback via star ratings or giving a
Figure 3, is an application which lets developers reproduce almost “thumbs up” or “thumbs down”.
ExSS-ATEC’20, March 2020, Cagliari, Italy Zürn, Eiband and Buschek
3.2 Design solution principle: Rich direct Decennial Aarhus Conference on Critical Alternatives (Aarhus, Denmark) (CA ’15).
Aarhus University Press, 169–180. https://doi.org/10.7146/aahcc.v1i1.21197
manipulation of recommendations [5] Malin Eiband, Charlotte Anlauff, Tim Ordenewitz, Martin Zürn, and Heinrich
Most notably, the presented examples foster direct manipulation Hussmann. 2019. Understanding Algorithms Through Exploration: Supporting
Knowledge Acquisition in Primary Tasks. In Proceedings of Mensch Und Computer
– and through that a sense of immediateness: For example, when 2019 (Hamburg, Germany) (MuC’19). ACM, New York, NY, USA, 127–136. https:
assigning weights to artists and genres in Apple Music, users do //doi.org/10.1145/3340764.3340772
[6] Malin Eiband, Sarah Theres Völkel, Daniel Buschek, Sophia Cook, and Heinrich
not have to look through a list of filter criteria but instead can im- Hussmann. 2019. When People and Algorithms Meet: User-reported Problems
mediately grasp and experiment with the system decision-making in Intelligent Everyday Applications. In Proceedings of the 24th International
in a playful way. Similarly, in Google Maps, customising a route is Conference on Intelligent User Interfaces (Marina del Ray, California) (IUI ’19).
ACM, New York, NY, USA, 96–106. https://doi.org/10.1145/3301275.3302262
done via dragging the suggested route on the map. Moreover, Time [7] Electronic Arts Inc. [n.d.]. Create Your Sim Demo - The Sims 4 Official Site.
Machine and Car Emulator support direct manipulation through Retrieved 2019-12-18 from https://www.ea.com/games/the-sims/the-sims-4/pc/
affordances (i.e. through a card stack and a real-life object). create-a-sim-demo
[8] GottaBeMobile . 2015. Apple Music Setup - YouTube. Retrieved 2019-12-18 from
We argue that direct manipulation of recommendations and https://youtu.be/ydYah_BXt9A?t=30
decision variables in combination with affordances form promis- [9] High-Mobility GmbH. 2017. How To Use HIGH MOBILITY’s Car Emulator. HIGH
MOBILITY Magazine (2017). Retrieved 2019-12-18 from https://medium.com/
ing research avenues towards finding ways to integrate What if? high-mobility/how-to-use-high-mobilitys-car-emulator-244200558a8a
exploration into everyday applications. [10] High-Mobility GmbH. 2018. Porsche NEXT OI Competition: Car Emulator - YouTube.
Retrieved 2019-12-18 from https://youtu.be/y_-Wp7uETBA
[11] Todd Kulesza, Margaret Burnett, Weng-Keen Wong, and Simone Stumpf. 2015.
3.3 Challenges Principles of Explanatory Debugging to Personalize Interactive Machine Learning.
While many expert systems for interactive machine learning such In Proceedings of the 20th International Conference on Intelligent User Interfaces
(Atlanta, Georgia, USA) (IUI ’15). ACM, New York, NY, USA, 126–137. https:
as Tensorboard1 already support What if? interactions, designing //doi.org/10.1145/2678025.2701399
them for laypersons in everyday use is challenging. These chal- [12] Brian Y. Lim and Anind K. Dey. 2011. Design of an Intelligible Mobile Context-
aware Application. In Proceedings of the 13th International Conference on Human
lenges include designing exploration in a way which does not dis- Computer Interaction with Mobile Devices and Services (Stockholm, Sweden) (Mo-
tract from the actual task users want to do [5], and which does bileHCI ’11). ACM, New York, NY, USA, 157–166. https://doi.org/10.1145/2037373.
not overwhelm users [11, 12]. Moreover, one fundamental premise 2037399
[13] Brian Y. Lim, Anind K. Dey, and Daniel Avrahami. 2009. Why and Why Not
for designing for What if? in this context should be that users are Explanations Improve the Intelligibility of Context-aware Intelligent Systems.
not always interested in exploration – the HCI research community Proceedings of the 27th International Conference on Human factors in Computing
should therefore think about how What if? can be seamlessly inte- Systems, 2119–2128. https://doi.org/10.1145/1518701.1519023
[14] Tim Miller. 2019. Explanation in Artificial Intelligence: Insights from the Social
grated into the interface, for example via on-demand approaches. Sciences. Artificial Intelligence 267 (2019), 1 – 38. https://doi.org/10.1016/j.artint.
Finally, while we argue that What if? might foster overall (feeling 2018.07.007
[15] An T. Nguyen, Matthew Lease, and Byron C. Wallace. 2019. Mash: Software
of) control of the system, future work should explore if and how Tools for Developing Interactive and Transparent Machine Learning Systems.
often people actually make use of this possibility. In Explainable Smart Systems Workshop at the 24th International Conference on
Intelligent User Interfaces (IUI ’19).
[16] Advait Sarkar. 2015. Confidence, Command, Complexity: Metamodels for Struc-
4 CONCLUSION tured Interaction with Machine Intelligence. In Proceedings of the 26th Annual
We presented a set of example UIs from various current applica- Conference of the Psychology of Programming Interest Group (PPIG 2015). 23–36.
[17] The European Parliament and Council. 2016. Regulation (EU) 2016/679 of the
tions to inspire working towards a richer UI design vocabulary for European Parliament and of the Council of 27 April 2016 on the protection
interactions with recommendations, in particular supporting What of natural persons with regard to the processing of personal data and on the
free movement of such data, and repealing Directive 95/46/EC (General Data
if? exploration. Our suggestion for a promising design solution Protection Regulation). Official Journal of the European Union (2016).
principle here has two key takeaways:
First, we propose to conceptually view key decision variables as
an integral part of any “recommendation item” shown to the user. In
other words, such decision variables need to have a representation
in the UI. Second, we propose to then design direct manipulation
interactions for these representations of the decision variables. Here,
our set of examples provides ideas for possible starting points to
work towards concrete UI designs.
REFERENCES
[1] Ashraf Abdul, Jo Vermeulen, Danding Wang, Brian Y. Lim, and Mohan Kankan-
halli. 2018. Trends and Trajectories for Explainable, Accountable and Intelligible
Systems: An HCI Research Agenda. In Proceedings of the 2018 CHI Conference on
Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18). ACM, New
York, NY, USA, Article 582, 18 pages. https://doi.org/10.1145/3173574.3174156
[2] Apple, Inc. 2018. Use Time Machine to restore deleted files or older files - Apple
Support. Retrieved 2019-12-18 from https://support.apple.com/en-us/HT209152
[3] Apple, Inc. 2019. How to use Memoji on your iPhone or iPad Pro - Apple Support.
Retrieved 2019-12-18 from https://support.apple.com/en-us/HT208986
[4] Alan F. Blackwell. 2015. Interacting with an Inferred World: The Challenge of
Machine Learning for Humane Computer Interaction. In Proceedings of The Fifth
1 https://www.tensorflow.org/tensorboard, accessed 17 December 2019.