Showing users recommended content has become a prominent way of integrating algorithmic decision-making in everyday intelligent applications (e.g. recommendations of films, music, news, routes). In this context, the research community has identified What if? questions as an approach for users to investigate and question such recommendations - yet many current applications seem limited in practically supporting this. We present a set of example GUIs and interaction techniques currently used in everyday recommendation systems in practice (e.g. Grammarly, Apple Music, Google Maps). Based on these example cases, we discuss possible UI extensions to explicitly support What if? interactions. From our analysis and reflection emerges the general approach of treating decision variables as a “first-class citizen” in UIs: We propose to 1) represent a recommended item's decision variables in the user interface (and not just the item itself), and 2) to enable direct manipulation of these decision variables for What if? explorations.
• Computer systems organization → Embedded systems; Redundancy; Robotics; • Networks → Network reliability.
Artificial intelligence has been integrated into many everyday
enduser products and services to improve the user experience and to
help users navigate an ever-increasing amount of data. To this aim,
many of these systems show users recommended content like films,
music, routes and news based on complex algorithmic inferences.
To date, the algorithmic decision-making process and the decision
variables leading to a recommendation are often not accessible to
users or hidden in the user interface, making it dificult for users to
navigate this “inferred world” [
To support users, the HCI research community has worked on
several approaches. One of them are so-called What if? questions,
which allow users to explore, investigate and question algorithmic
decision-making. However, in comparison to work that focuses on
deriving explicit explanations for system decisions (e.g. [
In this paper, we argue that What if? interaction with
recommender systems ofers rich – and till now underexplored –
opportunities for user support: What if? exploration may 1) allow users
to understand the system decision-making in an implicit way
without the need for specifically crafted explanations, as argued in [
This is in line with calls for more expressive feedback and
correction and fine-grained control options [
To provide inspiration for concrete design solutions, we present a set of example GUIs and interaction techniques currently used in everyday recommendation systems in practice, such as Grammarly, Apple Music, Google Maps, and the like. Based on these example cases, we discuss possible UI extensions to explicitly support What if? interactions. From our analysis and reflection emerges the general approach of treating decision variables as a “first-class citizen” in UIs: We propose to 1) represent a recommended item’s decision variables in the user interface (and not just the item itself ), and 2) to enable direct manipulation of these decision variables for What if? explorations. 2
In the following section we first present for inspiration some selected, popular examples of interactions that can be interpreted as What if? interactions and that are well established and already familiar to users. For this purpose, we identified prominent use cases of recommender systems from diferent domains, such as shopping, entertainment, news, location-based services, social media, fitness and health, as well as typical decision variables used in these applications.
We then describe other existing interactions outside the What if? context, which were not originally designed for this purpose nor are currently used in this context – but which we argue are promising candidates for use in a What if? setting for recommender systems.
The examples that we deem most relevant will now be presented briefly. Please note that this is not a comprehensive overview, but a collection meant to inspire reflections towards concrete UI elements that support What if? exploration. 2.1
Here we present three example interactions, which already support What if? exploration. These examples cover “classic” item recommendation (e.g. restaurants), recommendation for productivity (e.g. routing) as well as recommendation for creativity (e.g. text). 2.1.1 Location: Foursquare. The service Foursquare provides recommendations for restaurants and other places based on users’ location and their search and check-in history.
While the app aims to find relevant venues at the current location, it also allows users to change their search location, for example in a map view. This gives them the option to discover recommendations they would receive if they were at a location diferent from their current one (i.e. What if I would be at a diferent location? ). 2.1.2 Routing: Google Maps. Google Maps not only allows users to view maps and search for addresses and places, but also to generate a route from A to B using diferent means of transport. For car routes, Google Maps ofers two diferent dimensions that can be explored with What if?, namely time and path of the route.
By default, Google Maps assumes that users start at the current time, and includes current trafic information about the route (such as trafic jams, closures, etc.) and estimated travel time. However, users can also specify an individual time for departure or arrival, and Google Maps will then provide an estimated travel time for that time (i.e. What if I would start at a diferent time? ).
Another dimension is the path of the route. Google Maps provides a route suggestion and usually displays alternative routes greyed out on the map. In addition to the suggestions, the user can also customise the route by clicking on a point on the route and then freely moving it on the map (i.e. What if I would take a diferent path?).
The route length and the travel time are always displayed, so that users can easily compare diferent routing suggestions. 2.1.3 Text: Grammarly. Grammarly is an advanced spell checker that attempts to improve the quality of writing. In addition to providing traditional spell checking, it suggests alternative words and phrases that may better match a particular context and target audience.
For this, users can choose diferent options for the variables audience, formality and domain. Based on this choice, the system provides diferent suggestions for correction and improvement of the text. While a user might write for a specific audience and domain, the settings can be changed with one click. Thus, with this type of What if? exploration, the user can see the influence of these variables on the system recommendations (i.e. What if I would change the audience?). 2.2
In this section, we consider interactions that are not currently used specifically for What if? exploration, but which we argue appear to be promising candidates in this respect.
Specifically, we present feature weighting because it is applicable to any kind of recommender system, a temporal component in form of a time axis since many recommender decisions change over time (both through new content and changing user profiles), as well as virtual personas and emulation, as these allow for interaction with and manipulation of physical objects.
We first look at how these interactions are currently used and
then transfer them to the context of intelligent user interfaces by
describing a scenario in which they could be used for What if?
exploration.
2.2.1 Feature weighting: Apple Music. To alleviate the cold start
problem, users are asked about their personal preferences when
they first set up Apple Music (see [
In the context of What if?, this interaction could be used to
remove or weight certain features more or less to observe the efect
on the model result. More concretely, in recommender systems that
recommend further products based on characteristics of a product,
certain characteristics could be weighted more strongly by means
of this interaction, while others that users do not consider relevant
can be removed (i.e. What if I would change the importance of factor
X?).
2.2.2 Time axis: MacOS. Time Machine is a built-in backup solution
in MacOS and allows the user to view a specific document in all
available backup states. As soon as Time Machine is started for a
document, the states are displayed piled up like a stack of cards,
allowing the user to move through the versions along the time
axis [
Since user preferences may change over time, a time axis-like interaction could enable users to apply the status of a recommendation model from the past to today’s inventory. In this way, a “time travel” would reveal system learning over time (i.e. What if today’s inventory is recommended using my profile from 10 years ago? ).
Moreover, this interaction might be applicable for content which
inherently changes over time, such as news or social media posts.
The time dimension of the model could also be fixed here, and
instead the time dimension of the inventory could be altered (i.e.
What if music from 10+ years ago would be recommended today?).
2.2.3 Virtual persona: The Sims 4 & Memojis. In the game The
Sims 4 Electronic Arts Inc. [
What if? Interaction with Recommendations
Transferred to the context of What if? interaction, the
appearance of such a virtual persona could be used to visualise a specific
user model. Allowing users to change their persona (e.g. adapt
gender, age or interests) could give them the option to explore
system recommendations for other user models. This could be
particularly insightful in recommendation systems where the physical
appearance is an important factor, e.g. in dating applications such
as Tinder or personalised fitness apps such as Freeletics (i.e. What if
I would change my physical shape?).
2.2.4 Emulation: Car Emulator. Car Emulator [
Such an emulator-based UI approach might be used also to let users manipulate system decision variables in recommendation systems. For example, users could change product characteristics such as colour or brand – and see to what extent system recommendations change. This could not only be applied to the currently recommended product, but also to past recommendations (i.e. What if this product would have diferent characteristics? ). 3
The examples presented in the last section are meant to provide inspiration for our community to work towards a richer UI design vocabulary for What if? exploration in recommender systems – in particular one based on direct manipulation of decision variables. In this section, we summarise general desiderata for such a design vocabulary. Moreover, we discuss overarching implications of a What if? approach to supporting users in interaction with recommendations. 3.1
Designing for What if? raises the basic question of how we define a “recommendation”. In this paper, we suggest a new comprehensive notion: A recommendation comprises not only the recommended content itself (e.g. a film, product, etc.) but also the key decision variables. This view highlights that design solutions for What if? could – and should – not just engage users in exploration with the recommended content itself, but also with the decision variables based on which this recommendation was given. Casting these variables as an inherent part of the recommendation highlights their need to be presented in the UI. More concretely, this would enable new direct manipulation interactions for users to influence the decision-making process in diferent dimensions or at diferent stages, not only by providing feedback via star ratings or giving a “thumbs up” or “thumbs down”.
Most notably, the presented examples foster direct manipulation – and through that a sense of immediateness: For example, when assigning weights to artists and genres in Apple Music, users do not have to look through a list of filter criteria but instead can immediately grasp and experiment with the system decision-making in a playful way. Similarly, in Google Maps, customising a route is done via dragging the suggested route on the map. Moreover, Time Machine and Car Emulator support direct manipulation through afordances (i.e. through a card stack and a real-life object).
We argue that direct manipulation of recommendations and decision variables in combination with afordances form promising research avenues towards finding ways to integrate What if ? exploration into everyday applications. 3.3
While many expert systems for interactive machine learning such
as Tensorboard1 already support What if ? interactions, designing
them for laypersons in everyday use is challenging. These
challenges include designing exploration in a way which does not
distract from the actual task users want to do [
We presented a set of example UIs from various current applications to inspire working towards a richer UI design vocabulary for interactions with recommendations, in particular supporting What if ? exploration. Our suggestion for a promising design solution principle here has two key takeaways:
First, we propose to conceptually view key decision variables as an integral part of any “recommendation item” shown to the user. In other words, such decision variables need to have a representation in the UI. Second, we propose to then design direct manipulation interactions for these representations of the decision variables. Here, our set of examples provides ideas for possible starting points to work towards concrete UI designs.