Interaction design/HCI seems to be at a crossroad. On the one hand, it is still about designing for engaging user experiences (UX). Still, on the other hand, it seems to be increasingly about reducing interaction and automating human-machine interaction through AI. In this paper, we explore this seeming gap to move forward. First, we discuss the fundamental design rationality underpinning UX/engaging interaction and AI/automation of interaction from the viewpoints of classic theoretical standpoints. We then illustrate how these two come together in interaction design practice. Here we rely on two classic examples, i.e., an autonomous car and a social media platform. Through an interaction analysis of these two examples, we show 1) how interaction and automation are combined in the design, 2) how engaging interaction is dependent on a certain level of automation, and 3) how each example illustrates a different balance between interaction and automation. Based on this analysis, we propose the Engaging Interaction through Automation (EIA) - scale as a conceptual construct that takes these aspects into account to understand and analyze ways of combining interaction and automation in interaction design. We illustrate the use of the proposed EIA-scale, discuss its theoretical implications, and suggest it as a useful tool - when designing for engaging user experiences (UX) through automation, with AI as an interaction design material.
At first glance, it looks like HCI is at a crossroads. On the one hand, Artificial Intelligence (AI) is
sweeping across our area in the 3rd wave of AI explorations and deployments [
There is at the current moment a growing body of research on automation of everyday life (see, e.g.
[
A starting point for our analysis is to understand HCI and the history of efforts taken in HCI as a research and development program dedicated to understanding and designing for comfortable, engaging, and direct interactions with computers. Guiding principles for this direction have to make these computational machines usable in terms of ease of use (e.g., usability) and control (e.g., WYSIWYG, Fitts Law, direct manipulation, etc.), i.e., to design machines that carry out precisely what the user wants, with high precision. Here, “control and precision” has been a guiding goal behind these efforts. Accordingly, HCI has been about understanding user needs, requirements, and human capabilities (ranging from cognitive abilities to perception and our bodies) and designing interactive systems perfectly adjusted to human needs and activities.
To reach this goal of “control and precision” in interaction design, we have in HCI worked along with several design paradigms and principles, including the use of metaphors, skeuomorphic design, and symbols (to link interface elements to its corresponding functionality). We have borrowed and worked along the gestalt laws from psychology to make the user interfaces easy to see, understand, and use. In short, we have taken several approaches to arrive at a high level of usability (to enable intended actions to be made through the interface). We have worked with notions such as understandability, comfort, ease of use, and learnability to ensure that the user feels that they are “in control” and that they can carry out intended interactions with or through these interactive systems with a high degree of precision in the actions taken (that is, the essence of “WYSIWYG” – What You See Is What You Get).
But “control and precision” has not been the only goal driving this development. With a focus on the active user, who interacts with these systems through interaction modalities that allow for direct manipulation, navigation, explorations, and even tangible or embodied forms of interaction, there have been complementary goals for making these sessions engaging. In short, a complementary goal has been to design for engaging user experiences (UX). We see this in the development of VR caves (with related notions of “immersion” and feeling present), or for website and computer game design where the user should not just use the website or the game but should be entertained, attached, and committed. In short, by applying a user-centered approach, we have in HCI been occupied with the design of interactive systems along with user requirements, wants, and needs to ensure “precision and control” while also exploring aspects of what it should feel like, or be about, when using these interactive technologies – that is, UX – User Experience.
We now turn to the second contemporary movement, which is the use of AI to automate interaction. The move towards AI offers, among other things such as new functionality, an alternative to the classic HCI concepts of precision, ease of use, usability control, and engagement. In fact, along with the current AI movement, “ease of use” in many cases mean ‘no interaction’ or minimal interaction since the work is delegated to the intelligent machine, that is, to substitute interaction with automation. This paper refers to this as “automation of interaction.” At the same time, the second notion, “control,” is also changed – from control as a matter of the user “being in control,” and about being able to control and with precision carry out what they want to do with a computer, to a matter to “controlling” that the AI system is doing what it is expected to do (and to control that it is not doing something unwanted).
Here, this shift in the meaning of the notion of “control” has led to calls for explainable AI (ref), responsible AI (ref), and issues of ethics and trust in AI systems has been foregrounded (see, e.g., ref, ref) to ensure that the AI system is doing what is intended and expected. While the movement towards “precision and control” foregrounded direct manipulation and active and engaged users, this movement towards automation of interaction suggests that there are emerging worries to address related to how interactive systems should act – in relation to us, and on our behalf. In short, this trend seems to be about reducing or substituting interaction with automation of interaction.
According to Janlert & Stolterman [
A brief explanation of the model and its core concepts might be needed. First, the “states,” as
explained in [
The model also details the activity on both the artifact and user sides. For instance, states change as a result of an operation triggered by a user action or by the move (action) by the artifact. These moves appear as a cue for the user. These cues come to the user either as e-state changes or w-state changes.
The model is meant to be a tool for analyzing any form of human-artifact interaction. It serves our purpose well to investigate the relationship between automation and interaction.
Based on the model, we can now define any form of ‘automation of interaction’ as removing a pair of actions and moves from an interaction while leading to the same or similar outcome. We first focus on the extreme forms of the relationship between interaction and automation, that is, when there is no automation (full interaction) and when there is no interaction (full automation).
The extreme form of no automation of interaction means that the artifact does not perform any operations and moves other than those triggered explicitly by an action of the user. This means that the user has complete control of all activities and outcomes, which requires intimate knowledge and skill. It also means that the user needs to understand the artifact and the relationship between user actions and artifact moves.
The extreme form of full automation of interaction means that the artifact performs all its operations and moves without being triggered by any actions from the user. Instead, the artifact moves are based on its i-states or changes in the w-states. This means that the user has no control of activities and outcomes. It also means that the user does not need any particular knowledge or skills since the artifact performs all actions.
We can now see that ‘automation of interaction’ through AI means that we either substitute manmachine interaction with AI support that can automate the complex relationships between actions, operations, moves, and/or cues as the basic model of interaction expresses (fig 1). Typically, this can be implemented by designing a system that monitors user behaviors and expressions to figure out what is expected from the system. This also means that instead of reacting to precise user actions, the system moves based on interpretations of previous interaction(s) and estimations of the most effective moves to take (given the probability model that governs the system). In many cases, the reduction of interaction will lead to a loss of control and precision, but maybe with a gain in functionality, performance, and of course, lesser need to focus on interaction.
We now use this model of interaction to examine how AI and UX come together in practice. We do this by examining two examples – including an autonomous car and a social media platform. 3.1.
This first example illustrates how an automated car allows for other activities than steering the car while driving. In this first example, the autonomous vehicle (the “artifact,” fig 1) controls the driving of the vehicle by reading different w-states (e.g., lane assistant, reading the location of other cars, pedestrians, etc.). The reading of these changing w-states affects the i-state of the autonomous vehicle (e.g. it continuously updates its whereabouts based on GPS and sensor data), and it makes its next move based on this sensor data (e.g., turning left/right, accelerating, hitting the breaks, etc). In this example, the passengers (users) in this car can sit back, relax, and monitor the moves taken but do not need to take any actions. The user can feel relaxed while the car is autonomously and safely moving forward. The user experience (UX) design is in this example built around a model of reduced need for interaction. 3.2.
This second example illustrates how our actions online also affect what the social media platform will present to us in return. There are, of course many ways in which AI algorithms are used on social media. Here we only cover one small scenario to illustrate how to use the basic model of interaction (fig 1) to see how AI and UX come together in this example. So, for the user to get a good user experience (UX) while using a social media platform, we can assume that it is vital that relevant posts are shown to this particular person. Here, the AI system keeps track of the user's interaction history (that is, a log of the actions taken by the user). These actions could be to click on a link or “link” some content. Each click generates an operation to the social media platform (artifact), and the artifact responds with a move (e.g., showing that the like symbol has been clicked). At the same time, these actions are also processed by the artifact (i-state), together with thousands of other clicks on the platform, to adjust the flow of information presented to the individual user (cue). In short, by reading individual actions taken by the user (data for the AI,), the artifact can provide a more personalized user experience (UX).
These two examples illustrate: 1) how interaction and automation of interaction are combined in the design, 2) how engaging interaction is dependent on a certain level of automation, and 3) how each example illustrates a different balance between interaction and automation. In the first example, there is a high degree of automation. The balance between UX and AI is heavily leaning toward a high degree of automation, where very little interaction is needed. With the social media platform, the second example illustrates a different scenario. Here engaging interaction, from the viewpoint of gaining one’s interest, depends on how well the algorithms can estimate what content the user wants to see and engage with. Here, the balance is slightly different in that this estimation is also partly based upon which previous actions the user has taken. It is not an example of full automation, nor an example of plain interaction – it is instead an example of the interplay between AI and UX. This leads us to the third point here that it seems like the two examples operate along a scale in terms of how aspects of user interaction and automation of interaction are integrated in the interaction design.
Based on this simple analysis of these two examples, we will in the next section propose the Engaging Interaction through Automation (EIA) – scale as a tool for analyzing and guiding interaction design that seeks to arrive at engaging interaction by integrating aspects of user interaction and automation.
In the previous section, we could see how the basic model of interaction allowed for an analysis of the two examples – with a particular focus on how aspects of interaction and automation are combined in each design, how design for engaging interaction is dependent on a certain level of automation, and how these two examples illustrate different balances between interaction and automation. In short, it seems like engaging interaction can be designed through a combination of the two strands. Each example is “leaning” either towards design for less interaction or design for more interaction. As our point of departure, we propose the Engaging Interaction Through Automation (EIA) – scale (or dimension).
This is a simple scale from no automation requiring full interaction to full automation requiring no interaction. Still, we suggest that this scale makes it possible to map different designs along a scale (in a similar way as we have analyzed the two examples in this article). Instead of seeing AI and UX as two different directions for interaction design to take, we suggest this scale as a unifying framework where these two strands can be brought together for further analysis.
So, how can this proposed EIA-scale be used in interaction design practice? And what are the theoretical implications of our analysis? Well, we suggest that the EIA-scale can work as a valuable tool to examine existing interaction designs in terms of how the interactive systems basic model of interaction is geared towards no automation or towards full automation. We also think that the scale can be helpful as a tool for discussing future designs, at the drawing board, before any form of implementation. Here, questions can be asked about how active the user should be and how intrusive the system should be (e.g., monitoring the user, collecting data, sending notifications, etc.). If the design for engaging interaction is simultaneously a question of a certain level of automation, then this scale opens up for closer examinations of what the balance should be for a particular design. Also, and if we now shift our focus to the theoretical implications of our work, we suggest that our analysis foregrounds the importance of more detailed examinations of how the primary interaction model is configured in different interactive systems. One such analysis would help us as a field become better at “reading” different designs and help us to see how other systems operate in different ways about their users. We also think our work illustrates how AI is not coming along as a “disruptive technology” that might challenge the whole field of interaction design to move away from interaction in favor of automation. On the contrary, our analysis has shown how AI is needed for good UX in many cases. The overarching theoretical implication for HCI is accordingly to conceptually include this direction as we further explore interaction design for engaging interactions.
In this article, we discussed how interaction design/HCI seems to be at a crossroads. On the one hand, it is still about designing for “ease of use,” control and precision, and engaging user experiences (UX). Still, on the other hand, it seems to be increasingly about reducing interaction, about automating human-machine interaction through the use of AI. In this article, we have described these two strands and discussed these two rationalities for interaction design in relation to a basic interaction model. With this model as our unifying conceptual construct, we have examined two examples of how elements of UX and AI are combined in practice – for the shared goal of designing for automation and engaging interaction. In short, we demonstrate how AI and UX operate, not at a crossroad, but for the same purpose. We then moved from the two practical examples to a more general discussion on how these two are linked from the viewpoints of ‘interaction’ and ‘automation of interaction’. Based on our analysis, we introduce the Engaging Interaction through Automation (EIA) – scale to open up for discussions and further investigation of how different designs might lean more towards a combination of AI and UX elements that demand little or almost no interaction (full automation), or where AI is only operating in the background to allow for active and engaging user experiences. Having introduced this scale, we conclude the article with a discussion on how it can be used and its theoretical implications. Overall, we suggest that instead of seeing UX and AI as belonging to different trends or being at a crossroad, it is more constructive to further examine AI as a design material for UX design and further analyze how these two strands might scaffold each other in interaction design practice.