Intervention User Interfaces for the Smart Home

Abstract

Intervention User Interfaces (IUI) in smart homes help users to maintain the experience of control for themselves and provide a means for others to express their preferences and to keep them in the loop. Furthermore, IUIs can help to bridge the gap between abstaining from automation and a state of perfect automation. They help to augment imperfectly automated processes by providing the option to fill the gaps on the fly. In this paper, we discuss possible applications of IUIs for automated processes in smart homes.

-

________________________________________________________ Workshop proceedings Automation Experience across Domains In conjunction with CHI'20, April 26th, 2020, Honolulu, HI, USA Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). Website: http://everyday-automation.tech-experience.at

Introduction

Automation and developing automated processes have come a long way — from individually crafted solutions and specific industrial appliances to parts of everyday life for a wide portion of the population. One interesting research area where even laypersons are involved in automated processes is the smart home and with it the ever-increasing availability of Internet of Things (IoT) devices for domestic tasks [ 6 ]. With smart homes, frequently reoccurring and boring tasks are automated. Different acts of control – such as turning on the TV and stereo, dimming the lights, adjusting the temperature and postponing the start of the washing machine – can be combined and performed as a single "routine". The user can start such routines, or they can be triggered by time of day, calendar-event, specific weather conditions, movement, presence or the tweet of a celebrity. This offers great convenience for the users, e.g. to ensure the best home entertainment experience. However, users are also challenged by automated smart home processes as underlying procedures are harder to monitor and to control. Different processes are intertwined (e.g. by sharing resources or reacting to the same events) or serve as triggers to start each other. Once the alarm tone of the smartphone has gone off in the morning, the lights are already on (after a simulating the sunrise) and the coffee is brewed. When a user realizes that he wants to exercise more control or influence, several actions have already been performed or started, and instructions are forwarded to other systems. Thus, making it hard for the user to address the “correct” system — will turning the light off stop the coffee maker? For such cases – where a user wants to modify steps of automated processes in a smart home, seeks to stop one or all of them completely, or at least tries to understand the scope and effect of available alternatives – we propose the use of intervention user interfaces (IUI) [ 5 ]. They serve as a central interface for the adjustment and control of automated processes, and they either provide the option to stop automated procedures, e.g. by the push of a button, or they allow the user to exercise fine-grained control.

Intervention User Interfaces give the users the experience of being in control of an automated process and allow them to influence it according to their preferences or stop it completely. Intervention User Interfaces are designed to provide the user with the appropriate options in case of exceptions that are not covered by the current configuration because they were either not considered or simply omitted at that time.

Challenges of IoT-based Smart Homes

Smart homes offer routines e.g. for combining several acts of control in one command and setting different devices to specific states. They also allow to employ specific events, triggers or voice commands to initiate such routines. Such routines become part of a sociotechnical system that integrates the residents of the smart home, their relationships and dependencies, and the employed technology [ 3 ]. Often an enthusiastic household member is the driver of smart home adoption and takes care of the set-up [ 2 ]. However, the smart home and its smart capabilities do not only affect this driver but all the other residents.

Within these settings, the potential for intervention user interfaces is extensive. Our lives are constantly changing: new needs emerge, others disappear. The smart home has to keep up with these changes [ 1,2 ]. Defining a set of basic routines is easy; reliably implementing such routines is demanding. A rule set is needed to cover the many possible conditions: when should the rule apply?, for how long?, what are the exceptions?, what happens in case of a breakdown of the system?, how to consider guests or pets? Determining such rules and considering all possible exceptions and deviations is tedious. Especially, if the needs and demands of other residents also have to be considered. Considering them can also require costly components – such as strong enough actuators or specific sensors. An example where the determining of rules and employment of technology gets complex is the question: is someone home or not? — commonly referred to as presence detection. This information is the basis for several other rules and can kick off several actions: the alarm system is armed, the general room temperature is lowered, the vacuum robot is started and all windows are closed. There are several ways to implement this: Detecting the presence of smartphones or other Wi-Ficlients, the use of specialized transponders (often attached to the key chain), specialized presence detectors (sensing the presence of humans through infrared or movement), the state of the door lock or the alarm system. The simplest solution is a switch right next to the front door, which is flipped once the last person leaves the home or the first person enters. All of these solutions come with their own set of strengths and limitations. Detecting the presence of a smartphone is easy and cheap but will fail if the flight mode is used or the battery dies. Relying on specialized sensors can be costly and require construction work. “Tricking” the system will become hard, e.g. for a test or to adapt to a new situation without interfering with the ruleset. A human body cannot hide from the sensors to see what it would be like to be in an “empty” home, some sensors cannot reliably distinguish between a human and a cat. However, using a switch instead of sensors is easy but not reliable, as it is easy to forget to flip the switch during the morning hassle.

Another example refers to the central heating system. The classic ones rely on a fixed schedule and distinguish working hours during the week from weekends and have programs for day and night times. For decades they commonly offer a “party mode”. Once this mode is activated (through the push of a button), the regular schedule is overruled and the heating system is kept active through the current night. The next day the schedule will again apply. This is clearly a form of intervention into the system and its schedule: the change is temporary [ 5 ]. However, the reverse situation is not addressed in common heating systems: if the heating system is generally kept on until late an “early to bed mode” would be desirable and this feature is hardly present.

Modern smart home interface can offer a lot of control features — down to the lowest layer: e.g. the individual parameters of valves or the heating pumps rotation speed in rounds/minute. They can also be quite abstract and present a very high level overview of functionality where the desired states are captured in “scenes”, “modes” or “routines”. While the lower level of control is often the factory default, the latter needs to be designed by the users or by a consultant. The extent of individual controls can be overwhelming for users, especially occasional users, e.g. roommates, spouses or children. The level of abstract control requires implicit knowledge, which often only the designer has so that the complete scope and all possible outcomes can be difficult for others to understand. A smart home intervention user interface offers means to change the current state and presents possible options to do so and undo them if required. Such an option can be a common exception that is known to occur but is difficult to detect or plan, or is simply too rare to be implemented. Interventions can be predefined, created through EUD [ 4 ] or proposed by the smart home based on the users’ previous actions and behavior. The consequences are clearly stated or immediately observable and easy to undo without further knowledge [ 5 ]. This helps to include and respect the needs of not fully enthusiastic smart home users and enables them to interact more sophisticated with the smart home.

First empirical Insights into User's Requirements

Currently, we are studying what smart home enthusiasts try to automate, why they – for example – integrate their smoke detectors but not their doorbell into the system, what their visions are for further automation and their potential requirements for intervention user interfaces and how they would use them to intervene. The empirical bases are explorative interviews with people active in smart home online communities or attendees of smart home related events and fairs. So far, all our interviewees have been male and have been mainly responsible for the maintenance and configuration of their – mostly DIY – smart homes. The interviewees describe smart home as their hobby. The involvement of the other household members is divers. All participants discussed what they are configuring of the smart home with other members of the household and tried to “keep things working” for them. The other residents commonly started to rely on the introduced smart functions and automated features and asked the “guru” to restore functionality if it was not available anymore. However, the participants did not report serious conflicts caused by their automation attempts or the outcome of these attempts. The need for keeping “things working” was a common cause not to automate specific aspects or areas. About the aforementioned presence detection, one participant stated: “I have to find a solution which works reliably for two. And I haven’t found one, yet.” To still use presence related rules, the user placed a simple button right next to the front door as a workaround. Providing an easy to use interface — like the button — is close to the idea of intervention user interfaces. A possible way to implement it would be an intervention button. Like the “party button” of a central heating system or the button at the front door. The intervention button allows to stop/enable certain procedures — such as disabling all energy saving settings for the night of a party — or switch to another mode if the presence detecting sensors provide misleading signals. These forms of intervention allow other members to stay in the loop. They are empowered to interact with the system regardless of their knowledge of it. Through the provided undo function and the generally limited effect of the intervention, the fear of breaking the system is reduced. Furthermore, the use of interventions can provide a basis to initiate EUD. The whole group of smart home inhabitants is enabled to automate a new set of routine tasks or create personalized modes that meet their needs [ 4 ].

The spectrum between missing automation or smart functions and the perfect automation experience is continuous. Not being able to fully and securely automate some parts of the home due to external factors was a common theme in our interviews — e.g. due to restrictions by the landlord, high costs or fear of failures. Some fully abandoned and some postponed their automation plans until new sensors would be available or they move to another home. In some cases, they used “hacks” or workarounds: instead of buying a smart washing machine they monitor the power consumption through plug adapters to determine if the laundry is done, yet; the use of alternative firmware to gain more control was also common.

Discussion

The design of interfaces as an intervention user interface and the creation of interventions supports users to experience automation as if they are in control although they do not contribute to its configuration actively. Without intervention, their needs might be ignored or they might even refuse to use certain automated processes. The intervention user interface augments the capabilities of the system for non-standard cases and eases the management of errors in automated routines – e.g. if the smartphone battery is discharged and automatic presence detection fails, it would allow indicating the user's presence via voice command: "Assistant, I'm home". Further, users can adapt the automated behavior easily, and fill in the missing parts to cover all the rough edges of their lives. While the morning routine on working days probably suits most working days and requires only minor adjustments, weekends can be quite different. A relaxed Sunday with an extensive breakfast is very different from the morning after a party night or the morning before a city trip.

Conclusion and Outlook

We highlighted several areas in smart home settings that would benefit from intervention user interfaces. One is providing shortcuts for active users or for other residents to keep them in the loop and further enable them to become designers of the system themselves, e.g. through “intervention buttons” — which would resemble the widely known and adopted emergency stop buttons. However, instead of separated intervention interfaces or buttons, user interfaces for the smart home must incorporate the idea of interventions and enable all users to intervene and realize their needs to feel in control.

Just as intervention user interfaces bridge the gap between experts and non-experts, IUIs can bridge the gap between current automation capabilities and the desired level of automation by easing the handling of errors and unforeseen conditions. By providing shortcuts for exceptions, they enable the smart home designer to determine usable routines even if the sensor technology cannot reliably recognize every relevant constellation or the rule set is not yet complete. The fast pace of the smart home industry creates further options and possibilities for automation. With this development, however, the burden of determining the associated rule sets is also growing. Getting them right, complete and bug free will become almost impossible. Even if rules become superfluous through the use of AI agents, we will still need the ability to intervene to express our needs and enable us to live outside the confines of an idealized life. Intervention user interfaces can help to achieve this.

For further research, we plan to collect additional requirements from the various user groups of smart homes. We also plan to develop such intervention user interfaces and test them in real smart home settings. In a further study, we will investigate how users perceive the difference between determining rules (configuration) or using intervention user interfaces (interventions) to achieve the desired behavior. Therefore, we will place the users in an "Escape-Room" setting where they will learn and apply both ways of interaction.

References

1. Nazmiye Balta-Ozkan , Rosemary Davidson, Martha Bicket, and Lorraine Whitmarsh . 2013 . Social barriers to the adoption of smart homes . Energy Policy 63 : 363 - 374 .

2. A.J. Bernheim Brush , Bongshin Lee , Ratul Mahajan, Sharad Agarwal, Stefan Saroiu, and Colin Dixon . 2011 . Home automation in the wild: challenges and opportunities . Proceedings of the 2011 annual conference on Human factors in computing systems - CHI '11 , ACM Press, 2115 .

3. Scott Davidoff , Min Kyung Lee, Charles Yiu , John Zimmerman, and Anind K. Dey . 2006 . Principles of Smart Home Control . In P. Dourish and A . Friday, eds., UbiComp 2006: Ubiquitous Computing . Springer Berlin Heidelberg, Berlin, Heidelberg, 19 - 34 .

4. Thomas Herrmann , Christopher Lentzsch, and Martin Degeling . 2019 . Intervention and EUD: A Combination for Appropriating Automated Processes . In A. Malizia, S. Valtolina , A. Morch , A. Serrano , and A . Stratton, eds., End-User Development . Springer International Publishing, Cham, 67 - 82 .

5. Albrecht Schmidt and Thomas Herrmann . 2017 . Intervention user interfaces: a new interaction paradigm for automated systems . interactions 24 , 5: 40 - 45 .

6. Charlie Wilson , Tom Hargreaves, and Richard Hauxwell-Baldwin. 2015 . Smart homes and their users: a systematic analysis and key challenges . Personal and Ubiquitous Computing 19 , 2 : 463 - 476 .