More and more smart home devices are hitting the market. While current applications on mobile devices and classical interface elements (e.g., wall mounted switches) are used to interact with these devices, we investigate in this position paper what interaction paradigm might be used in the future. We discuss different smart home devices and argue what interaction paradigm might be most useful. As an increasing number of smart home devices can act as autonomous agents interface paradigms like intervention interfaces or autonomous interfaces might become the main communication channel between the smart home user and the smart home system of the future.
Copyright © 2018 for this paper held by its author(s). Copying permitted for private and academic purposes.
Smart home devices penetrate more and more our daily
lives. Systems such as smart heating or smart room lighting
received considerable attention lately. Due to a high level of
connectivity, they can be controlled via various types of user
interfaces. These include smartphone applications, voice
control (e.g., Amazon Alexa or Google Home), or classical
wall switches. Research also showed that AR applications
can support users during the interaction with a smart home
system [
Various smart home devices can be controlled. The room
temperature can be set to a certain degree and the
brightness and color of each light can individually be changed to
the user’s preferences. While this seems to be the ultimate
goal of home automation, the question arises if the user
still needs to be in the loop. With current ubiquitous
sensing technologies, body temperatures of the humans in the
room can be measured and room heating adapted to a level
that moves their body temperature to a decent level. Thus,
the system can fully act autonomously. On the other hand,
however, classical user interfaces should be designed in a
way that the user stays in control, following Shneiderman‘s
golden rules (e.g., support internal locus of control) [
In this paper, we discuss different approaches to control smart home devices. We provide examples in which the different interface paradigms clash in the smart home domain. We discuss potential solutions that tackle these issues and ways that have to be explored to design smart home systems in an optimal way.
Throughout the years, several interface paradigms have been used or envisioned to interact with smart home devices. In the following, we introduce four different types of interfaces and briefly discuss their characteristics regarding smart home.
Most devices in a common home environment offer interfaces such as switches which can be used to manipulate their behavior (e.g., a light switch that changes the state of a light bulb). As the complexity of these devices increases, these interfaces may become unsuitable for proper interaction and the control of the various features. Nevertheless, they offer an easy mapping and are currently the default way of controlling non-smart home devices.
With the advent of smart home devices, remote interfaces
gain more and more importance. Mainly these interfaces
are provided by an application running on a mobile device
like a smartphone or tablet. These applications allow
various types of control. They can be used to adjust the
temperature, the brightness, or the color of a light bulb to the
desire of the user. The user is in direct control on how the
smart home device behaves. The user also directly
understands what he or she is doing. In addition to mobile
applications, research introduced further means of interacting
such as gesture or posture detection [
The more a system is configured to work autonomously
the less input by the user is required. Particularly
continuous interaction is often superfluous. However, the user
desires an applicable level of control of the system. At this
point, the term intervention was implemented as an
interaction metaphor. Schmidt and Herrmann defined
intervention in the field of human-computer interaction as an action
of the user that influences the behavior of an automated
system [
With an increasing number of sensors deployed in homes as well as an increasing power of artificial intelligence and machine learning systems, a system fully autonomously controlling a smart home might be the reality soon. This, on the flip side, would result in the user abandons all control to the system. The question arising here is if the user needs to be in control of all smart home devices. The heating, for example, is mainly controlled by the user in cases in which he or she feels too cold or too warm. Given a thermal camera, for example, measuring the body temperature of the user as input, a fully autonomous system might easily be capable of adjusting the room temperature in a way that the user neither need control nor intervene.
Today, more and more home items are extended with the possibility to be remotely controlled. In the future, almost all traditional systems will receive these capabilities. We discuss some of the current smart home devices with respect to the interface that might be most appropriate. Thereby, we do not consider malfunctioning of sensors nor the setup procedure that both would require additional user input.
Considering a lighting system, a normal basic interaction in any house is the direct manipulation interface (e.g., on/off light switch). In smart houses, light could be handled automatically based on the user location and activity detected by sensors either embedded in the environment or on the user. This, however, requires the user intervention in case of exceptional activity (i.e. turning manually a reading lamp on). Mobile interfaces might be of good use if the user wants to control sophisticated features of the lighting system (e.g., changing color or light temperature). For the default operations (e.g., switching on and off), the mobile interface poses a lot of overloads (i.e., getting mobile, unlocking a phone, selecting an application, etc.) making it less useful.
Nowadays, most of the smart heating systems depend totally on the smart thermostat for detecting the in-room temperature and adjusting it to the user preference. While this satisfies the user expectation, it is not adaptable to the user needs. One good solution is to have the room temperature adjusted to the user’s body (i.e. human is cold ergo more heating). To be able to apply such approach is to provide intervention interface with options, especially if the adjustment of the heating system would clash with the appliance of the ventilation one. Taking the scenario of scheduled window openings, which is required by the user for ventilation purposes, the system would have a conflict of goals requiring the user to intervene.
While activating the heat and light systems might depend
on the user’s presence and activity, other devices such as
smart kitchen appliances might also be influences on the
user’s spontaneous likes or emotions (e.g., baking
chocolate cake when being sad). A smart coffee machine could
brew the coffee based on the current sleeping behavior
so that the coffee is ready once the user awakes. While
this certainly requires intervention from time to time,
remote control might rather be exceptional. Direct control also
might still be useful, particularly for devices that might pose
potential threats when used wrong (e.g., a stove that
enables itself). Further, more sophisticated kitchen system
would also involve direct input [
Cleaning robots already work mainly autonomously
today [
Smart door locks allow the user to provide access to their house remotely to friends waiting in front of the house when the host is late or to package delivery when nobody is at home. Since this is a crucial aspect and providing access to one’s house has huge privacy implications, we believe that this type of smart home device might rather be controlled directly or remotely. The user might prefer staying in control rather than being comforted by autonomous controls.
In the future, smart homes will also contain window
elements that are smart [
The examples presented in this work show that for most smart home devices the classical interfaces are not the most promising ones. We believe that intervention user interfaces, as well as autonomous interfaces, might be the interfaces that will shape the interaction with smart home devices. As soon as users trust the system enough to allow moving their control to the computer, classical interfaces will become an exception and mainly used for intervention only. However, there might always be devices for which the user wants to stay in control. One example for which the user might not easily pass the control to an artificial intelligence is the smart door lock since the potential privacy threat can be severe.
In the future, we plan to conduct large-scale assessments of how users currently use smart home devices and what interfaces they prefer. We believe that within the next years, user’s preferences will dramatically change as soon as they understand the benefit and gain more and more trust in smart home systems.