Several tools have been put forward for creating automations in smart homes both at a research and commercial level. However, often people still encounter difficulties in specifying them in order to obtain the desired behaviour. To design authoring tools in which even people without programming experience can flexibly indicate the desired automations it is important to have a better understanding of user needs and how they think and describe the automations. In this contribution, we introduce a user study with such goals and provide an initial discussion of what has emerged.
There is a recent increasing interest in
personalizing smart homes, environments where
Internet of Things (IoT) devices are present and
jointly used with Web services. An effective
approach to configure these environments is using
the trigger-action programming (TAP) paradigm
[
Although many studies focused on how to
make TAP rules more flexible and expressive [
From related work, it emerges that a study is missing where user-created automations are investigated both from a natural language and a formal structure perspective. This would help in better understanding how users map from an abstract automation idea into something that could be eventually executed by a home automation system. Not specifying a predefined list of services and devices would uncover which functionalities people expect from their smart environments. Also, not much attention has been devoted to analysing automations comprising multiple triggers and actions. A better understanding of how people think of operators and connectors between rule parts would be useful to designers of IoT tailoring systems, to provide functionalities that better mirror the users' mental models and expectations. We hence set up an experiment aimed at unravelling these aspects.
Participants in the study (N = 34, with some
knowledge in Web page development but with a
limited experience with IoT) were recruited
during a Digital Humanities university course.
They were first introduced to IoT and TAP
concepts, and some common examples of
automation were presented to them. In a second
meeting one week after, the concepts were
recalled, and they were presented with the tasks to
perform. Their assignment was to think about six
automations that can be useful for daily living in
their environment and define them first using a
natural language description, as they would
express them more spontaneously, and then a
formalized structure. For the structure, we refer to
the event-condition-action paradigm, as from
previous studies it emerged that it is suitable for
the configuration of IoT automations [
The provided structure required the name of
the rule, its high-level goal, and an optional
description of the context of activation [
To limit social pressure and to allow them to reason in a situated manner, participants had one week to accomplish the task. Also, to not force participants’ thinking into the provided rule structure, it was asked to first define automations using natural language descriptions and later convert them into the template.
Participants produced overall 204 automation rules, comprising 735 rule elements. Since a list of functionalities was not provided, they described the desired behaviours using a wealth of terms. To drive further analysis we hence examined the formal and natural language descriptions of the gathered automations and extracted a list of functionality classes (check the first column in Table 1). also to specify to which location each part of the automation refers (See Figure 1, lower part).
From the table, we can observe the high frequencies of “scheduling” (in the trigger part it refers to a generic time or date, or a fixed date like “event in the calendar for tomorrow”, while in the action part can be for instance “add the delivery date to the calendar”), “presence” (detection or non-detection in a room, home or away from home), and “notifications” (received on the smartphone, speaker, but also using a “signal lamp” in some cases) classes, the first two being the most used triggers, and the third the most common action.
Participants could use one or two terms to describe the final goal for the automation. Like before, we grouped the goals based on the most frequent terms and conceptual proximity (see Figure 1, upper part). In general, we found the goal to be much centred in a few classes, and that comfort is by far the most used goal, present in around half of the automations. Participants had
It emerges that most of the rule parts refer to the whole house, or no specific place (such as a notification received on the phone). However, they also mentioned specific rooms, locations outside the house, and also the possibility to link the automation to the room the user is currently in.
Some patterns emerge from the gathered data. Participants adopted two main approaches to describe automations using natural language. The first is a more direct style, asking the system to do the actions (“then activate”; “make the fridge create recipes”; “tell Alexa to do this”). The second is an impersonal approach, describing the changes that will then occur in the environment as the result of the action part (“the watering mechanism is activated”; “the cameras turn on and alarm notifications are activated”). Furthermore, automations were described using either a drier, rule-like approach (“if this, then do that”) or a descriptive one (“assuming the house has an entrance door with a smart handle, make sure that…”). Another distinction emerges in the triggers descriptions. In some cases, they directly used the verb to define what is occurring (“if the user is studying”; “if it’s raining”; “when it’s 8:00”; “if the kitchen temperature is below 17 degrees”). In other cases, they referred to the object that performs the sensing (“when the bed sensor detects the user”; “when the sensor measures soil humidity below 60%”). The brands of devices were not frequently used by participants, with the exception of voice assistants and gaming consoles.
Concerning the use of the template, participants formalised automations starting from the trigger part (mostly events first) and then the action part. The “And” operator was the most used to join a trigger functionality with its value and also as a connector between a trigger or action and the next rule element (when applicable), whilst the “Equal” operator was the most used to link the actions functionalities and values. In some cases, participants modified the standard operators, e.g., using “And (after 5 minutes)” as “next operator” between two actions.
An aspect that stands out is that a significant portion of the defined automations (more than 20%) cannot be expressed using a standard rule construct (we considered “standard” rules consisting of one or more events jointed with the “or” operator, eventually one or more conditions joined by the “and” or “or” operators, and one or more actions; alternatively, a structure with one or more conditions with one or more actions, where the start of a condition acts as the event). The most common non-standard construct identified in automation was related to timing aspects of the trigger parts, for instance, requiring to check whether a condition lasts for a specific amount of time; if an event has not been detected in a period; if something has not occurred in an interval after another event; or if it verifies in an interval before another event. Other found non-standard constructs were the timing aspects for actions (delayed notifications or actions gradually performed in steps), the programming-like constructs (“if-then-else” rules, complex Boolean conditions requiring parenthesis, counters), the use of routines and rule concatenations (“activate this action, then do this other check”), and the use of groups (“all the kids’ devices”). Some examples of these non-standard automations are: If the windows in the house are closed for more than 24 hours, and the air purifier detects high levels of CO2, the windows open and the purifier starts sanitizing the air.
This automation requires a check to be performed at the end of the 24 hours.
When it's 21:00, if the user has not yet called their grandparents' home number, send them a text message with the message «How are you?», then after 5 minutes switch off the TV and start a call to their number.
This automation needs a “wait” between the first and the other actions.
If the backrest is raised and our user does not get up, then after five minutes the alarm rings and the backrest starts to vibrate.
The participant specified that the five-minute check is a cyclical operator to be repeated every 5 minutes.
When the dog is hungry, he can press the button adjacent to the bowl up to twice a day to fill it with an exact dose of food. Once the button is pressed, the bowl will emit the voice message “Bravo” and then release the food.
A counter is required to implement this functionality.
About the limitation of this work, it should be noted that participants have similar backgrounds and ages (Masters’s degree students between 23 and 29 years old). This could have influenced the variety and the choice of functionalities for the produced automation rules.
We introduce and discuss a user study aimed at collecting automation rules without imposing device or interface constraints, using both a natural language and a formalized structure. We also reported on a preliminary high-level analysis of the collected data.
Participants used different approaches in
describing the desired automations using natural
language. However, they mainly used standard
operators while formalizing them. A challenge
that emerges is that a significant number of
automations require advanced constructs, often
time-related. In future work, we will study how to
allow users to define these temporal aspects (e.g.,
using Allen’s interval Algebra [
This work has been supported by the PRIN 2017 “EMPATHY: Empowering People in Dealing with Internet of Things Ecosystems”, https://www.empathy-project.eu/