Smart home installations, in which connected devices exchange data with each other, are becoming a widespread reality. These environments are based on automation rules, which may be specified even by end users. To make the process of automations creation and management easier, less prone to errors, and more engaging, we propose an approach based on mobile Augmented Reality (AR). In this paper, we describe how AR can be used to interact with the objects in the environment, and allow for the definition of coordinated behaviours between them. Starting with analysis of relevant literature and our previous experiences, we introduce four main aspects to consider when designing the user-application interaction, and how we are implementing them in a prototype AR automation management application. Further possibilities that AR can ofer in this context will be discussed as well.
Connected devices and services are always more present in our lives. We are often in
environments characterized by the presence of various types of devices, smart appliances, objects that
can exchange data between them, actualizing the Internet of Things (IoT) concept. One way to
exploit the new possibilities that the IoT ofers is using automations to make a dwelling more
personalized around its inhabitants’ needs. Generally speaking, an automation in the IoT context
can be defined in the form of a trigger-action rule, which activates a device’s functionality
or a service when a specific situation is detected, e.g., in the environment or another service.
Nowadays the most widespread application to define such rules is IFTTT 1. Automation rules
can use diferent formats that ofer diferent levels of expressivity. For example, IFTTT allowed
for rules comprised of a single trigger and a single action, and recently a commercial version
with more possibilities has been made available. Other approaches (e.g., [
Mattioli); https://giove.isti.cnr.it/Users/Fabio/index.html (F. Paternò) © 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings
Regardless of the adopted rule structure, automation rules are typically defined using visual
editors, but often such tools are not found to be immediate to understand and engaging, thus
their use is somewhat limited [
In the next section, we will define the background and introduce some related previous work. Then, the design of the proposed solution will be discussed with respect to the literature and to the challenges we faced so far. Next, further possibilities that can be explored will be introduced. Finally, we draw some conclusions and provide indications for future work.
The creation of an automation rule usually consists of selecting the desired functionality,
defining its parameters, and repeating this process for the various rule elements that comprise
the automation. As reported in the literature [
Previous research on augmented reality in smart homes focuses mostly on visualizing
information over a target object, controlling single devices, or creating simple rules [
We also based this investigation on our previous experience in mobile AR for smart homes
[
To ensure a seamless and pleasant experience, the representations of automations have to blend in a meaningful way with the real world, e.g., placing them over the objects they refer to. Objects can be identified using diferent techniques. The standard AR approach relies on the distinctive feature points of the object or uses specifically made images (markers). Another possibility is using machine learning, where object detection models such as YOLO3 or EficientDet [ 9] are used to detect the position of the object on the screen. The third approach is combining machine learning object detection with the AR frameworks capability of capturing the position of feature points and planes in the real world. Another solution is to not identify the objects during the use, but rather retrieve their positions as previously detected with another approach.
At the moment, the prototype is divided into two separated phases, “get object position” and “rule editor” (see Figure 1). In the “get object position” phase, the third (combined) approach is used to save the locations of objects that can be interacted with. In the “rule editor”, the positions of the objects are retrieved and used to generate a placeholder visualization (a lfoating exclamation mark) that indicates that they are ready for use. This allows for a more immediate interaction: when the application starts, the placeholder visualizations are already in the environment, not requiring a first identification stage or going near them to initiate the interaction.
Regarding rules expressivity, the goal is to allow for diferent automation structures to match the diferent possible user intents. Structures can be simple, comprised of one element in the trigger and one in the action part, or compound, with more elements in one part and/or in the other. Simple rule structures are the “event – action”, the “condition – action” and “not event – action” (when the check is whether an event has not been verified in a time interval). Compound rule structures regard multiple events (linked by the or operator), multiple conditions (using the and/or operators) and actions (using the sequential/parallel operators), possibly combined.
3https://pjreddie.com/darknet/yolo/
Regarding the AR visualizations of automations, the main concept is the definition of each part of the automation directly on the visualization related to that object or service. After a first definition of an automation part (e.g., “when the living room door is opened”), the user can continue to add more parts (e.g., the condition “and the living door window is open”, or the action “turn of the kitchen light”). Each selected rule element has its related representation, for example, a 3D activable switch is used with objects that can be turned on and of, while a slider for the selection of an integer value is displayed (see Figure 2). The connection between rule elements (Boolean operators, sequential or parallel operators for actions, and the link between the trigger and the action part of the rule) is also depicted. Connections can be rendered using wires or dynamic direction arrows between objects. Considering the size of a mobile device screen, this information must be presented in a balanced manner to not clutter the scene.
Describing automations in isolation, although useful, may not be enough. To be useful in a real situation where multiple automations can be active at the same time, a way to analyse them and to ensure that their efects are not in conflict should be introduced. For this reason, we are implementing a way to select and show multiple rules in the environment at the same time. If a conflict is detected (e.g., two contrasting actions triggered by diferent events) the user will be notified. A debugger, that allows simulating changes in the values of the objects to check for conflicts is also in development. The activation signal (originated from a simulated event) can visually show its efects on the other objects through the “wires” that connect the various rule elements.
Another research direction is how to integrate a recommendation system in the editor. Recommendations can suggest automations related to the current context: for example, about the selected rule element (as in [10]) or the detected activity. AR can be used to visualize the recommendations and their relations with already existing automations. Another possibility is to visualize the explanation of why a suggestion has been generated. However, specific inquiries about recommendations for AR are still in an initial phase (for example, [11]), and further research is required for investigating how to exploit them in order to support automation management.
In this position paper, we introduce a novel approach to automation rules that capitalizes on the relation between the objects and the environment, and make the automations perceivable through AR in order to improve their transparency and facilitate their management by end users. Our ultimate goal is not only to support end-user monitoring and creation of automations in isolation, but also to find an appropriate representation to define and make explicit the relations between objects. In future work, we will focus on the evaluation of this approach. A user test will be carried out to measure the usability of the prototype, the users’ willingness to adopt it, and the number and type of errors in the creation of automations. In addition, we will extend the application, integrating a debugger and a recommendation system specifically for its use.
This work has been supported by the PRIN 2017 “EMPATHY: Empowering People in Dealing with Internet of Things Ecosystems”, https://www.empathy-project.eu/.
Redondo, A modern approach to supporting program visualization: from a 2D notation to 3D representations using augmented reality, Multimed Tools Appl 80 (2021) 543–574. doi:1 0 . 1 0 0 7 / s 1 1 0 4 2 - 0 2 0 - 0 9 6 1 1 - 0 . [9] M. Tan, R. Pang, Q. V. Le, EficientDet: Scalable and Eficient Object Detection, in: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, Seattle, WA, USA, 2020, pp. 10778–10787. doi:1 0 . 1 1 0 9 / C V P R 4 2 6 0 0 . 2 0 2 0 . 0 1 0 7 9 . [10] A. Mattioli, F. Paternò, Recommendations for creating trigger-action rules in a block-based environment, Behaviour & Information Technology 40 (2021) 1024–1034. doi:1 0 . 1 0 8 0 / 0 1 4 4 9 2 9 X . 2 0 2 1 . 1 9 0 0 3 9 6 . [11] W. Büschel, A. Mitschick, R. Dachselt, Here and Now: Reality-Based Information Retrieval: Perspective Paper, in: Proceedings of the 2018 Conference on Human Information Interaction&Retrieval - CHIIR ’18, ACM Press, New Brunswick, NJ, USA, 2018, pp. 171–180. doi:1 0 . 1 1 4 5 / 3 1 7 6 3 4 9 . 3 1 7 6 3 8 4 .