1 A Visual Paradigm for Defining Task Automation Carmelo Ardito Abstract Università di Bari, In the last years, researchers are devoting many efforts to via Orabona, 4 improve technological aspects of the Internet of Things 70125 Bari, Italy (IoT), while little attention has dedicated to social and carmelo.ardito@uniba.it practical sides. Professional developers program the behavior of smart objects. In addition, often the Giuseppe Desolda functionality exposed by a single object are not able, alone, Università di Bari to exhaustively support the end users’ tasks. The via Orabona, 4 opportunities offered by IoT can be amplified if new high- 70125 Bari, Italy level abstractions and interaction paradigms enable also giuseppe.desolda@uniba.it non-technical users to compose the behavior of multiple objects. To fulfill this goal, we present a model to express Maristella Matera rules for smart object composition, which includes new DEI – Politecnico di Milanoo operators for defining rules coupling multiple events and Piazza Leonardo da Vinci, 32 conditions exposed by smart objects, and for defining 20134 – Milano, Italy temporal and spatial constraints on rule activation. Such matera@elet.polimi.it model has been implemented in a Web application whose composition paradigm has been designed during an elicitation study with 25 participants. Author Keywords End-User Development of Mashups, Visual Paradigms for ECA Rule Expression, Internet of Things. General Terms Design, Human Factors. Introduction The current integrated technologies confer intelligence to Copyright is held by the author/owner(s). any type of objects (also called things in this document) and AVI, June 07–10, 2016, Bari, Italy. connect them to the network. For example, a pressure sensor can make an office door handle smart, for example alerting a user located in another part of the world every 2 time someone comes into that office. The added value is IoT Challenges that the object, being part of a network of smart things (the From the technological point of view, such a large number of Internet of Things (IoT) can communicate with other objects things requires an adequate network infrastructure and or services, thus triggering dynamic behavior. If the door efficient communication protocols, especially due to the fact sensor is able to communicate with a sensor worn by a that integrated devices have very limited resources (e.g., person, than a remote user can also know who entered the CPU, RAM, memory, battery). Further issues, such as office. privacy (e.g., in the smart door scenario, if and when it is allowed to know who entered into the office) and safety One of the opportunities of the IoT is especially the (e.g., if a hacker is able to break a device, then it could possibility to collect in real time information concerning access all the devices it is connected with) must be events and behaviors happening in the real world. For addressed. example, a potential advantage of the IoT is to anticipate the needs of a human even before he is aware of it. From the HCI point of view, a major challenge is to enable even non-technical users to manipulate data and Today these scenarios may appear unrealistic since we are functionality of things in a simple and natural way. Today, in not pervaded with a significant number of sensors. However, fact, this is a prerogative almost always reserved to Cisco’s forecasting estimates that more than 50 billion smart developers who, through the use of specific programming objects will be deployed by 20201. There are already many languages, provide pre-packaged solutions to users. The contexts in which the IoT is adopted. For example, for most important challenge is to allow non-technical users to creating wearable devices, i.e. clothing and accessories such define and manage the connections between things, which as bracelets, watches, t-shirts, rings, shoes, can integrate represent the real benefit of IoT, especially in the next years sensors capable of detecting physiological parameters and when a growing amount of things will be available. actuators to communicate with the users. Domotics is another very active IoT context: pressure, volumetric and Some works in the literature propose mashup techniques for distance sensors, as well as infrared cameras, are able to addressing this issue. For example in [1] and [3] the ensure the security of homes; they permit a remote user to authors introduce two systems for the mashup of things for interact with the devices installed in the house (e.g., lights, home automation, both consisting of two design heating, blinds) and assist user to avoid hazards such as environments: one is devoted to electrical engineers who floods, fires or explosions. Other sectors that are define the behavior of devices through a visual beneficiating of the advent of IoT are Smart Cities, industry representation of logic operations and algebraic formulas; and environment through energy saving. the second one allows non-technical users to create a web page where they can include widgets to display data coming from things and synchronize their behavior based on a “wired” composition paradigm. The problem is however that several studies have demonstrated that this kind of 1 http://newsroom.cisco.com/feature- composition paradigm is not suitable for non-expert users content?type=webcontent&articleId=1208342 [5, 6, 8] as it forces them to deal with concepts like data 3 flow and parameter passing which cannot be mastered by In fact, most of these platforms currently used in the IoT people who are not expert in programming. One of the domain do not permit (for example IFTTT) or make it products available in the market is gluethings difficult (for example gluethings) to specify multiple events (http://www.gluethings.com), a Web platform for registering and actions. Similar difficulties arise when specifying and composing things. Unfortunately, also this system offers temporal or spatial constraints, e.g., to define the following a wire-based composition paradigm and also require users behavior: “If I’m in Rome and I post an image on Instagram to handle JSON to set parameters for the low-level behavior between 8.00 and 11.00 a.m., post the same picture also on of things – such practices are out of reach for laypeople, i.e., Twitter and Tumblr”. those users which represent the actual business opportunity. In order to identify composition paradigms able to guide To determine the success of the IoT, it is necessary to users in the definition of articulated rules, we wanted to investigate new approaches that, thanks to high-level elicit the end-user mental model, which is an aspect scarcely abstractions, can enable non-expert users to compose data explored in the field of task automation, as pointed out by and functionality of things, as well as the communication [7]. As the seed of our investigation, we were inspired by among them, by means of “natural” composition paradigm. the 5W model, which is adopted in several domains, such as journalism and customer analysis, and more in general in Outlook on the composition paradigm problem solving, to analyze the complete story about a fact. As mentioned above, things can be treated as services, It suggests describing a fact by answering the following because sensors and actuators have an URI that identify questions: them on the Internet. In the case of service mashups, platforms implementing an event-driven approach, as the  Who did it? one described in [4], permit to synchronize Web services so  What happened? that the event produced by/on a service (e.g., selection of a  When did it take place? word in a text) triggers an action of another service (e.g., a search using the selected word). We believe that an event-  Where did it take place? driven paradigm is suitable also for the manipulation of  Why did it happen? things offering the advantage of enabling the composition of We adopted the 5W model in an elicitation study with 25 things among themselves and also with other Web services. participants aimed at identifying, with the help of users, a Similar event-driven platforms (e.g., IFTT - notation for the specification of task automation rules. The https://ifttt.com/) are now emerging also for task customization of the 5W model, which we called Rule_5W, automation. They make it easy to connect two services helped us to highlight the elements that are essential for (things or APIs), choose a trigger, and thus create an action. creating complete meaningful rules for smart object They look very promising thanks to the simple and effective composition. In the Rule_5W model, “Who” is replaced by composition paradigm, even if it is intended for expert users “Which” for specifying the services involved in a rule. “What” and therefore requires programming skills. indicates the triggered events, as well as the actions to be 4 11 2 3 Figure 1. EFESTO: the interface for rule creation. activated. “When” and “Where” refer to the specification of, area in which a rule is defined. The left side is for specifying respectively, temporal and spatial conditions for triggering the triggering events, and the right side is to define the events and performing actions. Finally, “Why” is used for actions to be activated by the selected services. reporting a short description to explain the rule behavior to A wizard procedure, activated by the green “+” button a human reader, e.g., other users with whom the rule is highlighted by circle 2 in Errore. L'origine riferimento possibly shared. The paradigm has then been implemented non è stata trovata., guides the users in defining the in a Web platform that extends the capability of EFESTO, a events in a full-automated fashion. The wizard sequentially platform for the End-User Development (EUD) of Web shows some pop-up windows in which the service, the mashups through which data provided by Web APIs can be events and the conditions are specified. According to integrated into unified visualizations [2]. WYSIWYG approach, the wizard steps allow the user to EFESTO and its 5Ws Composition Paradigm define an event in terms of Which is the service to be In this section we illustrate through an example the main monitored for detecting the triggering event, What service features of the 5Ws composition paradigm. A user, who we event has to be monitored, When and Where the event has suppose is a female, creates a rule to automatically turn on to occur. The specification of When and Where conditions is the coffee machine and roll-up the shutters when her smart optional. At the end of the wizard procedure, the event is bracelet detects that she has just woken up or the smart defined and its summary appears under the “Events” area. alarm clock rings. To create this rule, the user clicks the Actions can be defined by clicking on the green “+” button “New Rule” button in the navigation bar (Errore. L'origine highlighted (circle 3 in Errore. L'origine riferimento non riferimento non è stata trovata., circle 1) and the è stata trovata.). The button activates a wizard that helps “Creating Rule” interface appears. The UI shows the main the user define an action in terms of Which 5 Figure 2. EFESTO: example of rule including two events and two actions. service will execute the action as a consequence of the the execution of the rule action(s) if all conditions of all event(s), What action the service has to perform and When events are satisfied. The “Or” button determines the and Where the action can be performed. definition of a new event that will cause the execution of the rule action(s), if the conditions of at least one event are In EFESTO, users may either define first all the events and satisfied. Once the rule is created (see Figure 2), it can be then the actions, or define first a basic rule with one event saved by entering a short description of the rule (the Why in and one action and then include new events and new the 5W model). actions. Events and actions can be added or removed at any time fostering a dynamic modification of the running Conclusion mashup. Further events can be added by clicking one of the One of the cornerstones of the future of the IoT will be to two green “+” buttons labeled And / Or. Choosing the “And” put in the hands of the end users simple software tools button starts the definition of a new event that will cause capable of making natural and powerful composition 6 between things. It is unimaginable that this possibility is [7] Ur, B., McManus, E., Ho, M. P. Y. and Littman, M. L. Practical reserved for a few experts. As already happened in the past trigger-action programming in the smart home. Proc. CHI with software tools (forums, social networks, CMS) that '14. ACM (2014), 803-812. made users to evolve from simple consumer to prosumer, [8] Wajid, U., Namoun, A. and Mehandjiev, N. Alternative even for the IoT have to be designed tools suitable even for Representations for End User Composition of Service-Based non-expert users. This paper has illustrated how a generic Systems. In End-User Development - ISEUD '11. LNCS 6654. platform for service mashups can be specialized for the Springer, 53-66, 2011. composition of services that enable accessing/controlling smart things. We are currently working on implementing the needed extensions. A demo, illustrating some preliminary results, will be given during the workshop. References [1] Blackstock, M. and Lea, R. IoT mashups with the WoTKit. Proc. IOT '12. IEEE (2012), 159-166. [2] Desolda, G., Ardito, C. and Matera, M. EFESTO: A platform for the End-User Development of Interactive Workspaces for Data Exploration. In Rapid Mashup Development Tools - Rapid Mashup Challenge in ICWE 2015. 591. Springer Verlag, 63 - 81, 2015. [3] Guinard, D., Trifa, V., Mattern, F. and Wilde, E. From the Internet of Things to the Web of Things: Resource-oriented Architecture and Best Practices. In Architecting the Internet of Things. 1. Springer Berlin Heidelberg, 97-129, 2011. [4] Matera, M., Picozzi, M., Pini, M. and Tonazzo, M. PEUDOM: A Mashup Platform for the End User Development of Common Information Spaces. In Web Engineering - ICWE '13. LNCS 7977. Springer, 494-497, 2013. [5] Namoun, A., Nestler, T. and De Angeli, A. Conceptual and Usability Issues in the Composable Web of Software Services. In Current Trends in Web Engineering - ICWE '10. LNCS 6385. Springer, 396-407, 2010. [6] Payne, S. J. Users’ Mental Models: The Very Ideas. In HCI Models, Theories, and Frameworks - Toward a Multidisciplinary Science. Morgan Kaufmann, 135-156, 2003.