When IoT technology were first proposed, the emphasis was on the creation of arrays of distributed connected sensors (Wireless Sensor Networks) to support automation systems relieving users from repetitive tasks (e.g., plants monitor or automatic controls). For this class of IoT systems, the interaction between the system and the user was of secondary priority. More recently, novel interactive systems have been proposed to empower people in different activities and contexts of everyday life [1,2,3,4,5]. A human-centered perspective has progressively emerged, in which the interactive capability of IoT-enhanced physical objects and spaces (hereinafter smart objects and smart spaces) becomes more and more central, raising new requirements and challenges for conceptual modelling. In this scenario, the scope of conceptual modeling goes beyond representing features related to hardware, software, connectivity, and communication among multiple devices, and also address the interactions between the users and the materials or spaces embedding such devices. This paper shortly discusses a first attempt to address this modelling challenge, which also highlights the potential of an interactioncentered modeling approach for research on End-User Development in the arena of interactive IoT systems. More details on the modeling approach can be found in [6].
EMPATHY: Empowering People in Dealing with Internet of Things Ecosystems. Workshop co-located with AVI 2020, Island of Ischia, Italy email: mattia.gianotti@polimi.it (M. Gianotti); fabiano.riccardi.polimi.it (F. Riccardi); giulia.cosentino@polimi.it (G. Cosentino); franca.garzotto@polimi.it (F. Garzotto); maristella.matera@polimi.it (M. Matera) orcid: 0000-0001-6035-3367 (M. Gianotti); 0000-0001-5510-3240 (F. Riccardi); 0000-0002-8560-4924 (G. Cosentino); 0000-0003-4905-7166 (F. Garzotto); 0000-0003-0552-8624 (M. Matera)
In the last decades researchers have proposed different modelling approaches for IoT systems related to Smart Spaces. Four major topics emerge: privacy and security of data transmission [7,8,9], orchestration of device behaviour [10,11,12,13], data gathering and propagation [14,15], and design of single devices and smart objects [16,11]. The role of the user is neglected or simply considered as a pure source of data, and existing approaches take into account only marginally (or not at all) the need of modeling human interaction in Interactive Smart Spaces. We instead argue that a human-centered perspective is needed to take into account some salient characteristics of these systems. As illustrated in the following section, new modeling issues also need to be investigated to go beyond representing hardware, software, and connectivity features of IoT devices and capture the user interaction.
The main abstractions of our conceptual approach are organized in two main sub-models: the Structural Model and the Interactive Behaviour Model. The Structural Model supports the representation of the human and technological "actors", their interaction capability, i.e., which actions they can perform and perceive ("sense") and which perceivable effects they can generate ("actuate"), as well as the digital contents that are involved in the user experience. The Interactive Behavior Model supports the representation of the interactive behaviour of all actors and how cross-interactions are orchestrated for the users to perform tasks and activities at different levels of complexity.
These two models derive from the extensive experience gained during the design of the Magic Room, a sophisticated multi-sensory smart space for children's play, learning, and rehabilitation that we developed in the context of a national project and installed at two local schools and two therapeutic centers in Italy [17,3,18,19]. In the following we present a limited example of the conceptual model of the Magic Room, extracted from the specification of one activity -Battleship, which is the classic battleship board game but enhanced in the smart space -making it more interactive and enormously engaging.
The structural model is built around the notions of Actor and Digital Resource. Actors are the building blocks of any ISS as their properties and their cross interactions enact the interactive experience. There are two categories of Actors: Technological Actors, characterized by a mix of digital and physical features, and Human Actors, i.e., the users. The Human Actors in the Magic Room are Child and Caregiver: the former interact with the smart space and the smart objects, the latter operate on a tablet application to control the activity execution and flow. The Technological Actors are the above mentioned devices and a number of Smart Objects: Smart Toys (embedding a variety of motion and pressure sensors and light or sound actuators), Paper-or plastic-based Identifiable Objects (RFID tagged items of different shapes), and a Smart Sphere that embeds a sensor (Near Object Detector) able to generate an object identifier when it detects the proximity of Identifiable Elements -physical items enriched with RFID tags).
At a high-level, technological actors are represented in terms of their interactive capability, as exemplified for the Smart Sphere in the Figure 2.
The diagrams reported in Figure 3 and 4 represent a portion of the Interactive Behaviour Model for the activity "Battleship game". It focuses on the Scene handling a turn of play. The defined flow of Interaction Tasks includes the selection of a cell in the battleship grid, which generates a feedback to the user, e.g., partial or full hits, water hits, end of game, and a request for further input. Figure 4 shows the details (i.e., the fragment of the Extended Activity Interaction Model) related to Interaction Task "getCellCoordinates", which refers to the selection of a cell in the battleship grid.
The diagram presents two alternative modalities to accomplish this Interaction Task, each one envisioning Interactions and Effects. The user can "point on" a virtual content (a cell of the Battle grid projected on the front screen) or can use the Smart Sphere, placing on top of it two Identifiable Cards for the cell coordinates, one with a number and one with a letter. Each box associated to an Interaction shows the Human Actor who executes the action(s) (Initiator) on the upper area of the box and the Participants, i.e., the Technological Actors involved in the interaction on the bottom area (Smart Sphere and Identifiable objects). The middle area of the box is devoted to specify the ECA rules, omitted in the figure for lack of space. Effects are represented by boxes where the top area is empty, being the stimuli initiated by system events without intentional human intervention. The enactment of these effects is triggered by a system event related to the status of the task execution.
A solution that eases the flexible definition of the interactive activities allows the ISS designers to overcome the barrier of single-purpose implementation and also enables the installation and execution of multiple activities. The abstractions presented in the previous sections guided the definition of a multi-layer architecture for the Magic Room characterized by modularity, flexibility, and extensibility. The Activity specifications in the Interactive Behavioural Model and the Technology Actor specifications in the Structural Model guides the definition of a JSON-based configuration file that the Execution Engine interprets as the low level rules governing the interactive capabilities of the Technological Actors.
This work has discussed some modeling requirements that characterize of Interactive Smart Spaces (ISSs), pinpointing the importance of considering the human as the principal actor in this class of systems, and addressing the interaction capabilities as fundamental for the empowering users in this spaces and enabling the accomplishment of complex tasks [20]. Our model and our overall approach can pave the ground towards innovative methods of conceptual design in the IoT arena, and may also lead to the definition of more modular and standardized technological architectures for future highly interactive IoT systems. An interaction-centered modeling approach is also the first step towards the definition of novel solutions to support the appropriation process of interactive IoT technology for end users, and adequate abstractions like the ones presented in this paper can provide a base of concepts upon which to create the building blocks for End-User Development methods and tools [2]. This issue is particularly important in contexts -like education and rehabilitation -where Interactive Smart Spaces could have a significant potential but there is a strong need of personalization of the experiences in these environments. This view raises new research challenges, addressing the way the interaction capabilities of ISS should be modelled through metaphors and design patterns that make sense to the users and would enable them to customize or even create from scratch their own interactive smart experiences.