The Internet of Things (IoT) enables the creation of Interactive Smart Spaces (ISSs) where diferent types of digital devices are integrated in the ambient or embedded in physical objects, and can sense human actions to control equipment, modify environmental parameters, or create multi-sensory efects. These IoT-enhanced interactive systems can support human activities in diferent contexts, e.g., education, entertainment, home assistance, rehabilitation, to name a few. We argue that a human-centered perspective in the design of ISSs is needed to take into account some salient characteristics of these systems. New conceptual modeling issues also need to be investigated to go beyond representing hardware, software, and connectivity features of IoT devices and to capture the user interaction.
In the last decades researchers have proposed diferent modelling approaches for IoT systems related to Smart Spaces. Four major topics emerge: privacy and security of data transmission [7, 8, 9], orchestration of device behaviour [10, 11, 12, 13], data gathering and propagation [14, 15], and design of single devices and smart objects [16, 11]. The role of the user is neglected or simply considered as a pure source of data, and existing approaches take into account only marginally (or not at all) the need of modeling human interaction in Interactive Smart Spaces. We instead argue that a human-centered perspective is needed to take into account some salient characteristics of these systems. As illustrated in the following section, new modeling issues also need to be investigated to go beyond representing hardware, software, and connectivity features of IoT devices and capture the user interaction.
The main abstractions of our conceptual approach are organized in two main sub-models: the Structural Model and the Interactive Behaviour Model. The Structural Model supports the representation of the human and technological “actors”, their interaction capability, i.e., which actions they can perform and perceive (“sense”) and which perceivable efects they can generate (“actuate”), as well as the digital contents that are involved in the user experience. The Interactive Behavior Model supports the representation of the interactive behaviour of all actors and how cross-interactions are orchestrated for the users to perform tasks and activities at diferent levels of complexity.
These two models derive from the extensive experience gained during the design of the Magic
Room, a sophisticated multi-sensory smart space for children’s play, learning, and rehabilitation that
we developed in the context of a national project and installed at two local schools and two therapeutic
centers in Italy[
The structural model is built around the notions of Actor and Digital Resource. Actors are the building blocks of any ISS as their properties and their cross interactions enact the interactive experience. There are two categories of Actors: Technological Actors, characterized by a mix of digital and physical features, and Human Actors, i.e., the users. The Human Actors in the Magic Room are Child and Caregiver: the former interact with the smart space and the smart objects, the latter operate on a tablet application to control the activity execution and flow. The Technological Actors are the above mentioned devices and a number of Smart Objects: Smart Toys (embedding a variety of motion and pressure sensors and light or sound actuators), Paper- or plastic-based Identifiable Objects (RFID tagged items of diferent shapes), and a Smart Sphere that embeds a sensor ( Near Object Detector) able to generate an object identifier when it detects the proximity of Identifiable Elements - physical items enriched with RFID tags).
At a high-level, technological actors are represented in terms of their interactive capability, as exemplified for the Smart Sphere in the Figure 2. Figure 1 illustrates an excerpt of Structural Model concerning the representation of the User Action Selection associated with the Child Human Actor. The action is specified first by describing what can be selected, either visual content or physical content. Then the diagram reports the basic actions through which Selection is performed, also in relationship with the Technological Actors enabling such actions.
The diagrams reported in Figure 3 and 4 represent a portion of the Interactive Behaviour Model for the activity “Battleship game”. It focuses on the Scene handling a turn of play. The defined flow of Interaction Tasks includes the selection of a cell in the battleship grid, which generates a feedback to the user, e.g., partial or full hits, water hits, end of game, and a request for further input. Figure 4 shows the details (i.e., the fragment of the Extended Activity Interaction Model) related to Interaction Task “getCellCoordinates”, which refers to the selection of a cell in the battleship grid.
The diagram presents two alternative modalities to accomplish this Interaction Task, each one envisioning Interactions and Efects . The user can "point on" a virtual content (a cell of the Battle grid projected on the front screen) or can use the Smart Sphere, placing on top of it two Identifiable Cards for the cell coordinates, one with a number and one with a letter. Each box associated to an Interaction shows the Human Actor who executes the action(s) (Initiator) on the upper area of the box and the Participants, i.e., the Technological Actors involved in the interaction on the bottom area (Smart Sphere and Identifiable objects). The middle area of the box is devoted to specify the ECA rules, omitted in the figure for lack of space. Efects are represented by boxes where the top area is empty, being the stimuli initiated by system events without intentional human intervention. The enactment of these efects is triggered by a system event related to the status of the task execution.
A solution that eases the flexible definition of the interactive activities allows the ISS designers to overcome the barrier of single-purpose implementation and also enables the installation and execution of multiple activities. The abstractions presented in the previous sections guided the definition of a multi-layer architecture for the Magic Room characterized by modularity, flexibility, and extensibility. The Activity specifications in the Interactive Behavioural Model and the Technology Actor specifications in the Structural Model guides the definition of a JSON-based configuration file that the Execution Engine interprets as the low level rules governing the interactive capabilities of the Technological Actors.
This work has discussed some modeling requirements that characterize of Interactive Smart Spaces
(ISSs), pinpointing the importance of considering the human as the principal actor in this class of
systems, and addressing the interaction capabilities as fundamental for the empowering users in this
spaces and enabling the accomplishment of complex tasks [20]. Our model and our overall approach
can pave the ground towards innovative methods of conceptual design in the IoT arena, and may also
lead to the definition of more modular and standardized technological architectures for future highly
interactive IoT systems. An interaction-centered modeling approach is also the first step towards
the definition of novel solutions to support the appropriation process of interactive IoT technology
for end users, and adequate abstractions like the ones presented in this paper can provide a base of
concepts upon which to create the building blocks for End-User Development methods and tools [
of CHI ’20, 2020, pp. 1–14. [19] M. Gelsomini, et al., Magika, a multisensory environment for play, education and inclusion, in: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems, 2019, pp. 1–6. [20] C. Ardito, P. Bottoni, M. F. Costabile, G. Desolda, M. Matera, M. Picozzi, Creation and use of service-based distributed interactive workspaces, J. Vis. Lang. Comput. 25 (2014) 717–726. URL: https://doi.org/10.1016/j.jvlc.2014.10.018. doi:10.1016/j.jvlc.2014.10.018.