Conceptual Modeling of Multisensory Smart Spaces Mattia Gianottia , Fabiano Riccardia , Giulia Cosentinoa , Franca Garzottoa and Maristella Materaa a Politecnico di Milano, Milan, Italy Abstract The Internet of Things (IoT) enables the creation of Interactive Smart Spaces (ISSs) where different types of digital devices are integrated in the ambient or embedded in physical objects, and can sense human actions to control equipment, modify environmental parameters, or create multi-sensory effects. These IoT-enhanced interactive systems can support human activities in different contexts, e.g., education, entertainment, home assistance, rehabilitation, to name a few. We argue that a human-centered perspective in the design of ISSs is needed to take into account some salient characteristics of these systems. New conceptual modeling issues also need to be investigated to go beyond representing hardware, software, and connectivity features of IoT devices and to capture the user interaction. Keywords Interactive Smart Spaces, Interactive IoT Smart Objects, Interaction modeling, 1. Introduction When IoT technology were first proposed, the emphasis was on the creation of arrays of distributed connected sensors (Wireless Sensor Networks) to support automation systems relieving users from repetitive tasks (e.g., plants monitor or automatic controls). For this class of IoT systems, the inter- action between the system and the user was of secondary priority. More recently, novel interactive systems have been proposed to empower people in different activities and contexts of everyday life [1, 2, 3, 4, 5]. A human-centered perspective has progressively emerged, in which the interactive capa- bility of IoT-enhanced physical objects and spaces (hereinafter smart objects and smart spaces) becomes more and more central, raising new requirements and challenges for conceptual modelling. In this scenario, the scope of conceptual modeling goes beyond representing features related to hardware, software, connectivity, and communication among multiple devices, and also address the interactions between the users and the materials or spaces embedding such devices. This paper shortly discusses a first attempt to address this modelling challenge, which also highlights the potential of an interaction- centered modeling approach for research on End-User Development in the arena of interactive IoT systems. More details on the modeling approach can be found in [6]. EMPATHY: Empowering People in Dealing with Internet of Things Ecosystems. Workshop co-located with AVI 2020, Island of Ischia, Italy email: mattia.gianotti@polimi.it (M. Gianotti); fabiano.riccardi.polimi.it (F. Riccardi); giulia.cosentino@polimi.it (G. Cosentino); franca.garzotto@polimi.it (F. Garzotto); maristella.matera@polimi.it (M. Matera) orcid: 0000-0001-6035-3367 (M. Gianotti); 0000-0001-5510-3240 (F. Riccardi); 0000-0002-8560-4924 (G. Cosentino); 0000-0003-4905-7166 (F. Garzotto); 0000-0003-0552-8624 (M. Matera) © 2020 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings http://ceur-ws.org ISSN 1613-0073 CEUR Workshop Proceedings (CEUR-WS.org) Figure 1: Excerpts from the Structural Model of the Magic Room: the specification of a User Action, Selection, by Human Actor Child. 2. Related work In the last decades researchers have proposed different modelling approaches for IoT systems related to Smart Spaces. Four major topics emerge: privacy and security of data transmission [7, 8, 9], or- chestration of device behaviour [10, 11, 12, 13], data gathering and propagation [14, 15], and design of single devices and smart objects [16, 11]. The role of the user is neglected or simply considered as a pure source of data, and existing approaches take into account only marginally (or not at all) the need of modeling human interaction in Interactive Smart Spaces. We instead argue that a human-centered perspective is needed to take into account some salient characteristics of these systems. As illustrated in the following section, new modeling issues also need to be investigated to go beyond representing hardware, software, and connectivity features of IoT devices and capture the user interaction. 3. Modeling dimensions The main abstractions of our conceptual approach are organized in two main sub-models: the Struc- tural Model and the Interactive Behaviour Model. The Structural Model supports the representation of the human and technological “actors”, their interaction capability, i.e., which actions they can per- form and perceive (“sense”) and which perceivable effects they can generate (“actuate”), as well as the digital contents that are involved in the user experience. The Interactive Behavior Model supports the representation of the interactive behaviour of all actors and how cross-interactions are orchestrated for the users to perform tasks and activities at different levels of complexity. These two models derive from the extensive experience gained during the design of the Magic Room, a sophisticated multi-sensory smart space for children’s play, learning, and rehabilitation that we developed in the context of a national project and installed at two local schools and two therapeutic centers in Italy[17, 3, 18, 19]. In the following we present a limited example of the conceptual model of the Magic Room, extracted from the specification of one activity - Battleship, which is the classic battleship board game but enhanced in the smart space - making it more interactive and enormously engaging. Figure 2: The high-level specification of a Technological Actor, Smart Sphere. 3.1. Structural Model The structural model is built around the notions of Actor and Digital Resource. Actors are the build- ing blocks of any ISS as their properties and their cross interactions enact the interactive experience. There are two categories of Actors: Technological Actors, characterized by a mix of digital and phys- ical features, and Human Actors, i.e., the users. The Human Actors in the Magic Room are Child and Caregiver: the former interact with the smart space and the smart objects, the latter operate on a tablet application to control the activity execution and flow. The Technological Actors are the above mentioned devices and a number of Smart Objects: Smart Toys (embedding a variety of motion and pressure sensors and light or sound actuators), Paper- or plastic-based Identifiable Objects (RFID tagged items of different shapes), and a Smart Sphere that embeds a sensor (Near Object Detector) able to generate an object identifier when it detects the proximity of Identifiable Elements - physical items enriched with RFID tags). At a high-level, technological actors are represented in terms of their interactive capability, as exemplified for the Smart Sphere in the Figure 2. Figure 1 illustrates an excerpt of Structural Model concerning the representation of the User Action Selection associated with the Child Human Actor. The action is specified first by describing what can be selected, either visual content or physical content. Then the diagram reports the basic actions through which Selection is performed, also in relationship with the Technological Actors enabling such actions. 3.2. Interactive Behaviour Model The diagrams reported in Figure 3 and 4 represent a portion of the Interactive Behaviour Model for the activity “Battleship game”. It focuses on the Scene handling a turn of play. The defined flow of Interaction Tasks includes the selection of a cell in the battleship grid, which generates a feedback to the user, e.g., partial or full hits, water hits, end of game, and a request for further input. Figure 4 shows the details (i.e., the fragment of the Extended Activity Interaction Model) related to Interaction Task “getCellCoordinates”, which refers to the selection of a cell in the battleship grid. The diagram presents two alternative modalities to accomplish this Interaction Task, each one envisioning Interactions and Effects. The user can "point on" a virtual content (a cell of the Battle Figure 3: Activity Interaction Model of Battleship game. Figure 4: Expanded Activity Interaction Model for the Interaction Task enabling the selection of cell coordi- nates. grid projected on the front screen) or can use the Smart Sphere, placing on top of it two Identifiable Cards for the cell coordinates, one with a number and one with a letter. Each box associated to an Interaction shows the Human Actor who executes the action(s) (Initiator) on the upper area of the box and the Participants, i.e., the Technological Actors involved in the interaction on the bottom area (Smart Sphere and Identifiable objects). The middle area of the box is devoted to specify the ECA rules, omitted in the figure for lack of space. Effects are represented by boxes where the top area is empty, being the stimuli initiated by system events without intentional human intervention. The enactment of these effects is triggered by a system event related to the status of the task execution. 3.3. From Models to Software Architectures A solution that eases the flexible definition of the interactive activities allows the ISS designers to overcome the barrier of single-purpose implementation and also enables the installation and execu- tion of multiple activities. The abstractions presented in the previous sections guided the definition of a multi-layer architecture for the Magic Room characterized by modularity, flexibility, and exten- sibility. The Activity specifications in the Interactive Behavioural Model and the Technology Actor specifications in the Structural Model guides the definition of a JSON-based configuration file that the Execution Engine interprets as the low level rules governing the interactive capabilities of the Technological Actors. 4. Conclusion and Future Work This work has discussed some modeling requirements that characterize of Interactive Smart Spaces (ISSs), pinpointing the importance of considering the human as the principal actor in this class of systems, and addressing the interaction capabilities as fundamental for the empowering users in this spaces and enabling the accomplishment of complex tasks [20]. Our model and our overall approach can pave the ground towards innovative methods of conceptual design in the IoT arena, and may also lead to the definition of more modular and standardized technological architectures for future highly interactive IoT systems. An interaction-centered modeling approach is also the first step towards the definition of novel solutions to support the appropriation process of interactive IoT technology for end users, and adequate abstractions like the ones presented in this paper can provide a base of concepts upon which to create the building blocks for End-User Development methods and tools [2]. This issue is particularly important in contexts - like education and rehabilitation - where Interactive Smart Spaces could have a significant potential but there is a strong need of personalization of the experiences in these environments. This view raises new research challenges, addressing the way the interaction capabilities of ISS should be modelled through metaphors and design patterns that make sense to the users and would enable them to customize or even create from scratch their own interactive smart experiences. References [1] F. Delprino, C. Piva, G. Tommasi, M. Gelsomini, N. Izzo, M. Matera, Abbot: a smart toy motivat- ing children to become outdoor explorers, in: Proc. of AVI’18, 2018, pp. 1–9. [2] G. Desolda, C. Ardito, M. Matera, Empowering end users to customize their smart environments: model, composition paradigms, and domain-specific tools, ACM Transactions on Computer- Human Interaction (TOCHI) 24 (2017) 1–52. [3] F. Garzotto, M. Gelsomini, Magic room: A smart space for children with neurodevelopmental disorder, IEEE Pervasive Computing 17 (2018) 38–48. [4] D. Petrelli, M. Lechner, The mesch project–material encounters with digital cultural heritage: Reusing existing digital resources in the creation of novel forms of visitor’s experiences, Proc. CIDOC’14 (2014). [5] G. Desolda, C. Ardito, H. Jetter, R. Lanzilotti, Exploring spatially-aware cross-device interaction techniques for mobile collaborative sensemaking, Int. J. Hum. Comput. Stud. 122 (2019) 1–20. URL: https://doi.org/10.1016/j.ijhcs.2018.08.006. doi:10.1016/j.ijhcs.2018.08.006. [6] M. Gianotti, R. Fabiano, G. Cosentino, F. Garzotto, M. Matera, Modeling interactive smart spaces, in: Proceedings of ER 2020, 2020, p. In print. [7] M. F. Arruda, R. F. Bulcão-Neto, Toward a lightweight ontology for privacy protection in IoT, in: Proceedings of SAC’ 19, ACM, 2019, pp. 880–888. [8] D. Sahinel, C. Akpolat, O. C. Görür, F. Sivrikaya, Integration of human actors in IoT and CPS landscape, in: Proc. of WF-IoT, 2019-04, pp. 485–490. [9] A. Skarmeta, J. L. Hernández-Ramos, J. B. Bernabe, A required security and privacy framework for smart objects, in: 2015 ITU Kaleidoscope: Trust in the Information Society (K-2015), 2015-12, pp. 1–7. [10] A. Bassi, M. Bauer, M. Fiedler, R. van Kranenburg, S. Lange, S. Meissner, T. Kramp, Enabling things to talk, Springer Nature, 2013. [11] G. Fortino, W. Russo, C. Savaglio, W. Shen, M. Zhou, Agent-oriented cooperative smart objects: From IoT system design to implementation, IEEE Transactions on Systems, Man, and Cybernet- ics: Systems 48 (2018-11) 1939–1956. [12] M. Maheswaran, J. Wen, A. Gowing, Design of a context aware object model for smart spaces, things, and people, in: Proc. of IEEE ICC’15, 2015-06, pp. 710–715. [13] D. Ning, Y. Wang, J. Guo, A data oriented analysis and design method for smart complex software systems of IoT, in: Proc. of ISSI’18, 2018-09, pp. 1–6. [14] B. Costa, P. F. Pires, F. C. Delicato, Modeling SOA-based IoT applications with SoaML4iot, IEEE, 2019-04, pp. 496–501. [15] K. Jahed, J. Dingel, Enabling model-driven software development tools for the internet of things, in: Proc. of MiSE ’19, IEEE Press, 2019, pp. 93–99. [16] M. Bermudez-Edo, T. Elsaleh, P. Barnaghi, K. Taylor, IoT-lite: A lightweight semantic model for the internet of things and its use with dynamic semantics, Personal Ubiquitous Comput. 21 (2017-06) 475–487. [17] F. Garzotto, E. Beccaluva, M. Gianotti, F. Riccardi, Interactive multisensory environments for primary school children, in: Proc. of CHI ’20, CHI ’20, Association for Computing Machinery, New York, NY, USA, 2020, p. 1–12. [18] M. Gelsomini, G. Leonardi, F. Garzotto, Embodied learning in immersive smart spaces, in: Proc. of CHI ’20, 2020, pp. 1–14. [19] M. Gelsomini, et al., Magika, a multisensory environment for play, education and inclusion, in: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems, 2019, pp. 1–6. [20] C. Ardito, P. Bottoni, M. F. Costabile, G. Desolda, M. Matera, M. Picozzi, Creation and use of service-based distributed interactive workspaces, J. Vis. Lang. Comput. 25 (2014) 717–726. URL: https://doi.org/10.1016/j.jvlc.2014.10.018. doi:10.1016/j.jvlc.2014.10.018.