=Paper=
{{Paper
|id=Vol-2702/sample-5col
|storemode=property
|title=Conceptual Modeling of Multisensory Smart Spaces
|pdfUrl=https://ceur-ws.org/Vol-2702/EMPATHY_2020_paper_5.pdf
|volume=Vol-2702
|authors=Mattia Gianotti,Fabiano Riccardi,Giulia Cosentino,Franca Garzotto,Maristella Matera
|dblpUrl=https://dblp.org/rec/conf/avi/GianottiRCGM20
}}
==Conceptual Modeling of Multisensory Smart Spaces==
https://ceur-ws.org/Vol-2702/EMPATHY_2020_paper_5.pdf
Conceptual Modeling of Multisensory Smart Spaces
Mattia Gianottia , Fabiano Riccardia , Giulia Cosentinoa , Franca Garzottoa and
Maristella Materaa
a
Politecnico di Milano, Milan, Italy
Abstract
The Internet of Things (IoT) enables the creation of Interactive Smart Spaces (ISSs) where different types of
digital devices are integrated in the ambient or embedded in physical objects, and can sense human actions
to control equipment, modify environmental parameters, or create multi-sensory effects. These IoT-enhanced
interactive systems can support human activities in different contexts, e.g., education, entertainment, home
assistance, rehabilitation, to name a few. We argue that a human-centered perspective in the design of ISSs is
needed to take into account some salient characteristics of these systems. New conceptual modeling issues also
need to be investigated to go beyond representing hardware, software, and connectivity features of IoT devices
and to capture the user interaction.
Keywords
Interactive Smart Spaces, Interactive IoT Smart Objects, Interaction modeling,
1. Introduction
When IoT technology were first proposed, the emphasis was on the creation of arrays of distributed
connected sensors (Wireless Sensor Networks) to support automation systems relieving users from
repetitive tasks (e.g., plants monitor or automatic controls). For this class of IoT systems, the inter-
action between the system and the user was of secondary priority. More recently, novel interactive
systems have been proposed to empower people in different activities and contexts of everyday life
[1, 2, 3, 4, 5]. A human-centered perspective has progressively emerged, in which the interactive capa-
bility of IoT-enhanced physical objects and spaces (hereinafter smart objects and smart spaces) becomes
more and more central, raising new requirements and challenges for conceptual modelling. In this
scenario, the scope of conceptual modeling goes beyond representing features related to hardware,
software, connectivity, and communication among multiple devices, and also address the interactions
between the users and the materials or spaces embedding such devices. This paper shortly discusses a
first attempt to address this modelling challenge, which also highlights the potential of an interaction-
centered modeling approach for research on End-User Development in the arena of interactive IoT
systems. More details on the modeling approach can be found in [6].
EMPATHY: Empowering People in Dealing with Internet of Things Ecosystems. Workshop co-located with AVI 2020, Island of
Ischia, Italy
email: mattia.gianotti@polimi.it (M. Gianotti); fabiano.riccardi.polimi.it (F. Riccardi); giulia.cosentino@polimi.it (G.
Cosentino); franca.garzotto@polimi.it (F. Garzotto); maristella.matera@polimi.it (M. Matera)
orcid: 0000-0001-6035-3367 (M. Gianotti); 0000-0001-5510-3240 (F. Riccardi); 0000-0002-8560-4924 (G. Cosentino);
0000-0003-4905-7166 (F. Garzotto); 0000-0003-0552-8624 (M. Matera)
© 2020 Copyright for this paper by its authors.
Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
Workshop
Proceedings
http://ceur-ws.org
ISSN 1613-0073 CEUR Workshop Proceedings (CEUR-WS.org)
Figure 1: Excerpts from the Structural Model of the Magic Room: the specification of a User Action, Selection,
by Human Actor Child.
2. Related work
In the last decades researchers have proposed different modelling approaches for IoT systems related
to Smart Spaces. Four major topics emerge: privacy and security of data transmission [7, 8, 9], or-
chestration of device behaviour [10, 11, 12, 13], data gathering and propagation [14, 15], and design of
single devices and smart objects [16, 11]. The role of the user is neglected or simply considered as a
pure source of data, and existing approaches take into account only marginally (or not at all) the need
of modeling human interaction in Interactive Smart Spaces. We instead argue that a human-centered
perspective is needed to take into account some salient characteristics of these systems. As illustrated
in the following section, new modeling issues also need to be investigated to go beyond representing
hardware, software, and connectivity features of IoT devices and capture the user interaction.
3. Modeling dimensions
The main abstractions of our conceptual approach are organized in two main sub-models: the Struc-
tural Model and the Interactive Behaviour Model. The Structural Model supports the representation of
the human and technological “actors”, their interaction capability, i.e., which actions they can per-
form and perceive (“sense”) and which perceivable effects they can generate (“actuate”), as well as the
digital contents that are involved in the user experience. The Interactive Behavior Model supports the
representation of the interactive behaviour of all actors and how cross-interactions are orchestrated
for the users to perform tasks and activities at different levels of complexity.
These two models derive from the extensive experience gained during the design of the Magic
Room, a sophisticated multi-sensory smart space for children’s play, learning, and rehabilitation that
we developed in the context of a national project and installed at two local schools and two therapeutic
centers in Italy[17, 3, 18, 19]. In the following we present a limited example of the conceptual model
of the Magic Room, extracted from the specification of one activity - Battleship, which is the classic
battleship board game but enhanced in the smart space - making it more interactive and enormously
engaging.
Figure 2: The high-level specification of a Technological Actor, Smart Sphere.
3.1. Structural Model
The structural model is built around the notions of Actor and Digital Resource. Actors are the build-
ing blocks of any ISS as their properties and their cross interactions enact the interactive experience.
There are two categories of Actors: Technological Actors, characterized by a mix of digital and phys-
ical features, and Human Actors, i.e., the users. The Human Actors in the Magic Room are Child
and Caregiver: the former interact with the smart space and the smart objects, the latter operate
on a tablet application to control the activity execution and flow. The Technological Actors are the
above mentioned devices and a number of Smart Objects: Smart Toys (embedding a variety of motion
and pressure sensors and light or sound actuators), Paper- or plastic-based Identifiable Objects (RFID
tagged items of different shapes), and a Smart Sphere that embeds a sensor (Near Object Detector) able
to generate an object identifier when it detects the proximity of Identifiable Elements - physical items
enriched with RFID tags).
At a high-level, technological actors are represented in terms of their interactive capability, as
exemplified for the Smart Sphere in the Figure 2. Figure 1 illustrates an excerpt of Structural Model
concerning the representation of the User Action Selection associated with the Child Human Actor.
The action is specified first by describing what can be selected, either visual content or physical content.
Then the diagram reports the basic actions through which Selection is performed, also in relationship
with the Technological Actors enabling such actions.
3.2. Interactive Behaviour Model
The diagrams reported in Figure 3 and 4 represent a portion of the Interactive Behaviour Model for
the activity “Battleship game”. It focuses on the Scene handling a turn of play. The defined flow of
Interaction Tasks includes the selection of a cell in the battleship grid, which generates a feedback to
the user, e.g., partial or full hits, water hits, end of game, and a request for further input. Figure 4
shows the details (i.e., the fragment of the Extended Activity Interaction Model) related to Interaction
Task “getCellCoordinates”, which refers to the selection of a cell in the battleship grid.
The diagram presents two alternative modalities to accomplish this Interaction Task, each one
envisioning Interactions and Effects. The user can "point on" a virtual content (a cell of the Battle
Figure 3: Activity Interaction Model of Battleship game.
Figure 4: Expanded Activity Interaction Model for the Interaction Task enabling the selection of cell coordi-
nates.
grid projected on the front screen) or can use the Smart Sphere, placing on top of it two Identifiable
Cards for the cell coordinates, one with a number and one with a letter. Each box associated to an
Interaction shows the Human Actor who executes the action(s) (Initiator) on the upper area of the
box and the Participants, i.e., the Technological Actors involved in the interaction on the bottom area
(Smart Sphere and Identifiable objects). The middle area of the box is devoted to specify the ECA
rules, omitted in the figure for lack of space. Effects are represented by boxes where the top area
is empty, being the stimuli initiated by system events without intentional human intervention. The
enactment of these effects is triggered by a system event related to the status of the task execution.
3.3. From Models to Software Architectures
A solution that eases the flexible definition of the interactive activities allows the ISS designers to
overcome the barrier of single-purpose implementation and also enables the installation and execu-
tion of multiple activities. The abstractions presented in the previous sections guided the definition
of a multi-layer architecture for the Magic Room characterized by modularity, flexibility, and exten-
sibility. The Activity specifications in the Interactive Behavioural Model and the Technology Actor
specifications in the Structural Model guides the definition of a JSON-based configuration file that
the Execution Engine interprets as the low level rules governing the interactive capabilities of the
Technological Actors.
4. Conclusion and Future Work
This work has discussed some modeling requirements that characterize of Interactive Smart Spaces
(ISSs), pinpointing the importance of considering the human as the principal actor in this class of
systems, and addressing the interaction capabilities as fundamental for the empowering users in this
spaces and enabling the accomplishment of complex tasks [20]. Our model and our overall approach
can pave the ground towards innovative methods of conceptual design in the IoT arena, and may also
lead to the definition of more modular and standardized technological architectures for future highly
interactive IoT systems. An interaction-centered modeling approach is also the first step towards
the definition of novel solutions to support the appropriation process of interactive IoT technology
for end users, and adequate abstractions like the ones presented in this paper can provide a base of
concepts upon which to create the building blocks for End-User Development methods and tools [2].
This issue is particularly important in contexts - like education and rehabilitation - where Interactive
Smart Spaces could have a significant potential but there is a strong need of personalization of the
experiences in these environments. This view raises new research challenges, addressing the way
the interaction capabilities of ISS should be modelled through metaphors and design patterns that
make sense to the users and would enable them to customize or even create from scratch their own
interactive smart experiences.
References
[1] F. Delprino, C. Piva, G. Tommasi, M. Gelsomini, N. Izzo, M. Matera, Abbot: a smart toy motivat-
ing children to become outdoor explorers, in: Proc. of AVI’18, 2018, pp. 1–9.
[2] G. Desolda, C. Ardito, M. Matera, Empowering end users to customize their smart environments:
model, composition paradigms, and domain-specific tools, ACM Transactions on Computer-
Human Interaction (TOCHI) 24 (2017) 1–52.
[3] F. Garzotto, M. Gelsomini, Magic room: A smart space for children with neurodevelopmental
disorder, IEEE Pervasive Computing 17 (2018) 38–48.
[4] D. Petrelli, M. Lechner, The mesch project–material encounters with digital cultural heritage:
Reusing existing digital resources in the creation of novel forms of visitor’s experiences, Proc.
CIDOC’14 (2014).
[5] G. Desolda, C. Ardito, H. Jetter, R. Lanzilotti, Exploring spatially-aware cross-device interaction
techniques for mobile collaborative sensemaking, Int. J. Hum. Comput. Stud. 122 (2019) 1–20.
URL: https://doi.org/10.1016/j.ijhcs.2018.08.006. doi:10.1016/j.ijhcs.2018.08.006.
[6] M. Gianotti, R. Fabiano, G. Cosentino, F. Garzotto, M. Matera, Modeling interactive smart spaces,
in: Proceedings of ER 2020, 2020, p. In print.
[7] M. F. Arruda, R. F. Bulcão-Neto, Toward a lightweight ontology for privacy protection in IoT,
in: Proceedings of SAC’ 19, ACM, 2019, pp. 880–888.
[8] D. Sahinel, C. Akpolat, O. C. Görür, F. Sivrikaya, Integration of human actors in IoT and CPS
landscape, in: Proc. of WF-IoT, 2019-04, pp. 485–490.
[9] A. Skarmeta, J. L. Hernández-Ramos, J. B. Bernabe, A required security and privacy framework
for smart objects, in: 2015 ITU Kaleidoscope: Trust in the Information Society (K-2015), 2015-12,
pp. 1–7.
[10] A. Bassi, M. Bauer, M. Fiedler, R. van Kranenburg, S. Lange, S. Meissner, T. Kramp, Enabling
things to talk, Springer Nature, 2013.
[11] G. Fortino, W. Russo, C. Savaglio, W. Shen, M. Zhou, Agent-oriented cooperative smart objects:
From IoT system design to implementation, IEEE Transactions on Systems, Man, and Cybernet-
ics: Systems 48 (2018-11) 1939–1956.
[12] M. Maheswaran, J. Wen, A. Gowing, Design of a context aware object model for smart spaces,
things, and people, in: Proc. of IEEE ICC’15, 2015-06, pp. 710–715.
[13] D. Ning, Y. Wang, J. Guo, A data oriented analysis and design method for smart complex software
systems of IoT, in: Proc. of ISSI’18, 2018-09, pp. 1–6.
[14] B. Costa, P. F. Pires, F. C. Delicato, Modeling SOA-based IoT applications with SoaML4iot, IEEE,
2019-04, pp. 496–501.
[15] K. Jahed, J. Dingel, Enabling model-driven software development tools for the internet of things,
in: Proc. of MiSE ’19, IEEE Press, 2019, pp. 93–99.
[16] M. Bermudez-Edo, T. Elsaleh, P. Barnaghi, K. Taylor, IoT-lite: A lightweight semantic model
for the internet of things and its use with dynamic semantics, Personal Ubiquitous Comput. 21
(2017-06) 475–487.
[17] F. Garzotto, E. Beccaluva, M. Gianotti, F. Riccardi, Interactive multisensory environments for
primary school children, in: Proc. of CHI ’20, CHI ’20, Association for Computing Machinery,
New York, NY, USA, 2020, p. 1–12.
[18] M. Gelsomini, G. Leonardi, F. Garzotto, Embodied learning in immersive smart spaces, in: Proc.
of CHI ’20, 2020, pp. 1–14.
[19] M. Gelsomini, et al., Magika, a multisensory environment for play, education and inclusion, in:
Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems, 2019,
pp. 1–6.
[20] C. Ardito, P. Bottoni, M. F. Costabile, G. Desolda, M. Matera, M. Picozzi, Creation and use of
service-based distributed interactive workspaces, J. Vis. Lang. Comput. 25 (2014) 717–726. URL:
https://doi.org/10.1016/j.jvlc.2014.10.018. doi:10.1016/j.jvlc.2014.10.018.