Advanced Interaction Paradigms to Define Smart Visit Experiences in the Internet of Things Era Carmelo Ardito Alessio Malizia Università di Bari Brunel University London Abstract via Orabona, 4 Uxbridge, UK The growing spread of smart objects is changing the 70125 Bari, Italy alessio.malizia@brunel.ac.uk way humans interact with technologies since the carmelo.ardito@uniba.it interaction they propose is more and more physical and Maristella Matera less virtual. From an HCI perspective, one of the most Giuseppe Desolda Politecnico di Milano interesting aspects regards how non-technical end Università di Bari Piazza Leonardo da Vinci, 32 users can program the behavior of such smart objects. via Orabona, 4 20134, Milano, Italy This poster presents an ongoing project on three novel 70125 Bari, Italy maristella.matera@polimi.it interaction paradigms that support the creation of giuseppe.desolda@uniba.it smart visit experiences. Rosa Lanzilotti Author Keywords Università di Bari Internet of Things; Advanced Interaction Paradigms; via Orabona, 4 Tangible Programming. 70125 Bari, Italy rosa.lanzilotti@uniba.it ACM Classification Keywords H.5.2 Human-centered computing~Ubiquitous and Copyright is held by the author/owner(s). mobile devices CHItaly ’17, September 18-20, 2017, Cagliari, Italy. Introduction The possibility to combine thousands of sensors and actuators enables smart objects to interact with humans and environments in unlimited ways, thus opening the opportunity to make everything “smart”, e.g. cars, clothes, buildings, etc. This phenomenon fostered the birth of the Internet of Things, a term coined in 2009 by Kevin Ashton to describe a system where the Internet is connected to the physical world creative and rich contexts, like CH, smart objects are via ubiquitous sensors 1. not conceived as “simple” devices exposing events and actions, but they bring with themselves their own In domains like Cultural Heritage (CH), smart objects semantics. For example, a smart card depicting an can be installed in museums, archaeological parks and Egyptian vase is not a simple hexadecimal code that exhibitions to create smart visit experiences, i.e. can be read by an RFID reader but represents a find scenario where visitors acquire CH content by dated back to a certain époque, discovered in a interacting with the surrounding environment and particular place, with an ancient name, etc. This smart objects included in it. However, to achieve such semantics could be included in ECA rules to simplify engaging scenarios, some issues still need to be solved. their definition and also to trigger the access to properties that the tangible objects (not only their From an HCI perspective one challenging goal regards sensors and actuators) features in the CH site context. how non-technical users (e.g. museum curators), can In order to enrich smart objects with semantics, users be enabled to make multiple smart objects interact should be able to define a set of sensible attributes. To Figure 1. The smart version of the Mouth of among them. In CH the role of domain experts is this aim, we are investigating novel interaction Truth we built for the Tyrion scenario currently quite limited: they can at last configure single paradigms that allow non-technical users to define objects that visitors bring across the CH site to receive sensible attributes on smart objects in a natural and personalized content when they reach some interactive simple way. In the following, we shortly describe the hot spots [5]. To support the design of more effective, preliminary design of three novel paradigms. engaging and attractive visit experiences, novel and more powerful composition paradigms are needed to A Scenario of Smart Visit Experience in synchronize the behavior of multiple devices. Cultural Heritage Domain To better understand the meaning and benefits of a This poster presents an ongoing project that aims to smart visit experience, in the following we report on a provide non-technical users with interaction scenario. The main person is Tyrion, a guide of an mechanisms to create smart visit experiences. This archeological park. He typically organizes its tours by paper follows a work we recently carried out in the area arranging visitors in groups of ten persons and going of task-automation systems [3], where we designed a with them in the park, stopping at some of the most tool called EFESTO-5W for simplifying the creation of significant points to explain their history. Tyrion wants Event-Condition-Action (ECA) rules combining smart to enrich this format by playing a serious game during object events/actions [4]. Some studies have shown the visit. The game he envisioned is based on two types that the EFESTO-5W composition paradigm effectively of smart objects: 1) a smart version of the “Mouth of guides users in establishing the behavior of multiple Truth” (MoT), i.e., a Roman marble disc with a relief smart objects [4]. However, we observed that in more carving of a man's face whose mouth, according to 1 http://www.rfidjournal.com/articles/view?4986. legend, closes if a liar sticks his hand in it (a smart, This last rule refers to the meaning the expert gives to small scale version is visible in Figure 1); 2) two decks each smart object, in particular the smart card is of twenty smart cards, each depicting an archaeological associated with the name “Egyptian” instead of and find. When the tour starts, visitors are assigned to two hexadecimal code, while the MoT position is called teams that receive a deck of card. During the visit, “FORNACE” instead of specifying GPS coordinates. If every time Tyrion stops at a point of interest, during his experts can define this type of rules, representing an narration he poses a question and asks each team to abstraction that speaks their language, they are select one card they consider relevant to that question. facilitated in creating a smart visit experience. Visitors have to put the selected card inside the MoT, whose eyes blink green if the card is right and blink red The current version of the EFESTO-5W prototype lacks otherwise. In addition, points are assigned/removed to the possibility to define sensible attributes to be each team depending on their answers. At the end of included in ECA rules. Therefore, this project aims to the game, the winning team wins a park souvenir. define novel, intuitive interaction paradigms that permit to extend the native properties of a smart object Toward Advanced Interaction Paradigms to (events and actions) with custom attributes, in order to Program Smart Visit Experiences exploit them while synchronizing smart objects. The To create smart visit experiences like the one described attribute definition becomes a preliminary phase that in the Tyrion scenario, end users need to program the end users have to carry out before synchronizing smart behavior of smart objects by defining ECA rules. For objects. Given the nature of smart objects, i.e., devices example, to determine when a smart card is correctly with which users can physically interact through their read by the MoT, a rule like the following has to be sensors and actuators, we think that the mechanisms created by using EFESTO-5W: to define such attributes have to take into account the physicality of the interaction. In other words, traditional IF (read smart card code is = AB123456 AND MoT GPS visual interfaces could not be adequate to define position is lat=12.3456;long=12.3456) custom attributes, while more concrete and tangible THEN the eyes of MoT blink of color (0,255,0) paradigms can better fit the smart-objects world. The syntax for rule definition is far the end-user language since it refers to technical aspects, like the The starting point of this research was the identification smart card hexadecimal code or the the GPS of a set of attributes end users can define on smart coordinates reached by the MOT during the visit. A objects. We found that data types like string, number more natural rule with the same behavior would be: and geographic position can be used to define a wide variety of significant attributes on smart objects. IF (read smart card is = Egyptian vase AND MoT Starting from these attributes, we designed three position is FURNACE) different composition paradigms during a design THEN the eyes of MoT blink green workshop study involving 28 users arranged in groups of 5/6 participants. The first paradigm is a tangible solution based on the attribute (e.g. a brown pin for a string attribute). When use of real objects representing the three types of user approaches a passive object, its pins are enriched attributes. Initially, each group was asked to identify, by their names and values, according to a semantic- for each type of attribute, at least one object of the real zoom technique. Users can collect all the useful world whose affordance refers to the attribute meaning. attributes, also editing their names/values, and then For example, for the string attribute, objects like pens, they can scan a smart object to paste the collected inkwell and sheet were proposed. Then, they were attributes. asked to propose interaction mechanisms to combine physical attributes with smart objects. All suggestions The third paradigm is a tactile solution based on the were mediated by a mobile device. In the resulting use of a tabletop interactive display. Conversely to the paradigm users have to scan (e.g., by using the device previous paradigms, this solution was designed camera) the smart object and then the physical exploiting their knowledge and expertise on interactive Figure 2. A sketch visioning the tactile attributes representing the type of attribute they want displays [1, 2]. According to our vision, the tabletop paradigm: a smart card depicting an ancient to assign, or vice versa. Each time a physical attribute surface is a workspace that facilitates the association vase is on the tabletop and a set of tangible is scanned, a pop-up appears on the end-user mobile between attributes and smart objects. The attributes attributes are in the smart card area in order device asking to define the attribute details by writing are represented as tangible objects, for example the to associate some attributes. its name and value. ones used in the tangible paradigm. To assign an attribute to a smart object, users start by putting on The second paradigm is a pervasive solution based on the surface a smart object (e.g. a smart card); the use of the real world as source of attributes. The afterwards, a proximity area appears around the smart surrounding environment is conceived as a set of object (e.g. a rounded halo) meaning that physical passive objects with their attributes. Let’s think, for attributes can be placed inside it (see Figure 2). Each example, of a museum in which there are paintings time a physical attribute is put inside the area, a pop- annotated with QR-codes that can be scan with a up on the surface asks user to define the attribute mobile phone to read detailed information (e.g., style, name and value. painter, history). These passive objects could be exploited to copy their attributes and paste them in Conclusion smart objects. During our study, each group was asked This paper has presented an ongoing research aiming to propose, at a high level, a solution to capture the at investigating interaction paradigms that support attributes of passive objects and to send them to the domain experts in the creation of smart visit smart objects. In the resulting idea, a smartphone is experiences. Three different composition paradigms used to explore the surrounding environment in an have been identified, i.e. tangible, pervasive and tactile augmented reality fashion, but everything is visualized paradigms. As future work, we will develop three in black and white, except the passive objects that prototypes, each one implementing a paradigm, to be exposes attributes. Indeed, they are augmented with compared to assess their usability and their support to colored pins, each one associated with a type of the creative design of smart interactive experiences. References 1. Ardito C., Buono P., Costabile M. F., and Desolda G. (2015). Interaction with large displays: a survey. ACM Computing Survey 47(3), 1-38. 2. Bellucci A., Malizia A., and Aedo I. (2014). Light on horizontal interactive surfaces: Input space for tabletop computing. ACM Comput. Surv. 46(3), 1- 42. 3. Coronado M. and Iglesias C. A. (2016). Task Automation Services: Automation for the Masses. IEEE Internet Computing 20(1), 52-8. 4. Desolda G., Ardito C., and Matera M. (2017). Empowering end users to customize their smart environments: model, composition paradigms and domain-specific tools. ACM Transactions on Computer-Human Interaction (TOCHI) 24(2), Article 12 (April 2017), 52 pages. 5. Petrelli D. and Lechner M. (2014). The meSch project – Material EncounterS with digital Cultural Heritage: Reusing existing digital resources in the creation of novel forms of visitor’s experiences. In Proc. of the International Committee for Documentation of ICOM (CIDOC '14). Dresden, Germany, 6 - 11 September.