Interactive Experiences SmartObjects '18, in conjunction with CHI '18, Montreal, Canada An End-User Development Approach for Crafting Smart Interactive Experiences Abstract Carmelo Ardito Giuseppe Desolda Antonio Piccinno Despite the advantages that Internet of Things (IoT) University of Bari Aldo Moro University of Bari Aldo Moro University of Bari Aldo Moro technology offers, there are still important issues to be via Orabona, 4 via Orabona, 4 via Orabona, 4 solved to increase its practical impact. The 70125 - Bari, Italy 70125 - Bari, Italy 70125 - Bari, Italy opportunities offered by IoT can be amplified if new carmelo.ardito@uniba.it giuseppe.desolda@uniba.it antonio.piccinno@uniba.it approaches, based on high-level abstractions and adequate interaction paradigms, are conceived to Paolo Buono Rosa Lanzilotti involve directly non-technical users in configuring the University of Bari Aldo Moro University of Bari Aldo Moro behavior of their smart objects. In this paper, we via Orabona, 4 via Orabona, 4 present our End-User Development approach, which we 70125 - Bari, Italy 70125 - Bari, Italy would like to discuss at the workshop together with the Paolo.buono@uniba.it maria.costabile@uniba.it challenges our future research implies. Maria Francesca Costabile Maristella Matera Author Keywords University of Bari Aldo Moro Politecnico di Milano Internet of Things; Smart Object Modelling; Smart Visit via Orabona, 4 Piazza Leonardo da Vinci, 32 Experience; Cultural Heritage. 70125 - Bari, Italy 20134 - Milano, Italy ACM Classification Keywords maria.costabile@uniba.it Maristella.matera@polimi.it Software and its engineering~Integrated and visual development environments • Software Copyright © 2018 for this paper held by its author(s). Copying and its engineering~Visual languages. permitted for private and academic purposes. Introduction and Motivation Smart objects are increasingly pervading the environments we live in. If enabled to exploit the abundance of resources (object functionality, produced data, related applications), end users could compose the “behavior” of the surrounding environment to accommodate their everyday needs. However, programming the behavior of smart objects is currently 23 Interactive Experiences SmartObjects '18, in conjunction with CHI '18, Montreal, Canada a prerogative reserved for professional developers, as it these disadvantages occur because the proposed requires the use of scripting languages and tools that platforms are too “general”, claiming that one single can also vary depending on the underlying hardware. design might satisfy the requirements of many Furthermore, the available objects often expose very domains. For example, in Cultural Heritage (CH), specific functionalities that do not result in useful guides and curators are non-professional in Computer services able to accommodate users’ needs. Science who might create objects that visitors of CH sites can bring with themselves, touch and manipulate In order to directly involve non-technical users in for experiencing the site by receiving personalized configuring the behavior of their smart objects, new information. However, by observing CH experts approaches, based on high-level abstractions and engaged in the definition of smart objects behavior, we adequate interaction paradigms, have to be conceived. realized that, even without being required to write any Our research capitalizes on years of experience on End- code, it is still difficult for them to manage technical User Development (EUD), a discipline that concepts. We, therefore, propose higher-level encompasses methods, techniques, tools, and abstractions to allow end users themselves (i.e., CH methodologies to allow professionals of a given domain experts in the case study proposed in this paper) to to master domains in which they are not professionals define custom properties to characterize the semantics [1-4]. In the IoT context, we developed EFESTO-5W, a of smart objects, thus helping them to make sense of platform implementing an EUD approach to customize the available smart devices and digital resources and and synchronize the behavior of resources, like smart facilitate the definition of their cross interactions. These objects and Web services, through Event-Condition- features have been implemented in EFESTO-SE, an Actions (ECA) rules [5]. With respect to other Task- extension of EFESTO-5W, which was evaluated Automation Systems (TASs) (see, for example, those in involving 14 professional cultural-heritage guides who [6]), whose adopted graphical notations often do not were asked to create an interactive visit experience. match the mental model of most users [7], EFESTO-5W The study showed that our hypotheses on extending promotes a richer set of high-level abstractions and the semantics of smart objects by means of custom operators to define rules and a visual notation that, attributes were valid with respect to the expectations of despite the intrinsic complexity related to managing CH stakeholders. We are further exploring and verifying events and actions, is affordable even by non- this aspect in some recently undertaken research programmers. activities. In particular, we are interested in evaluating TASs are typically conceived as general purpose if our proposal can stimulate the creativity of CH systems, but their generality often implies a scarce stakeholder as smart-experience designers: the adoption by specific communities of end users [8]. hypothesis is that if the smart objects make evident the EFESTO-5W is also general purpose but can be relationship they have with the content, CH customized to several application domains. Our professionals can better identify how to adopt such position, which also derives from observing people devices to convey the CH-site content to visitors. adopting our EUD tools during field studies, is that 24 Interactive Experiences SmartObjects '18, in conjunction with CHI '18, Montreal, Canada This paper is organized as follows. The next section A usage scenario describes, with the help of a usage scenario, the Molly is the CH guide of the archaeological park of challenges of defining a smart interactive experience Egnathia, an ancient Roman city in Southern Italy. After the outdoor visit through the ruins, where Molly and how our approach addresses them. The last section explains the history of the city and illustrates the concludes the paper and suggests some research issues function of every place in the park, the visit continues that would matter to further discuss at the workshop. in the park museum. Molly engages visitors in playing a serious game in the “smart” rooms of the museum. Smart Experiences in the CH domain Here, display cases containing ancient objects are Even in the CH domain, IoT research has primarily instrumented with sensors able to detect NFC coins1 focused on technical features, e.g., how to program provided to each visitor before the game starts. During networks of sensors and actuators and how to ensure the game, Molly asks different questions and, their interoperability [9-12]. Very few approaches try to accordingly, sets the sensors of the display cases in facilitate the configuration of smart objects and their different modalities by means of an app installed on her smartphone. For example, she sets the “Age” modality advantage is limited to programming single objects that and asks visitors to find the display cases where Roman the visitors bring across the CH site to receive objects are shown. The visitors move through the personalized content when they reach hot spots [13]. It museum, identify the cases matching Molly’s request is still hard for CH experts (e.g., site curators and and touch them with their coin. If they are successful, professional guides) to synchronize the behavior of the light inside cases turns green and the visitor’s multiple devices in order to create smart visit current score is increased. Then, Molly asks other questions and sets the display cases in the experiences where different sensors and actuators, corresponding modality, thus the game continues. The installed in the environment or embedded in tangible synchronizations between cases and NFC coins are objects manipulated by visitors, actively react to some established by the guide using our platform through the detected events. creation of ECA rules. Defining the smart visit experience In our ongoing research, we are investigating how non- From the previous scenario, it is evident that the technical users, like CH guides, can personalize the personalization of a smart visit experience might not be behavior of smart objects in order to customize visit limited to a trivial synchronization of smart objects, but experiences depending on their skills and capabilities, it might also require creating digital narratives threads as well as on the needs of specific visitors. To that professionals themselves need to put in context understand how ECA rules can be exploited to define with respect to the CH-site content. Driven by these smart visit experiences, let us consider the scenario emerging requirements, we introduce the notion of described in the following section. custom attributes, as a means to characterize smart objects not only by native events and actions (as conceived in many IoT platforms) but also by properties 1 Near-field communication device, resembling an ancient Roman coin, used for identifying each visitor. 25 Interactive Experiences SmartObjects '18, in conjunction with CHI '18, Montreal, Canada that the domain experts (i.e., the designers of the the CH guide needs to define, for each display case, smart experience) can define to assign semantics to the couplings with the NFC coins, she has to create a rule objects. Such semantics empowers and simplifies the for each coupling. In addition, such rules refer to creation of ECA rules, as it can exploit an enriched technical terminology (e.g., the NFC-coin code) that vocabulary based on user-defined terms. Visual does not correspond to the language adopted by the mechanisms also simplify the creation of custom domain experts. Figure 1 illustrates an example of ECA attributes and their association to smart objects. rule that defines the synchronization between a single case (i.e. case 3) and a specific coin. Thus, Molly has to To understand some of the advantages of custom replicate this rule for coupling all the other cases and attributes, let us go back to the above scenario. Since coins. Figure 1. Example of ECA rule determining the behavior of a single case and coin. 26 Interactive Experiences SmartObjects '18, in conjunction with CHI '18, Montreal, Canada In our proposal, before creating ECA rules, Molly parks in Southern Italy. The guides, who participated interacts with a visual tool offered by our platform, individually and underwent the same procedure, were which allows her to assign custom attributes to each required to design a smart visit experience. case by manipulating widget interfaces, without the We collected different qualitative data. All the need of coding. Custom attributes can be seen as interactions were audio-video recorded by using an conceptual tools that can allow designers to external camera. To evaluate user satisfaction, a characterize the basic elements of a smart experience questionnaire with 23 statements was administered at (i.e., smart objects and rules) with a semantics related the end of the study. The first statement was the Net to the content to be conveyed during the smart Promoter Score (NPS) question [14], typically used to experience. measure, on a scale between 0 and 10, how likely users In the example of Figure 2, she defines and assigns the would recommend the product to a friend or a attributes Age, representing the age of the artifacts colleague. The NPS score is equal to 57, i.e., excellent; contained by the cases, Points, representing the it indicates an attitude towards suggesting this system number of points the visitor gains if the answer is to other CH guides. This encouraging result is also correct, Blinking time, indicating for how many seconds confirmed by the analysis of the SUS questions, which the case has to blink. From now on, the creation of ECA gave us detailed indications about the perceived system rules can exploit this terminology (see for example usability and learnability. The SUS global score Figure 3). In addition, more general rules, i.e., was 81.1/100 (SD = 14.1), which is higher than the parametric, can be created. In Molly’s scenario, she average SUS scores (69.5) computed based on one does not need to define a multitude of very similar thousand studies reported in [15]. rules for coupling every single case and coin, since they The thematic analysis carried out on the transcribed are all encompassed by the single rule shown in Figure data, triangulated with the questionnaire results, 3. allowed us to identify important themes, also Evaluation with CH Guides highlighting the presence of some usability issues to be We recruited 14 professional guides (5 female) aged addressed. We summarize them in the following between 18 and 50 (x̅ = 37.9, SD = 8.2), operating in section, since we are confident that they would enable different museums, archaeological sites and natural further discussion at the workshop. 27 Interactive Experiences SmartObjects '18, in conjunction with CHI '18, Montreal, Canada Figure 2. Platform tool for defining and assigning custom attributes. Figure 3. A single rule determining the behavior of multiple cases and coins. 28 Interactive Experiences SmartObjects '18, in conjunction with CHI '18, Montreal, Canada Discussion and conclusions than ECA rules creation. Other metaphors for the This article has presented our perspective on the EUD property assignment, than the one we implemented in of smart experiences. The work described here provides EFESTO-SE, could be perceived as more usable. For the first results of a larger research that aims to example, one participant suggested a spreadsheet- promote smart objects as components of a smart based solution: users could use a tabular format in experience that can bring with themselves evident which they allocate smart objects in rows, CA names in connections with the semantics of content for which columns, and then specify CA values in cells located at they facilitate the access. We showed how we extended the intersection between rows and columns. The and customized a generic composition paradigm, tabular format was also adopted in the elicitation study initially conceived for the EUD of IoT systems, to by the CH guides to specify CAs and their values. respond to the need of exploiting IoT to mediate Stimulating creativity in smart-experience design. narrative and content-appropriation goals for Another important aspect in smart-experience design is interactive visits to CH sites. However, some challenges the adoption of paradigms that can stimulate creativity. are still open. We summarize them in the following with The evaluation study demonstrated that EFESTO-SE the aim of stimulating discussion among the workshop has a potential in supporting the design of smart participants. Even though the reported aspects experiences. However, discussions with participants emerged from the application of our EUD paradigm to revealed that there is still room for improvements. the CH domain, we believe they can be of general validity and should be taken into account when defining Supporting and fostering technical skills growth. EUD frameworks for IoT also in other domains. Another aspect emerged during the discussion with the CH experts regards the customization activities that go Constraining the flow of design activities. Based beyond the smart-object programming supported by on the observed problems and participants’ EFESTO-SE. Indeed, 6 out of 14 participants stated suggestions, we believe that our environment for the that, after a certain period of EFESTO-SE usage, they creation of smart experiences should be redesigned to would like to be supported in extending the smart provide a robust guidance to users. For example, a object capabilities by integrating new sensors and wizard procedure can guide users in configuring an actuators, avoiding to involve every time IT experts. initial, limited core set of smart objects, together with Even if this activity seems an aspect that only their CAs and basic ECA rules controlling them. Later, technicians can accomplish, today there are hardware users can freely continue expanding this core set until and software solutions that satisfy this goal. For obtaining the final and complete smart experience. example, mCookies2 is an alternative to Arduino that Simplifying the paradigm for CA definition. As can support people who have an interest in the “Do It emerged from triangulating questionnaire results with Yourself” paradigm for electronics. It consists of a set users’ comments, CA definition resulted more difficult 2 http://microduinoinc.com/products/mcookie/ 29 Interactive Experiences SmartObjects '18, in conjunction with CHI '18, Montreal, Canada of magnetic, color-coded modules, which can be staked Y. Dittrich, G. Fischer and A. Piccinno (eds.). in a LEGO fashion. Springer LNCS 6654, 53-66. 8. Fabio Casati. 2011. How End-User Development Will References Save Composition Technologies from Their 1. Carmelo Ardito, Paolo Buono, Maria Francesca Continuing Failures. In End-User Development - IS- Costabile, Rosa Lanzilotti and Antonio Piccinno. EUD 2011. M. Costabile, Y. Dittrich, G. Fischer and 2012. End users as co-designers of their own tools A. Piccinno (eds.). Springer. LNCS 6654, 4-6. and products. Journal of Visual Languages & 9. A. Chianese and F. Piccialli. 2014. Designing a Computing, 23, 2: 78-90. Smart Museum: When Cultural Heritage Joins IoT. 2. Maria Francesca Costabile, Daniela Fogli, Piero In Proceedings of the International Conference on Mussio and Antonio Piccinno. 2007. Visual Next Generation Mobile Apps, Services and Interactive Systems for End-User Development: A Technologies (NGMAST '14), 300-306. Model-Based Design Methodology. IEEE 10. Vincenzo Mighali, Giuseppe Del Fiore, Luigi Patrono, Transactions on Systems, Man, and Cybernetics - Luca Mainetti, Stefano Alletto, Giuseppe Serra and Part A: Systems and Humans, 37, 6: 1029-1046. Rita Cucchiara. 2015. Innovative IoT-aware Services 3. Gerhard Fischer, Elisa Giaccardi, Yunwen Ye, Alistair for a Smart Museum. In Proceedings of the Sutcliffe and Nikolay Mehandjiev. 2004. Meta- International Conference on World Wide Web (WWW design: a manifesto for end-user development. '15 Companion), 547-550. Communications of the ACM, 47, 9: 33-37. 11. Francesco Piccialli and Angelo Chianese. 2017. The 4. Henry Lieberman, Fabio Paternò, Markus Klann and Internet of Things Supporting Context-Aware Volker Wulf. 2006. End-User Development: An Computing: A Cultural Heritage Case Study. Mob. Emerging Paradigm. In End User Development. H. Netw. Appl., 22, 2: 332-343. Lieberman, F. Paternò and V. Wulf (eds.). Springer 12. Francesco Piccialli and Angelo Chianese. 2017. A Netherlands, 1-8. location‐based IoT platform supporting the cultural 5. Giuseppe Desolda, Carmelo Ardito and Maristella heritage domain. Concurrency and Computation: Matera. 2017. Empowering End Users to Customize Practice and Experience, 29, 11. their Smart Environments: Model, Composition 13. Daniela Petrelli and Monika Lechner. 2014. The Paradigms, and Domain-Specific Tools. ACM Trans. meSch project – Material EncounterS with digital Comput.-Hum. Interact., 24, 2: 1-52. Cultural Heritage: Reusing existing digital resources 6. Gabriella Lucci and Fabio Paternò. 2015. Analysing in the creation of novel forms of visitor’s How Users Prefer to Model Contextual Event-Action experiences. In Proceedings of the International Behaviours in Their Smartphones. In End-User Committee for Documentation of ICOM (CIDOC '14) Development - IS-EUD 2015. P. Díaz, V. Pipek, C. 14. Douglas B Grisaffe. 2007. Questions about the Ardito, C. Jensen, I. Aedo and A. Boden (eds.). ultimate question: conceptual considerations in Springer International Publishing. LNCS 9083, 186- evaluating Reichheld's net promoter score (NPS). 191. Journal of Consumer Satisfaction, Dissatisfaction 7. Usman Wajid, Abdallah Namoun and Nikolay and Complaining Behavior, 20: 36. Mehandjiev. 2011. Alternative Representations for 15. Aaron Bangor, Philip Kortum and James Miller. End User Composition of Service-Based Systems. In 2009. Determining what individual SUS scores End-User Development - IS-EUD 2011. M. Costabile, mean: Adding an adjective rating scale. Journal of usability studies, 4, 3: 114-123. 30