In recent years, smart objects are increasingly pervading the environments we live in. Several studies highlight that Cultural Heritage (CH) is a very promising domain for IoT adoption, since this technology can favor the definition of smart visit experiences that engage visitors by allowing them to acquire CH content while interacting with the surrounding smart environment and the smart objects included in it. This paper presents new End-User Development approaches and the related abstractions that can support CH experts to create customizable visit experiences within museums and other cultural sites.
• Software and its engineering → Application specific development environments; Internet of things; smart object behavior; tangible programming; end-user development
The recent Internet of Things (IoT) phenomenon and the
availability of a number of smart objects are pushing researchers to
exploit this technology in diferent domains [ 9]. Moreover, an
increasing number of end users, i.e., non-technical people not experts
in computer science, nor willing to be, want to use such a plethora
of smart devices for their daily activities, work, entertainment or
other purposes [
End-User Development (EUD) is a discipline that encompasses
methods, techniques, methodologies, situations, and socio-technical
environments that enable non-technical people to tailor software
systems to their needs and desires by modifying or even creating
software artefacts without requiring to write programs [
Diferent tools supporting non-technical users to configure smart
object behavior have been recently proposed. In particular, the
socalled Task-Automation (TA) tools [14] enable the combination of
social services, data sources and sensors, and have become popular
as they ofer very easy and intuitive paradigms to synchronize the
behavior of objects and applications [
Despite the popularity of such tools, very often their graphical
notations for rule specification do not match the mental model
of most users [
Some studies have shown that the EFESTO-5W composition paradigm efectively guides users in establishing the behavior of multiple smart objects [18]. However, we observed that in more creative and rich contexts, like CH, smart objects cannot be simply treated as "low-level" devices exposing events and actions, but they bring with themselves their own semantics that refer to their role, within the CH site, of mediators for content acquisition by visitors. For example, a smart card depicting an Egyptian vase should not be simply considered as a hexadecimal code that can be read by an RFID reader; rather it represents a find dated back to a certain époque, discovered in a particular place, having an ancient name. This semantics could be exploited to simplify the definition of ECA rules by stakeholders who are more interested in the semantics of the objects (i.e., what content they can convey during the visit), more than in the signals they can emit or they can receive as input. In order to enrich smart objects with semantics, it should be possible for domain experts to define the sensible attributes that are representative of such a semantics.
To reach this goal, we aim to extend our platform for ECA rule definition by implementing intuitive and natural interaction paradigms that allow non-technical users to define sensible attributes on smart objects. In the following, we describe the preliminary design of three novel paradigms. 3
To better explain what a smart visit experience is, we describe a scenario where the main persona is Molly, a guide of the archaeological park of Egnathia, which was an ancient Roman city in Southern Italy. Molly first guides a group of visitors through the ruins; she explains the history of the city and illustrates the places in the park. Then, the visit continues in the park museum. Molly engages visitors in playing a serious game in the "smart" rooms of the museum. Here, display cases containing ancient objects are instrumented with sensors able to detect NFC coins (resembling an ancient Roman coin, used for identifying each visitor) each visitor is provided with. During the game, Molly asks diferent questions and, accordingly, sets the sensors of the display cases in diferent modalities by means of an app installed on her smartphone. For example, she sets the "Age = Roman" modality and asks visitors to ifnd the display cases where Roman objects are shown. The visitors move through the museum, identify the cases matching Molly’s request and insert one of their own coins in the case coin slot. If they are successful, the light inside cases turns green and the visitor’s current score is increased. Then, Molly asks other questions and sets the display cases in the corresponding modality, thus the game continues focusing on the topic of the new modality.
The synchronizations between cases and NFC coins are established by the guide using our platform through the creation of ECA rules, which define the synchronization between a single case and a specific coin. Thus, Molly has to replicate this rule for coupling all the other cases and coins.
From the previous scenario, it is evident that the personalization of a smart visit experience might not be limited to a trivial synchronization of smart objects, but it might also require creating digital narratives threads that professionals themselves need to put in context with respect to the CH-site content. Driven by these emerging requirements, we introduced the notion of custom attributes [5], as a means to characterize smart objects not only by native events and actions (as conceived in many IoT platforms) but also by properties that the domain experts (i.e., the designers of the smart experience) can define to assign semantics to the objects. Such semantics empowers and simplifies the creation of ECA rules. Visual mechanisms have been proposed to simplify the creation of custom attributes and their association to smart objects [5].
To understand some of the advantages of custom attributes and how they can be visually defined in EFESTO-5W, let us go back to the above scenario. Before creating ECA rules, Molly interacts with a tool ofered by our platform, which allows her to assign attributes to each case by manipulating widget interfaces, without the need of writing code. In the example of Figure 1, she defines and assigns the Age attribute, representing the age of the artifacts contained by the cases, Points, representing the number of points the visitor gains if the answer is correct, Blinking time, indicating for how many seconds the case has to blink. From now on, the creation of ECA rules can exploit this terminology (see for example Figure 2). In Molly’s scenario, she does not need to define a multitude of very similar rules for coupling every single case and coin, since they are all encompassed by the single rule shown in Figure 2.
Albeit the process illustrated above supports end users in defining terms about a domain that can be useful when creating ECA rules [5], we believe that there is room to improve the proposed visual approach by means of more natural paradigms that facilitate the attributes creation, stimulate the designers creativity and foster the technology acceptance. Therefore, the ongoing research aims to investigate intuitive and natural interaction paradigms that allow to extend the native properties of a smart object (events and actions) with custom attributes, in order to exploit them while synchronizing smart objects. 4
The starting point of this research was the identification of a set of attribute types that the end users can exploit to describe the semantic properties of smart objects. We found that types like text, number, and geographic position can be used to define a wide variety of significant attributes on smart objects. Starting from these attributes, we designed three diferent composition paradigms during a design workshop study involving 28 users arranged in groups of 5/6 participants.
The first paradigm is a tangible solution based on the use of real objects representing the three types of attributes. Initially, each group was asked to identify, for each type of attribute, at least one object of the real world whose afordance refers to the attributed meaning. For example, for the string attribute, objects like pens, inkwell and sheet were proposed. Then, they were asked to propose interaction mechanisms to combine physical attributes with smart objects. In the resulting paradigm, each physical attribute can be associated to a smart object by 1) putting close the smart objects and the custom attributes they want to associate and 2) assigning attribute name and value by using a post-it. For example, in Figure 3 in the left side of the desk, a smart card depicting a vase has 3 physical attributes around, each with a post-it indicating its name and value. The design process implemented by this paradigm thus occurs in the physical world. When the associations are established, their digitalization/formalization is carried out by using a mobile app that, by taking a picture of the desk, recognizes the smart objects, the physical attributes, the associations among them and the attribute names and values reported on the post-it, and creates the custom attributes on the smart objects.
The second paradigm is a pervasive solution based on the use of the real world as a source of attributes. The surrounding environment is conceived as a set of passive objects with their attributes. Let us think, for example, to a museum in which there are paintings and cases labeled with QR-codes that can be scanned with a mobile phone to visualize additional information (e.g., style, painter, history). These passive objects could be exploited to copy their attributes and paste them into the smart objects. During our study, each group was asked to figure out a solution to capture the attributes of passive objects and to send them to the smart objects. In the resulting idea, a smartphone is used to explore the surrounding environment in an augmented reality fashion; here everything is visualized in black and white, except the passive objects that expose attributes. Indeed, they are augmented with colored pins, each one associated with a type of attribute (e.g., a brown pin for a string attribute). When a user approaches a passive object, its pins are enriched by their names and values, according to a semantic-zoom technique. Users can collect all the useful attributes, also edit their names/values, and then they can scan a smart object to paste the collected attributes.
The third paradigm is a tactile solution based on the use of a tabletop interactive display that was designed by some of the authors of this paper exploiting their knowledge and expertise on interactive displays [2]. The tabletop surface becomes a workspace that facilitates the association between attributes and smart objects. The attributes are represented as tangible objects, for example, the ones used in the tangible paradigm. To assign an attribute to a smart object, users putting on the surface the smart object (e.g. a smart card); afterward, a proximity area appears around the smart object (e.g., a rounded halo) meaning that physical attributes can be placed inside it (see Figure 5). Each time a physical attribute is put inside the area, a pop-up on the surface asks users to define the attribute name and value. 5
This paper has presented an ongoing research on interaction paradigms that can support domain experts in the creation of smart visit experiences. Three diferent composition paradigms have been identified, i.e. tangible, pervasive and tactile. As future work, we are going to evaluate them by assessing their usability, their ability to support the creative design of smart interactive experiences, and their acceptance in real contexts, also comparing them with further composition paradigm we implemented and 3D possibilities [4].
The research described in this paper was supported by the Project "Opera Lirica e Realtà Aumentata - opeRA" - Prog. n. F/050075/01/X32 - MISE - D.M. 1 Giugno 2016 "Horizon 2020 - PON 2014/2020". [2] Carmelo Ardito, Paolo Buono, Maria Francesca Costabile, and Giuseppe Desolda. 2015. Interaction with Large Displays: A Survey. ACM Computing Surveys (CSUR) 47, 3, Article 46 (Feb. 2015), 38 pages. https://doi.org/10.1145/2682623 [3] Carmelo Ardito, Paolo Buono, Maria Francesca Costabile, Rosa Lanzilotti, and Antonio Piccinno. 2012. End Users as Co-designers of Their Own Tools and Products. Journal of Visual Languages & Computing 23, 2 (2012), 78 – 90. https: //doi.org/10.1016/j.jvlc.2011.11.005 Special issue dedicated to Prof. Piero Mussio. [4] Carmelo Ardito, Paolo Buono, Maria Francesca Costabile, Rosa Lanzilotti, and Adalberto L. Simeone. 2009. Comparing Low Cost Input Devices for Interacting with 3D Virtual Environments. In Proceedings of the 2Nd Conference on Human System Interactions (HSI’09). IEEE Press, Piscataway, NJ, USA, 289–294. http: //dl.acm.org/citation.cfm?id=1689359.1689411 [5] Carmelo Ardito, Paolo Buono, Giuseppe Desolda, and Maristella Matera. 2018.
From Smart Objects to Smart Experiences: An End-User Development Approach. International Journal of Human-Computer Studies 114 (2018), 51 – 68. https://doi. org/10.1016/j.ijhcs.2017.12.002 Advanced User Interfaces for Cultural Heritage. [6] Carmelo Ardito, M. Francesca Costabile, Giuseppe Desolda, Rosa Lanzilotti, Maristella Matera, and Matteo Picozzi. 2014. Visual Composition of Data Sources by End Users. In Proceedings of the 2014 International Working Conference on Advanced Visual Interfaces (AVI ’14). ACM, New York, NY, USA, 257–260. https://doi.org/10.1145/2598153.2598201 [7] Carmelo Ardito, Maria Francesca Costabile, Giuseppe Desolda, Markus Latzina, and Maristella Matera. 2015. Making Mashups Actionable Through Elastic Design Principles. In End-User Development, Paloma Díaz, Volkmar Pipek, Carmelo Ardito, Carlos Jensen, Ignacio Aedo, and Alexander Boden (Eds.). Springer International Publishing, Cham, 236–241. https://doi.org/10.1007/978-3-319-18425-8_22 [8] Carmelo. Ardito, Maria Fransesca Costabile, and Rosa Lanzilotti. 2010. Gameplay on a Multitouch Screen to Foster Learning About Historical Sites. In Proceedings of the International Conference on Advanced Visual Interfaces (AVI ’10). ACM, New York, NY, USA, 75–78. https://doi.org/10.1145/1842993.1843006 [9] Luigi Atzori, Antonio Iera, and Giacomo Morabito. 2010. The Internet of Things: A Survey. Computer Networks 54, 15 (Oct. 2010), 2787–2805. https://doi.org/10. 1016/j.comnet.2010.05.010 [10] Alessandro Bianchi, Danilo Caivano, and Giuseppe Visaggio. 2000. Method and Process for Iterative Reengineering of Data in a Legacy System. In Proceedings of the Seventh Working Conference on Reverse Engineering (WCRE’00) (WCRE ’00). IEEE Computer Society, Washington, DC, USA, 86–. http://dl.acm.org/citation. cfm?id=832307.837101 [11] Federico Cabitza, Daniela Fogli, Rosa Lanzilotti, and Antonio Piccinno. 2017. Rulebased Tools for the Configuration of Ambient Intelligence Systems: A Comparative User Study. Multimedia Tools and Applications 76, 4 (Feb. 2017), 5221–5241. https://doi.org/10.1007/s11042-016-3511-2 [12] Stefano Ceri, Florian Daniel, Maristella Matera, and Federico M. Facca. 2007.
Model-driven Development of Context-aware Web Applications. ACM Transactions on Internet Technology (TOIT) 7, 1, Article 2 (Feb. 2007). https://doi.org/10. 1145/1189740.1189742 [13] Angelo Chianese and Francesco Piccialli. 2014. Designing a Smart Museum: When Cultural Heritage Joins IoT. In Proceedings of the 2014 Eighth International Conference on Next Generation Mobile Apps, Services and Technologies (NGMAST ’14). IEEE Computer Society, Washington, DC, USA, 300–306. https://doi.org/10. 1109/NGMAST.2014.21 [14] Miguel Coronado and Carlos A. Iglesias. 2016. Task Automation Services: Automation for the Masses. IEEE Internet Computing 20, 1 (Jan. 2016), 52–58. https://doi.org/10.1109/MIC.2015.73 [15] Maria Francesca Costabile, Daniela Fogli, Piero Mussio, and Antonio Piccinno. 2007. Visual Interactive Systems for End-User Development: A Model-Based Design Methodology. IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans 37, 6 (Nov 2007), 1029–1046. https://doi.org/10.1109/ TSMCA.2007.904776 [16] Florian Daniel, Maristella Matera, and Giuseppe Pozzi. 2008. Managing Runtime Adaptivity Through Active Rules: The Bellerofonte Framework. Journal of Web Engineering 7, 3 (Sept. 2008), 179–199. http://dl.acm.org/citation.cfm?id=2011271. 2011273 [17] Giuseppe Desolda, Carmelo Ardito, and Maristella Matera. 2016. EFESTO: A Platform for the End-User Development of Interactive Workspaces for Data Exploration. In Rapid Mashup Development Tools, Florian Daniel and Cesare Pautasso (Eds.). Springer International Publishing, Cham, 63–81. https://doi.org/ 10.1007/978-3-319-28727-0_5 [18] Giuseppe Desolda, Carmelo Ardito, and Maristella Matera. 2017. Empowering End Users to Customize Their Smart Environments: Model, Composition Paradigms, and Domain-Specific Tools. ACM Transactions on Computer-Human Interaction (TOCHI) 24, 2, Article 12 (April 2017), 52 pages. https://doi.org/10.1145/3057859 [19] Gerhard Fischer, Daniela Fogli, and Antonio Piccinno. 2017. Revisiting and Broadening the Meta-Design Framework for End-User Development. Springer International Publishing, Cham, 61–97. https://doi.org/10.1007/978-3-319-60291-2_4 [20] Piccialli Francesco and Chianese Angelo. 2017. A Location-based IoT Platform Supporting the Cultural Heritage Domain. Concurrency and Computation: