The recent Internet of Things (IoT) phenomenon and the availability of a number of smart objects are pushing researchers to exploit this technology in different domains [9]. Moreover, an increasing number of end users, i.e., non-technical people not experts in computer science, nor willing to be, want to use such a plethora of smart devices for their daily activities, work, entertainment or other purposes [21,23,28]. Several studies highlight that Cultural Heritage (CH) is one of the most promising domains for IoT adoption, since this technology can favour the definition of smart visit experiences that engage visitors by allowing them to acquire CH content while interacting with the surrounding environment and the smart objects included in it [8,29,31,33]. So far, even in the CH domain, research on IoT has primarily focused on technical features of smart objects [13,20,26,30], while few approaches are trying to facilitate the adoption of such a technology by end users in order to foster the personalization of smart object behaviours. This lack limits the social and practical benefits of IoT; it creates barriers in all those usage scenarios where experts, e.g., museum curators and guides in CH sites, would like to personalize the behavior of smart objects but they usually do not have the required programming skills.
End-User Development (EUD) is a discipline that encompasses methods, techniques, methodologies, situations, and socio-technical environments that enable non-technical people to tailor software systems to their needs and desires by modifying or even creating software artefacts without requiring to write programs [3,15,19,24]. Based on years of experience of the authors in EUD and the works we recently carried out in the area of task-automation systems [5,18], this paper presents an ongoing research that aims to provide non-technical users, as CH experts, with natural interaction mechanisms to create smart visit experiences.
Different tools supporting non-technical users to configure smart object behavior have been recently proposed. In particular, the socalled Task-Automation (TA) tools [14] enable the combination of social services, data sources and sensors, and have become popular as they offer very easy and intuitive paradigms to synchronize the behavior of objects and applications [11,25]. Through Web editors, users can synchronize the behavior of smart objects by defining Event-Condition-Action (ECA) rules [27], a paradigm largely used for the specification of active systems (see for example [12,16]), which in the IoT domain can be fruitfully exploited to express how and when some object behaviors have to be activated in reaction to detected events. Many of such tools offer visual, intuitive interaction paradigms to graphically sketch ECA rules specifying the interaction among the objects, for example by means of graphs that represent how events and data parameters propagate among the different objects to achieve their synchronization. This is not a trivial task and it often requires data and system re-engineering [10] together with test [22].
Despite the popularity of such tools, very often their graphical notations for rule specification do not match the mental model of most users [32]. Another limiting factor is that the expressive power of the ECA rules that can be specified is limited to very simple synchronized behaviors. To overcome these limitations, we designed a tool called EFESTO-5W for simplifying the creation of ECA rules that combine smart object events/actions [18]. Some of the EFESTO-5W visual mechanisms are inspired to the ones available in the previous version of the platform used for data source composition [1,6,7,17].
Some studies have shown that the EFESTO-5W composition paradigm effectively guides users in establishing the behavior of multiple smart objects [18]. However, we observed that in more creative and rich contexts, like CH, smart objects cannot be simply treated as "low-level" devices exposing events and actions, but they bring with themselves their own semantics that refer to their role, within the CH site, of mediators for content acquisition by visitors. For example, a smart card depicting an Egyptian vase should not be simply considered as a hexadecimal code that can be read by an RFID reader; rather it represents a find dated back to a certain époque, discovered in a particular place, having an ancient name. This semantics could be exploited to simplify the definition of ECA rules by stakeholders who are more interested in the semantics of the objects (i.e., what content they can convey during the visit), more than in the signals they can emit or they can receive as input. In order to enrich smart objects with semantics, it should be possible for domain experts to define the sensible attributes that are representative of such a semantics.
To reach this goal, we aim to extend our platform for ECA rule definition by implementing intuitive and natural interaction paradigms that allow non-technical users to define sensible attributes on smart objects. In the following, we describe the preliminary design of three novel paradigms.
To better explain what a smart visit experience is, we describe a scenario where the main persona is Molly, a guide of the archaeological park of Egnathia, which was an ancient Roman city in Southern Italy. Molly first guides a group of visitors through the ruins; she explains the history of the city and illustrates the places The synchronizations between cases and NFC coins are established by the guide using our platform through the creation of ECA rules, which define the synchronization between a single case and a specific coin. Thus, Molly has to replicate this rule for coupling all the other cases and coins.
From the previous scenario, it is evident that the personalization of a smart visit experience might not be limited to a trivial synchronization of smart objects, but it might also require creating digital narratives threads that professionals themselves need to put in context with respect to the CH-site content. Driven by these emerging requirements, we introduced the notion of custom attributes [5], as a means to characterize smart objects not only by native events and actions (as conceived in many IoT platforms) but also by properties that the domain experts (i.e., the designers of the smart experience) can define to assign semantics to the objects. Such semantics empowers and simplifies the creation of ECA rules. Visual mechanisms have been proposed to simplify the creation of custom attributes and their association to smart objects [5].
To understand some of the advantages of custom attributes and how they can be visually defined in EFESTO-5W, let us go back to the above scenario. Before creating ECA rules, Molly interacts with a tool offered by our platform, which allows her to assign attributes to each case by manipulating widget interfaces, without the need of writing code. In the example of Figure 1, she defines and assigns the Age attribute, representing the age of the artifacts contained by the cases, Points, representing the number of points the visitor gains if the answer is correct, Blinking time, indicating for how many seconds the case has to blink. From now on, the creation of ECA rules can exploit this terminology (see for example Figure 2). In Molly's scenario, she does not need to define a multitude of very similar rules for coupling every single case and coin, since they are all encompassed by the single rule shown in Figure 2.
Albeit the process illustrated above supports end users in defining terms about a domain that can be useful when creating ECA rules [5], we believe that there is room to improve the proposed visual approach by means of more natural paradigms that facilitate the attributes creation, stimulate the designers creativity and foster the technology acceptance. Therefore, the ongoing research aims to investigate intuitive and natural interaction paradigms that allow to extend the native properties of a smart object (events and actions) with custom attributes, in order to exploit them while synchronizing smart objects.
The starting point of this research was the identification of a set of attribute types that the end users can exploit to describe the semantic properties of smart objects. We found that types like text, number, and geographic position can be used to define a wide variety of significant attributes on smart objects. Starting from these attributes, we designed three different composition paradigms during a design workshop study involving 28 users arranged in groups of 5/6 participants. The first paradigm is a tangible solution based on the use of real objects representing the three types of attributes. Initially, each group was asked to identify, for each type of attribute, at least one object of the real world whose affordance refers to the attributed meaning. For example, for the string attribute, objects like pens, inkwell and sheet were proposed. Then, they were asked to propose interaction mechanisms to combine physical attributes with smart objects. In the resulting paradigm, each physical attribute can be associated to a smart object by 1) putting close the smart objects and the custom attributes they want to associate and 2) assigning attribute name and value by using a post-it. For example, in Figure 3 in the left side of the desk, a smart card depicting a vase has 3 physical attributes around, each with a post-it indicating its name and value. The design process implemented by this paradigm thus occurs in the physical world. When the associations are established, their digitalization/formalization is carried out by using a mobile app that, by taking a picture of the desk, recognizes the smart objects, the physical attributes, the associations among them and the attribute names and values reported on the post-it, and creates the custom attributes on the smart objects.
The second paradigm is a pervasive solution based on the use of the real world as a source of attributes. The surrounding environment is conceived as a set of passive objects with their attributes. Let us think, for example, to a museum in which there are paintings and cases labeled with QR-codes that can be scanned with a mobile phone to visualize additional information (e.g., style, painter, history). These passive objects could be exploited to copy their attributes and paste them into the smart objects. During our study, each group was asked to figure out a solution to capture the attributes of passive objects and to send them to the smart objects. In the resulting idea, a smartphone is used to explore the surrounding environment in an augmented reality fashion; here everything is visualized in black and white, except the passive objects that expose attributes. Indeed, they are augmented with colored pins, each one associated with a type of attribute (e.g., a brown pin for a string attribute). When a user approaches a passive object, its pins are enriched by their names and values, according to a semantic-zoom technique. Users can collect all the useful attributes, also edit their names/values, and then they can scan a smart object to paste the collected attributes.
The third paradigm is a tactile solution based on the use of a tabletop interactive display that was designed by some of the authors of this paper exploiting their knowledge and expertise on interactive displays [2]. The tabletop surface becomes a workspace that facilitates the association between attributes and smart objects. The attributes are represented as tangible objects, for example, the ones used in the tangible paradigm. To assign an attribute to a smart object, users putting on the surface the smart object (e.g. a smart card); afterward, a proximity area appears around the smart object (e.g., a rounded halo) meaning that physical attributes can be placed inside it (see Figure 5). Each time a physical attribute is put inside the area, a pop-up on the surface asks users to define the attribute name and value.
This paper has presented an ongoing research on interaction paradigms that can support domain experts in the creation of smart visit experiences. Three different composition paradigms have been identified, i.e. tangible, pervasive and tactile. As future work, we are going to evaluate them by assessing their usability, their ability to support the creative design of smart interactive experiences, and their acceptance in real contexts, also comparing them with further composition paradigm we implemented and 3D possibilities [4].
The research described in this paper was supported by the Project "Opera Lirica e Realtà Aumentata -opeRA" -Prog. n. F/050075/01/X32 -MISE -D.M. 1 Giugno 2016 "Horizon 2020 -PON 2014/2020".