25 An Authoring Environment for Smart Objects in Museums: the meSch Approach Martin Risseeuw Mark T. Marshall Abstract WAAG Society Daniela Petrelli The meSch project addresses the challenges of creating Amsterdan, Netherlands Sheffield Hallam University a personally meaningful, sensorily rich, and socially martin@waag.org Sheffield, UK expanded museum visitor experience through tangible m.marshall@shu.ac.uk and embodied interaction with digital content. It is of Dario Cavada d.petrelli@shu.ac.uk paramount importance that cultural heritage eCTRL professionals are directly involved in the design of Trento, Italy Thomas Kubitza those experiences. The meSch approach is to empower cavada@ectrlsolutions.com University of Stuttgart cultural heritage professionals with tools that guide Stuttgard, Germany them through a do-it-yourself process of creating or Elena Not Thomas.Kubitza@vis.uni- adapting digitally augmented experiences for their own Massimo Zancanaro stuttgart.de museum spaces, therefore reducing the barriers of FBK – Fondazione Bruno Kessler introducing Internet of Things technology in cultural Trento, Italy heritage spaces. not@fbk.eu zancana@fbk.eu Paste the appropriate copyright/license statement here. ACM now Author Keywords supports three different publication options: Tangible and Embodied Interaction; Authoring Tool; • ACM copyright: ACM holds the copyright on the work. This is the historical approach. Cultural Heritage. Copyright • License:isThe held by theretain author(s) author/owner(s). copyright, but ACM receives an exclusive publication license. AVI, JuneAccess: • Open 07–10, The2016, Bari, author(s) Italy wish to pay for the work to be open ACM Classification Keywords access. The additional fee must be paid to ACM. H.5.m. Information interfaces and presentation (e.g., This text field is large enough to hold the appropriate release statement HCI): Miscellaneous; See assuming it is single-spaced in Verdana 7 point font. Please do not change the size of this text box. Each submission will be assigned a unique DOI string to be included here. 26 Introduction the German defense line along the European west coast Within the Material Encounters with Digital Cultural erected during World War II to block the allied invasion. Heritage (meSch 1) project [3], we are working on The exhibition was meant to reflect on how the city of technology to support the whole museum visiting The Hague and its citizens had been affected. The experience by fostering a vision where materiality and meSch platform has been used to enrich the display of rich sensory interaction add to the cognitive aspects for historical artefacts with voices from the past, such as a personally meaningful experience. There is a wide those of the German soldiers, the Dutch civilians and recognition that such approach may greatly improve the civil servants 3. Visitors could play these narratives both the visit and also visitors’ appreciation for the by using ‘smart replicas’, reproductions of real museum museum’s cultural values [1]. objects (e.g. a reproduction of a surrogate tea bag, a 3D-printed replica of a Delft blue mug, and so on) enhanced with sensors. By placing their smart replica on hotspots on interactive cases visitors could listen to narratives and see historical photographs and video clips thus enriching the original museum objects on display 4. From April to November 2015, more than 14,000 visitors interacted with the meSch smart replicas at the hotposts. The active involvement of cultural heritage Figure 1 A visitor interacting with a smart replica in the Atlantic Wall exhibition [2]; professionals in the preparation of this kind of experiences is of paramount importance, even if they The idea behind meSch is taking objects out of their often lack technical skills [3]. The meSch approach cases (or using replicas like in [2]) to let visitors consists in fostering reuse over creation from scratch. A experience objects’ shape and weight while learning catalogue of reusable narrative and interaction about them (figure 1), to engage the visitors at a strategies is provided as starting points that can be physical level and put information in context without adapted and refined. the distractions induced by interacting with extraneous devices, apps, and touch screens. The ingredients for a digitally augmented experience As an example, the meSch platform has been The approach to experience creation and demonstrated at the Atlantic Wall exhibition at Museon personalization in meSch is based on a clear separation in The Hague, The Netherlands 2. The Atlantic Wall was 3http://mw2016.museumsandtheweb.com/glami/the-hague-and- 1 http://mesch-project.eu the-atlantic-wall-war-in-the-city-of-peace/ 2 http://www.museon.nl/en/exhibitions/hague-and-atlantic-wall 4 https://www.youtube.com/watch?v=sK3AdQU9kkc 27 of content from interaction, and aims at facilitating the to a specific exhibit); (iii) the device: the description of preparation and the reuse of (i) narrative threads that the actual hardware required for deploying the system can be adapted to different visitors and types of and its technical requirements; (iv) the interaction experience and (ii) interaction strategies that describe script: the interaction rules that govern when how content should be released in context while presentations should be started or how the system experiencing the objects and the space. behaviour should be adjusted to personalize the experience to visitors’ context (e.g., a set of rules that activate the projection of the most appropriate content about an exhibit when visitors are in its vicinity). This multilayer definition of the experience schema concept supports the following properties: (i) reusability: an experience schema defines a skeleton of experience that can be reused with different content and in different physical settings to create many experience instances; (ii) modularity: parts of an experience schema can be replaced to create a new experience schema. For example, the interaction script can be changed to introduce different rules for firing Figure 2. The meSch Magazine: the starting point for browsing and adapting digitally augmented experiences the presentation of the same content in context. Another example is the possibility to experiment the We define a declarative formalism, the experience same type of experience with an alternative hardware schema that supports the description of the content set-up; (iii) compositionality: different experience items, the available interactions, the semantic schemas can be composed together to create more annotations and the conditional rules that govern the articulated augmented exhibitions; (iv) reduction of adaptive composition of the experience that is delivered complexity: authors can concentrate on the preparation to visitors at runtime [4]. This is done through a 4- of the necessary content and on its grouping according layer data structure including: the narrative, the to the narrative dimensions exploited in the experience appliance, the device and the interaction script: (i) the in a dedicated step. The rules for putting content in narrative: a set of carefully selected digital content context are represented separately and can be edited items, annotated into alternative thematic treads and by experience designers; (v) declarativity: the levels of detail that revolve around the objects and declarations contained in the appliance section of an places of an exhibition; (ii) the appliance: a declarative experience schema allow to specify in a formal way the specification of the capabilities of the technology type of elementary interactions and behaviour events embedded in the museum premises (such as, for that are available to shape the experience; this allows example, the possibility to detect the visitors’ proximity to abstract from the actual hardware details and the 28 low level sensor logs, thus facilitating the Once they find a digitally augmented experience that experimentation of different technological solutions. may suit their needs, curators may replicate the “recipe” of the experience (which consists in its The meSch scenario of authoring a digitally experience schema) and iteratively adapt it to their augmented visit experience specific needs. A catalogue of reusable narrative and interaction strategies with step-by-step instructions on how to The first and the simplest step consists in adjusting the instantiate them for a specific museum and the type of names of the semantic dimensions used to annotate visit personalization that they support is provided to the content network (e.g. thematic threads, output assist curators in their task. This catalogue is offered in languages, types of audience) to the specific domain of the form of an electronic magazine of digitally the new exhibition (Figure 3). augmented museum experiences already tested in museums or outdoor cultural heritage sites (see figure 2). Curators may browse the magazine and take inspiration from what other museums have successfully adopted for their visitors. Figure 3. The semantic dimensions of the experience schema for the Atlantic Wall exhibition: changing the dimensions is an easy approach to adapt the content structure when porting the Figure 4 The editing form to define a content item (in this case experience schema to a new exhibition. an mp4 video linked to a specific point of interest, C1-Dunes, a location of the exhibition Atlantic Wall) 29 The following step consists in replacing the content annotate the content and the sensor devices used to material: the narrative part of the experience schema. capture the visitor’s interaction (Figure 5). Figure 4 shows how a curator can define new content (a narrative item) for a specific theme, point of interest When the editing is complete, the package of the and language. During the phase of content preparation, content items that build up the narrative and the rules curators may use their proprietary content or benefit that govern the context-dependent play during the visit from the contextual search facilities that support the are transferred to the onsite server that manages their search in Europeana (a virtual European library with application. the goal to make Europe's cultural heritage accessible 5) of content items that are relevant for the current Preliminary Evaluation experience editing task. The authoring tool was developed through an iterative design-evaluation-redesign process in strict collaboration with cultural heritage professionals. A preliminary evaluation cycle took place over a period of three weeks in the form of a distributed expert evaluation on the first advanced graphical mockup of the system. In particular, 11 members of the meSch consortium participated in the collaborative effort, as well as 4 cultural heritage professionals, who are not part of the meSch consortium. The evaluation consisted in a series of small tasks that each curator had to perform individually to adapt a simple recipe. The curators were asked to reflect aloud on their actions on the interface and to provide impromptu comments. In Figure 5. The mapping between the semantic categories that total, 159 comments were collected. The majority of annotate the content and the devices that sense the interaction the comments and feedbacks gathered concerned context (for example an NFC tag attached to the smart replica ameliorations of the interface, the navigation within the is associated to language and theme preferences of the visitor) application, the clarity and intuitiveness of the design The interaction scripts may be adapted as well though and features regarding the collaborative and open use that may require higher technical skills. Yet, a curator of the platform. Many comments were also gathered on can customize an interaction script to the specific needs the vocabulary and terminology used within the of a new exhibition by setting the script parameters: for interface (e.g. the metaphors of the “magazine”, the example, the distance range for recognizing proximity “recipe”). Many comments were also relevant for the and the mapping between the semantic categories that intuitiveness and ease of reuse of the available scripts, as this part of the authoring environment is maybe the 5http://www.europeana.eu/ most “technical” one. 30 This formative evaluation greatly influenced the Acknowledgement following stage of partial redesign and actual This work was done as part of the Material EncounterS implementation of the authoring interface. The second with digital Cultural Heritage (meSch) project. The high-fidelity prototype was then tested for usability by project (2013-2017) receives funding from the other three cultural heritage professionals who had to European Community’s Seventh Framework Programme perform in autonomy a controlled task of recipe reuse, ‘ICT for access to cultural resources’ (ICT Call 9: FP7- resulting in further ameliorations. The third version of ICT-2011-9) under the Grant Agreement 600851. the system will now be used in open workshops (“authoring feasts”) to engage potential users of the References system in creating their own smart objects. 1. Dudley, S. (2010). Museum materialities: Objects, sense and feeling. In S. Dudley (ed.). Museum Materialities: Objects, Engagements, Conclusion Interpretations. Routledge. In this paper, we described a research prototype for 2. Marshall, M.T., Dulake N., Ciolfi L., Duranti D., authoring tangible and embodied interaction with digital Kockelkorn H., Petrelli D. (2016) Using Tangible content by cultural heritage professionals. The Smart Replicas as Controls for an Interactive approach is to empower cultural heritage professionals Museum Exhibition. In Proceedings of TEI 2016 with tools that guide them through a do-it-yourself 3. Petrelli D., Ciolfi L., van Dijk D., Hornecker E., Not process of creating or adapting digitally augmented E., Schmidt A. (2013) Integrating material and experiences for their own museum spaces. We propose digital: a new way for cultural heritage. ACM a formalism, the experience schema, that is based on a interactions magazine 20 (4), 58 clear separation of content from interaction, and aims 4. Zancanaro, M., Not E., Petrelli D., Marshall M., van at facilitating the preparation and the reuse of narrative Dijk T., Risseeuw M., van Dijk D., Venturini A., threads that can be adapted to different visitors and Cavada D., Kubitza T. (2015) Recipes for tangible types of experience and interaction strategies that and embodied visit experiences. In Proceedings of MW2015: Museums and the Web 2015. describe how content should be released in context while experiencing the objects and the space.