An Authoring Environment for Smart Objects in Museums: the meSch Approach

Abstract

The meSch project addresses the challenges of creating a personally meaningful, sensorily rich, and socially expanded museum visitor experience through tangible and embodied interaction with digital content. It is of paramount importance that cultural heritage professionals are directly involved in the design of those experiences. The meSch approach is to empower cultural heritage professionals with tools that guide them through a do-it-yourself process of creating or adapting digitally augmented experiences for their own museum spaces, therefore reducing the barriers of introducing Internet of Things technology in cultural heritage spaces.

-

Paste the appropriate copyright/license statement here. ACM now supports three different publication options: • ACM copyright: ACM holds the copyright on the work. This is the historical approach. • License: The author(s) retain copyright, but ACM receives an exclusive publication license. • Open Access: The author(s) wish to pay for the work to be open access. The additional fee must be paid to ACM.

This text field is large enough to hold the appropriate release statement assuming it is single-spaced in Verdana 7 point font. Please do not change the size of this text box.

Each submission will be assigned a unique DOI string to be included here.

Introduction

Within the Material Encounters with Digital Cultural Heritage (meSch1) project [ 3 ], we are working on technology to support the whole museum visiting experience by fostering a vision where materiality and rich sensory interaction add to the cognitive aspects for a personally meaningful experience. There is a wide recognition that such approach may greatly improve both the visit and also visitors’ appreciation for the museum’s cultural values [ 1 ]. The idea behind meSch is taking objects out of their cases (or using replicas like in [ 2 ]) to let visitors experience objects’ shape and weight while learning about them (figure 1), to engage the visitors at a physical level and put information in context without the distractions induced by interacting with extraneous devices, apps, and touch screens.

As an example, the meSch platform has been

demonstrated at the Atlantic Wall exhibition at Museon in The Hague, The Netherlands2. The Atlantic Wall was

1 http://mesch-project.eu

the German defense line along the European west coast erected during World War II to block the allied invasion. The exhibition was meant to reflect on how the city of The Hague and its citizens had been affected. The meSch platform has been used to enrich the display of historical artefacts with voices from the past, such as those of the German soldiers, the Dutch civilians and the civil servants3. Visitors could play these narratives by using ‘smart replicas’, reproductions of real museum objects (e.g. a reproduction of a surrogate tea bag, a 3D-printed replica of a Delft blue mug, and so on) enhanced with sensors. By placing their smart replica on hotspots on interactive cases visitors could listen to narratives and see historical photographs and video clips thus enriching the original museum objects on display4. From April to November 2015, more than 14,000 visitors interacted with the meSch smart replicas at the hotposts.

The active involvement of cultural heritage professionals in the preparation of this kind of experiences is of paramount importance, even if they often lack technical skills [ 3 ]. The meSch approach consists in fostering reuse over creation from scratch. A catalogue of reusable narrative and interaction strategies is provided as starting points that can be adapted and refined.

The ingredients for a digitally augmented experience

The approach to experience creation and personalization in meSch is based on a clear separation 3http://mw2016.museumsandtheweb.com/glami/the-hague-andthe-atlantic-wall-war-in-the-city-of-peace/ 2 http://www.museon.nl/en/exhibitions/hague-and-atlantic-wall 4 https://www.youtube.com/watch?v=sK3AdQU9kkc of content from interaction, and aims at facilitating the preparation and the reuse of (i) narrative threads that can be adapted to different visitors and types of experience and (ii) interaction strategies that describe how content should be released in context while experiencing the objects and the space. We define a declarative formalism, the experience schema that supports the description of the content items, the available interactions, the semantic annotations and the conditional rules that govern the adaptive composition of the experience that is delivered to visitors at runtime [ 4 ]. This is done through a 4layer data structure including: the narrative, the appliance, the device and the interaction script: (i) the narrative: a set of carefully selected digital content items, annotated into alternative thematic treads and levels of detail that revolve around the objects and places of an exhibition; (ii) the appliance: a declarative specification of the capabilities of the technology embedded in the museum premises (such as, for example, the possibility to detect the visitors’ proximity to a specific exhibit); (iii) the device: the description of the actual hardware required for deploying the system and its technical requirements; (iv) the interaction script: the interaction rules that govern when presentations should be started or how the system behaviour should be adjusted to personalize the experience to visitors’ context (e.g., a set of rules that activate the projection of the most appropriate content about an exhibit when visitors are in its vicinity). This multilayer definition of the experience schema concept supports the following properties: (i) reusability: an experience schema defines a skeleton of experience that can be reused with different content and in different physical settings to create many experience instances; (ii) modularity: parts of an experience schema can be replaced to create a new experience schema. For example, the interaction script can be changed to introduce different rules for firing the presentation of the same content in context. Another example is the possibility to experiment the same type of experience with an alternative hardware set-up; (iii) compositionality: different experience schemas can be composed together to create more articulated augmented exhibitions; (iv) reduction of complexity: authors can concentrate on the preparation of the necessary content and on its grouping according to the narrative dimensions exploited in the experience in a dedicated step. The rules for putting content in context are represented separately and can be edited by experience designers; (v) declarativity: the declarations contained in the appliance section of an experience schema allow to specify in a formal way the type of elementary interactions and behaviour events that are available to shape the experience; this allows to abstract from the actual hardware details and the low level sensor logs, thus facilitating the experimentation of different technological solutions.

The meSch scenario of authoring a digitally augmented visit experience

A catalogue of reusable narrative and interaction strategies with step-by-step instructions on how to instantiate them for a specific museum and the type of visit personalization that they support is provided to assist curators in their task. This catalogue is offered in the form of an electronic magazine of digitally augmented museum experiences already tested in museums or outdoor cultural heritage sites (see figure 2). Curators may browse the magazine and take inspiration from what other museums have successfully adopted for their visitors. Once they find a digitally augmented experience that may suit their needs, curators may replicate the “recipe” of the experience (which consists in its experience schema) and iteratively adapt it to their specific needs.

The first and the simplest step consists in adjusting the names of the semantic dimensions used to annotate the content network (e.g. thematic threads, output languages, types of audience) to the specific domain of the new exhibition (Figure 3).

Figure 4 The editing form to define a content item (in this case an mp4 video linked to a specific point of interest, C1-Dunes, a location of the exhibition Atlantic Wall) The following step consists in replacing the content material: the narrative part of the experience schema. Figure 4 shows how a curator can define new content (a narrative item) for a specific theme, point of interest and language. During the phase of content preparation, curators may use their proprietary content or benefit from the contextual search facilities that support the search in Europeana (a virtual European library with the goal to make Europe's cultural heritage accessible5) of content items that are relevant for the current experience editing task. The interaction scripts may be adapted as well though that may require higher technical skills. Yet, a curator can customize an interaction script to the specific needs of a new exhibition by setting the script parameters: for example, the distance range for recognizing proximity and the mapping between the semantic categories that

5http://www.europeana.eu/

annotate the content and the sensor devices used to capture the visitor’s interaction (Figure 5).

When the editing is complete, the package of the content items that build up the narrative and the rules that govern the context-dependent play during the visit are transferred to the onsite server that manages their application.

Preliminary Evaluation

The authoring tool was developed through an iterative design-evaluation-redesign process in strict collaboration with cultural heritage professionals. A preliminary evaluation cycle took place over a period of three weeks in the form of a distributed expert evaluation on the first advanced graphical mockup of the system. In particular, 11 members of the meSch consortium participated in the collaborative effort, as well as 4 cultural heritage professionals, who are not part of the meSch consortium. The evaluation consisted in a series of small tasks that each curator had to perform individually to adapt a simple recipe. The curators were asked to reflect aloud on their actions on the interface and to provide impromptu comments. In total, 159 comments were collected. The majority of the comments and feedbacks gathered concerned ameliorations of the interface, the navigation within the application, the clarity and intuitiveness of the design and features regarding the collaborative and open use of the platform. Many comments were also gathered on the vocabulary and terminology used within the interface (e.g. the metaphors of the “magazine”, the “recipe”). Many comments were also relevant for the intuitiveness and ease of reuse of the available scripts, as this part of the authoring environment is maybe the most “technical” one.

This formative evaluation greatly influenced the following stage of partial redesign and actual implementation of the authoring interface. The second high-fidelity prototype was then tested for usability by other three cultural heritage professionals who had to perform in autonomy a controlled task of recipe reuse, resulting in further ameliorations. The third version of the system will now be used in open workshops (“authoring feasts”) to engage potential users of the system in creating their own smart objects.

Conclusion

In this paper, we described a research prototype for authoring tangible and embodied interaction with digital content by cultural heritage professionals. The approach is to empower cultural heritage professionals with tools that guide them through a do-it-yourself process of creating or adapting digitally augmented experiences for their own museum spaces. We propose a formalism, the experience schema, that is based on a clear separation of content from interaction, and aims at facilitating the preparation and the reuse of narrative threads that can be adapted to different visitors and types of experience and interaction strategies that describe how content should be released in context while experiencing the objects and the space.

Acknowledgement

This work was done as part of the Material EncounterS with digital Cultural Heritage (meSch) project. The project (2013-2017) receives funding from the European Community’s Seventh Framework Programme ‘ICT for access to cultural resources’ (ICT Call 9: FP7ICT-2011-9) under the Grant Agreement 600851.

References

1. Dudley , S. ( 2010 ). Museum materialities: Objects, sense and feeling . In S. Dudley (ed.). Museum Materialities: Objects, Engagements, Interpretations. Routledge.

2. Marshall , M.T. , Dulake N. , Ciolfi L. , Duranti D. , Kockelkorn H. , Petrelli D. ( 2016 ) Using Tangible Smart Replicas as Controls for an Interactive Museum Exhibition . In Proceedings of TEI 2016

3. Petrelli D. , Ciolfi L., van Dijk D. , Hornecker E. , Not E. , Schmidt A. ( 2013 ) Integrating material and digital: a new way for cultural heritage . ACM interactions magazine 20 (4) , 58

4. Zancanaro , M. , Not E. , Petrelli D. , Marshall M. , van Dijk T. , Risseeuw M. , van Dijk D. , Venturini A. , Cavada D. , Kubitza T. ( 2015 ) Recipes for tangible and embodied visit experiences . In Proceedings of MW2015: Museums and the Web 2015 .