This work presents an overview of the EU-funded project EMOTIVE (Emotive virtual cultural experiences through personalized storytelling). EMOTIVE works from the premise that cultural sites are, in fact, highly emotional places, seedbeds not just of knowledge, but of emotional resonance and human connection. From 2016-2019, the EMOTIVE consortium will research, design, develop and evaluate methods and tools that can support the cultural and creative industries in creating narratives and experiences which draw on the power of 'emotive storytelling', both on site and virtually. This work focuses on the project objectives and results so far and presents identified challenges.
Storytelling applies to nearly everything we do. Whether it is to inform, persuade,
entertain, motivate or inspire, we all tell stories every day of our lives
The project EMOTIVE (Emotive virtual cultural experiences through personalized storytelling - https://www.emotiveproject.eu) is a Research and Innovation (RIA) action, addressing the topic CULT-COOP-08-2016: Virtual museums and social platform on European digital heritage, memory, identity and cultural interaction. EMOTIVE works from the premise that cultural sites are, in fact, highly emotional places. That regardless of age, location or state of preservation, they are seedbeds not just of knowledge, but of emotional resonance and human connection. From 20162019, the EMOTIVE consortium will research, design, develop and evaluate methods and tools that can support the cultural and creative industries in creating narratives and experiences which draw on the power of ―emotive storytelling‖. The output of this process will be a number of prototype tools and applications for heritage professionals and visitors that produce interactive, personalized, emotionally resonant digital experiences for museums and cultural sites.
This work provides an overview of the project objectives and partners and outlines its main research outcomes and open issues. Section 2 presents the EMOTIVE consortium and 3 an overview of the project and its objectives. Sections 4 and 5 highlight the project main findings, 6 the open challenges and 7 concludes the paper. 2
The EMOTIVE consortium brings together the resources of 8 participating organizations from 5 European countries (UK, Greece, France, Italy and Ireland), each an expert in their respective field and with significant research experience. The consortium includes industrial partners: EXUS Software Ltd (EXUS), Diginext Sarl (DXT), NOHO Limited (NOHO); academia and research institutions: ATHENA Research and Innovation Center in Information, Communication & Knowledge Technologies (ATHENA), University of York (YORK), Institut National de Recherche en Informatique et Automatique (INRIA), Consiglio Nazionale delle Ricerche (CNR), University of Glasgow (UGLA) and user-related partners: YORK, UGLA, and NOHO.
The research organisations in EMOTIVE carry out the foundational scientific research of the project. On the one hand lies ATHENA, a research centre closely affiliated with the University of Athens (UoA), Department of Informatics and Telecommunications) in the sector of Information and Knowledge Technologies, digital cultural heritage applications and virtual reality, INRIA in image-based and mixed rendering techniques, and CNR in developing new approaches and novel techniques for exploiting digital fabrication techniques. On the other hand, the YORK Department of Archaeology and UGLA provide their extensive expertise in the conceptualization and evaluation of digital experiences for cultural heritage.
Academic and applied research is complemented by the industrial partners EXUS and DXT who bring not only their technical expertise into the project, but also the prospect of commercial exploitation of EMOTIVE results. The team is complemented by NOHO, an SME which designs and creates video and interactive content for the cultural heritage sector with a strong emphasis on narrative design.
YORK and UGLA provide access to the EMOTIVE consortium to two UNESCO World heritage archaeological sites of great importance, the Çatalhöyük Neolithic site and the Roman Antonine Wall site in the UK and also the Hunterian Museum, one of the leading university museums in the world.
The principal objective of the EMOTIVE project is to research, design, develop and evaluate methods and tools that can support the cultural and creative industries in creating digital cultural heritage experiences, on-site and virtual, which draw on the power of 'emotive storytelling'. This means storytelling that can engage visitors, trigger their emotions, connect them to other people around the world, and enhance their understanding, imagination and, ultimately, their experience of cultural sites and content. EMOTIVE will do this by providing the means to authors of cultural products to create high-quality, interactive, personalized digital stories that can highlight the unique and particular characteristics of the specific sites and collections.
The EMOTIVE project targets two main groups of users: 1. Authors: they are members of the cultural and creative industries in charge of creating interactive cultural experiences (i.e., interactive stories) and making them available to the visitors.
2. Visitors: they are the people visiting the site and experiencing it through the cultural experiences created by the authors
More specifically, the main objectives of the EMOTIVE project include: Design a framework of best practices and guidelines for creating emotive cultural scenarios/stories for virtual and on-site museums, with a visitorcentric approach (Section 4.1) Implement an integrated set of authoring tools for all stages of production of an EMOTIVE story combined with an experiencing system, a powerful and generic infrastructure for storing, deploying, and presenting the EMOTIVE stories on mobile and desktop devices coupled with a storytelling engine. (Section 4.2) Develop simple methods of reconstructing physical space and producing digital 3D environments (through image-based modelling) and physical 3D objects. (Section 4.3) Investigate the production of physical artefacts designed to enhance the visitor experience, encouraging further visits to the cultural site. (Section 4.4) Develop a methodology for the meaningful, well-rounded evaluation of tools and experiences for cultural heritage (Section 5) and apply and evaluate the project methodology in a number of use cases (Section 4.5)
The following sections provide an overview of the aforementioned objectives. 4 4.1
The EMOTIVE project aims to move away from privileging didactic learning outcomes to explore other ways audiences feel and experience cultural heritage. Employing emotive storytelling in the context of cultural heritage is a unique form of art, with distinct differences from other art forms, such as movies, books, or video games. Heritage sites have tended to rely on didactic presentation of information and conventional publication media (physical signage, printed brochures and maps, etc.) to engage their audiences. However, when confronted with the opportunity to push beyond these traditions - to deploy personal, emotive narrative that is delivered via various means, including digitally - their promise for attracting and retaining visitors, and for reshaping our understandings and appreciation of the past, seems vast.
Through emotive experiences, visitors are able to interact with the site‘s
interpretation, change it and move it in the direction they want. The significance of such an
approach for facilitating engagement, resonance, care and commitment to cultural
heritage cannot be overestimated
The first version of the EMOTIVE conceptual framework (EMOTIVE Deliverable 5.1) is constantly being updated to reflect the findings of the iterative design, usercentered approach followed within the project. 4.2
The goal of the EMOTIVE Authoring Tool (EAT) is to enable cultural heritage and creative industry professionals to create compelling emotive storytelling experiences. These experiences will then be executed on site through mobile apps and situated displays or off-site, through web applications.
The EAT has been designed to offer a wide variety of tools to address a variety of issues. The first requirement to which the EAT answers is to allow the various authors to gradually construct the experiences using different tools according to the experience development stage. The EAT thus responds to the diversity of the professions of the authors who use it and is meant to be used by authors with different degrees of expertise, ranging from average computer users without any knowledge of computer science, to computer programmers. For this reason, the EAT is structured in conceptual layers, with each one of them addressed using a single or several of the implemented tools. These include the following:
Story Design Editor. The Story Design Editor is a web application that enables authors to design an interactive storytelling experience. This tool is meant to allow authors to draft their first ideas using a simple text editor, build the narrative interactivity within this experience scenario and prescribe the way that these ideas need to be implemented in the final experience.
Storyboard Editor. The Storyboard Editor is a web application that allows end users with storytelling skills to produce the experience they designed in the Story design editor. This editor is addressed to less experienced authors and allows them to develop their ideas with simple activity templates. Authors with technical skills can develop the experience further, if needed, using the Visual Scenario Editor.
Visual Scenario Editor. The Visual Scenario Editor enables authors to create compelling storytelling experiences featuring visually rich content, notably in virtual or augmented reality through visual programming mechanisms, wizards and a Scripting Editor for computer programmers to code the most complex parts of storytelling experiences.
Floorplan Editor. The Floor Plan Editor is a web application that allows building virtual environments based on floor plans, panoramas and images. The floor plans can then be integrated with the experience produced by the Storyboard Editor and made available on the EMOTIVE platform for offsite visits.
Mixed Reality Plugin. The Mixed Reality Plugin is a set of tools for the Unity game engine that allow game developers to import 3D reconstruction assets of a real scene obtained from photos. In addition, it allows photorealistic rendering of the reconstructed scene by means of image based rendering algorithms. Additional tools are provided with the plugin to allow basic manipulation of the reconstructed scene.
Section 4.5 presents examples of the use of the various EAT tools to develop emotive experiences. The tools are available for demonstration and testing to interested external parties after communication with the EMOTIVE project team. 4.3
In the past years, significant progress has been made in different fields of Computer
Vision and Computer Graphics, which in conjunction with the increase of
computational power thanks to hardware improvements has led to a different approach for
modeling and rendering of real environments based on the acquisition of real
photographsn. This Image-Based Rendering
Producing such environments using IBR usually requires the following steps:
Photographs acquisition: given a certain area of interest, we need to capture as
much data as possible by taking pictures of the parts of the scene that we want to
include in the reconstruction and the virtual immersive experience. We need some
overlap between photographs in order to apply Multi-View Stereo algorithms (MVS) on
that dataset of images
Multi-View Stereo Reconstruction
representing the reconstruction, the calibration of the cameras along this space and the original images that conform the dataset, we can use IBR algorithms in order to render the scene, preserving view dependent effects like reflection highlights and giving and overall better look in terms of ―realism‖ if we compare it to the result of the traditional pipeline of modeling and shading by hand. Another advantage of rendering by IBR algorithms is that it scales well when increasing the complexity of the scene (number of light sources, number of polygons, etc.).
By using this approach, a non-technical person can easily create the virtual space
by taking pictures, and running an offline process of reconstruction (done only once).
The project hads developed several new IBR algorithms and some in Unity to import
such assets, modify them to a certain extent and render them as a background
allowing content creators and other skilled persons to develop emotional engaging stories
as any other video game made with Unity 3D engine. We have also developed a new
approach allowing small modifications to be made in the scene, which is one of the
major challenges in IBR
Interaction with physical exhibit replicas has been considered to foster engagement for visitors. However this kind of enhanced experience comes with a cost: the production of replicas is still definitely not cheap and can easily reach hundreds of euros for each custom object. Even the recent advances in digital fabrication technologies, through the use of 3D printing machines based on Fused Deposit Modeling, have not significantly addressed this problem: these devices are usually very slow and the creation of a single moderately sized object (in the order of tens of centimeters) can require more than of one day of device time.
These technological / economic constraints pose significant limits in the easy use of physical replicas in the museum context, given that handling the objects wears down the objects and therefore imposes scheduled substitution practices.
For all these reasons we have developed a new fabrication technology based on casting that is aimed to enable the cheap production of replicas in small numbers. The main idea is to leverage silicon mold casting, an approach used often for high-quality reproduction of art objects but addressing its limitations and complexity by mean of sophisticated geometry processing algorithms. With this approach, a physical object is submerged in liquid silicone; the cured silicone forms a mold around the object; then, the prototype is extracted by manually cutting and opening the silicone mold. Multiple copies can be cast by filling the silicone mold with a liquid casting material such as resin or gypsum.
While conceptually simple, silicone mold casting may become extremely challenging when applied to non-trivial shapes, and requires the intervention of skilled professionals. For example, objects with handles usually need a set of carefully placed extra cuts to make the extraction physically possible. Moreover, venting pipes have to be attached to the prototype object before submersion in liquid silicone, to let the air flow out and avoid artifacts in the replicas due to trapped air bubbles.
We have developed a new method
The project has also produced experiences for other institutions that serve as Living Labs for the project and test and apply its concepts and technology including the York Minster Cathedral.
Central to the project is a user- and visitor-centric design approach and an iterative collaborative design and evaluation process. The evaluation work at EMOTIVE includes developing a methodology for the meaningful, well-rounded evaluation of tools and experiences, evaluating the usability and functionality of individual EMOTIVE tools, and the quality and effect of the EMOTIVE experiences.
An important objective of EMOTIVE is to study, assess and analyse the effect that the tools and experiences designed as part of the project have on both authors and end users/visitors. The project also aims to evaluate how the overall interpretative philosophy underpinning it and the power of ‗EMOTIVE experiences‘ engages different types of users and in different contexts and situations.
The project employs a multi-part, qualitative and quantitative, mixed-methods evaluation framework, which draws upon and combines approaches applied over many years in museum studies, psychology, media, education, cultural studies, and HCI. EMOTIVE evaluation will examine both usability and user experience of tools, methodologies and experiences, combining externally reporting on user experience (e.g., via observation of visitor emotional and bodily expression), and through both verbal and non-verbal visitor self-reports (e.g., via both visual and word-based questionnaire, and spoken interview). 6
The EMOTIVE approach to digital cultural heritage has identified several open issues and challenges at the conceptual, design and evaluation level.
As already mentioned, the project focuses on emotions and the challenge of their
integration on the project conceptual framework.
Related to the challenge of defining a conceptual framework for emotive experiences is the issue of evaluation. The challenge is to define and describe a detailed methodological and practical approach for evaluating emotive experiences, with the aspiration to become a useful framework for others in the field. It is of key importance to identify what should be evaluated in different types of experiences and what is the most effective way to accomplish it. To this day there has not been a standard evaluation framework to guide cultural heritage experience evaluation and, more importantly, not one that takes emotions into account.
Furthermore, over the last few years, UX evaluation has also looked at
physiological reactions (e.g. skin conductance, heart rate, facial muscle activity, cortical activity,
startle reflex or eye blink magnitude) as measurements of the intensity and quality of
an individual‘s internal affective experience
Lastly, EMOTIVE has been working on the authoring of experiences, seeking to understand the needs and support the complex workflow of their design and implementation. Engaging with different cultural heritage practitioners for the testing of its toolset and their application in real world settings is very challenging but crucial. 7
This paper presented an overview of the EMOTIVE project, focusing on its overall conceptual framework for emotive experiences in cultural heritage as well as on the tools developed within the project to support the authoring of such experiences. The emotive approach is addressed both to on-site and virtual visitors and work within the project is also dedicated to the evaluation perspectives of such experiences.
EMOTIVE will continue developing its methodology and tools through a usercentered iterative design approach, working closely with its two cultural heritage sites as well as with external sites, including the York Minster, the Athens Ancient Agora and various other museums and sites, so that there is a continuous feedback loop between the project and its intended users, authors and visitors.