Mixed reality environments allow collaborative interaction with shared virtual objects to extend beyond participants wearing head mounted displays. Teams of collaborators can now share perspectives within a virtual space, and share their experiences with others through the use of green screens and realtime camera mixing. Our work proposes to explore commodity-based mixed reality as a personalizable persuasive technology. Specifically, our case studies are designed to study heightened connectedness to environmental and societal changes in Western Canada. By allowing participants to share their personal experience of history and the environment interactively through a series of small shifts in perspective, we hope to better empower individuals to act on global issues.
Our virtual reality (VR), augmented reality (AR), and mixed reality (MR) projects1 leverage new technologies to increase engagement and collaboration to enhance education, learning, and data analytics environments. A great deal of synergy exists with Virtual Reality for Impact2 , which is a program sponsored by VR headset maker HTC to use new technologies to immerse people into experiences that can create connections and empathy by sharing perspectives. The goal is to motivate society to act on global issues based on a shared sense of commonality within the United Nations Sustainable Development Goals (SDGs). Specific goals that touch on connections to environment and society include:
Goal 11. Make cities and human settlements inclusive, safe, resilient and sustainable Goal 12. Ensure sustainable consumption and production patterns 1 Supported in part through an Innovation Grant from Ericsson. 2 http://vrforimpact.com/
Goal 16. Promote peaceful and inclusive societies for sustainable development, provide access to justice for all and build effective, accountable institutions at all levels Recently, HTC Vive started a large scale, persuasive computing initiative called VR for Impact.
Our work-in-progress asks the question, can we heighten connectedness and empathy by personalizing the persuasive technologies involved in mixed reality with modeled user interactions which detect perception and attitude shifts? Specifically, this short paper overviews both the technology involved and the case studies we are pursuing. Section 2 overviews the hardware, software, and approximate costs for creating a proposed mixed reality museum installation at the BC Royal Museum. Section 3 outlines the historical contexts for the shared perspectives, involving over 140,000 images showing changes in Canada’s mountain environments and more than 700 ethnographies written about Coast Salish peoples. 2
Projects like Microsoft’s RoomAlive [
In terms of software, we leverage a popular framework for development of VR applications, Unity4, which can be augmented with plugins enabling easy deployment of mixed reality extensions. Open source projects capable of large scale execution across heterogeneous environments are also increasingly easy to deploy.
Interactions with virtual characters in our learning and analytics environments are created using the Microsoft Bot Framework, which allow us to imbue artificial personalities with realistic traits. With spoken input occurring through the Microsoft speech recognition engine, this leads to constrained, but natural feeling, dialogues not only with other participants but also with the system.
Modeling of user behaviors and attitudes is an area of particularly active research.
From explicit modeling techniques such as structured dialogue trees, to probabilistic
drama management techniques [
In terms of hardware, Table 1 shows how a minimally equipped mixed-reality
museum installation can be created less than $15,000 Canadian Dollars.
Our work-in-progress consists of two case studies underway, both currently developed
as interactive web-based mapping applications. First is the Mountain Legacy Project
[
The project started with upwards of 120,000 systematic glass plate negatives taken by land surveyors 1861-1953. More recently re-photographed images have been carefully added as a basis for comparison and change detection (Figure 2). The resulting image pairs can be analyzed, and the environmental impact of aggressive resource harvesting, such as clear cutting, can be made immediately evident to participants in this study.
The second case study involves Ethnographic Mapping in the Coast Salish World
[
Ultimately, these case studies will be combined, along with artifacts and scenarios
provided by our collaborators at the Royal BC Museum, to create an interactive
educational game which present situations drawn from these environmental and ethnographic
data sources, and presenting challenges in investigating historical and current patterns
of resource use and their effect on the world. The effective use of in-museum interactive
applications has been well- established [
By shifting of perspectives into the scenarios within this investigation, and modeling of user behaviors, it is hoped that attitudes can be quantified and effected throughout the gameplay. Our hypothesis is that even watching this gameplay can be of significant educational value, but that maximum attitudinal benefit is derived by the additional engagement and agency of being fully immersed within the context of a persuasive educational game. For example, users could watch each other, in a mixed reality environment, while they navigate through an historic scene. While exploring artifacts, the group could try to determine which people they are interacting with, and where they are in terms of time and place. In the context of our proposed methodology, personalized persuasion relies on feedback implicit and explicit user modeling data captured during the exploration of a historical context. Ultimately, we are attempting to detect changes in an individual that may correlate to a perspective shift. Once detected, we can guide the individual within their exploration of the environmental and societal conditions involved in the educational gameplay scenario.
Considering each set of goals aligned with their respective environmental or societal theme, we can highlight our proposed approach and key challenges as follows:
Goal 13. Take urgent action to combat climate change and its impacts Goal 15. Protect, restore and promote sustainable use of terrestrial ecosystems, sustainably manage forests, combat desertification, and halt and reverse land degradation and halt biodiversity loss Proposal. Immersive exploration of changes introduced by exploitative, unsustainable practices over time.
Challenge. Do small, shared perspective shifts heighten personal connectedness to environment? Goal 11. Make cities and human settlements inclusive, safe, resilient and sustainable Goal 16. Promote peaceful and inclusive societies for sustainable development, provide access to justice for all and build effective, accountable institutions at all levels Proposal. Immersive exploration of examples of people and place that defied discrete boundaries.
Challenge. Do personalized identity shifts heighten empathy to other groups within historic contexts? 5
In order for the United Nations Sustainable Development Goals to be realized globally, we need to start by motivating individuals locally. Personalizing persuasive technologies, such as mixed reality, may be an import way to increase the likelihood of desired behavior changes. By leveraging feedback through interaction with virtual objects and conversational systems, we plan to detect perspective shifts and capitalize on persuasive opportunities within a mixed reality setting.
The development of map-based analytics is well underway, and the gamification of these immersive analytic experiences is a natural next step supported by well-known techniques from the gaming industry. The integration of conversational systems and rich attitudinal modeling is more challenging, but using powerful cloud-based cognitive computing platforms in combination reduces much of this speech recognition and conversational dialogue burden. The analysis of attitudes and attitudinal shifts from response behavior is perhaps the most forward-looking and challenging of the approaches being worked towards in this paper, but also the challenge with the most implications for the design of effective learning and persuasive gaming applications.