=Paper=
{{Paper
|id=Vol-2089/1_ralph
|storemode=property
|title=Personalizing Virtual Experiences: Metrics for Persuasive Prototypes
|pdfUrl=https://ceur-ws.org/Vol-2089/1_Ralph.pdf
|volume=Vol-2089
|authors=Rachel Ralph,Andy Wynden,Matt Richardson,Merrick Paquin-Mannington,Julian Levy,Eric Dahl,Aldyn Chwelos,Tania Lado Insua,Derek Jacoby,Yvonne Coady,Larry Bafia,Emily Burt,JongJik Kim,William Parkinson
|dblpUrl=https://dblp.org/rec/conf/persuasive/RalphWRPLDCIJCB18
}}
==Personalizing Virtual Experiences: Metrics for Persuasive Prototypes==
https://ceur-ws.org/Vol-2089/1_Ralph.pdf
Personalizing Virtual Experiences: Metrics for
Persuasive Prototypes
Rachel Ralph, Andy Wynden, Matt Richardson, Merrick Paquin-Mannington,
Julian Levy, Eric Dahl, Aldyn Chwelos, Tania Lado Insua,
Derek Jacoby, Yvonne Coady1
Larry Bafia, Emily Burt, JongJik Kim2
William Parkinson3
1 University of Victoria, Victoria, BC, Canada
2 The Centre for Digital Media, Vancouver, BC, Canada
3 Urthecast, Vancouver, BC, Canada
rachel.ralph@ubc.ca
Abstract. In these early days of educational Virtual Reality (VR) applications,
it is critical to establish best practices for exploring quantitative and qualitative
data revealing the subtle relationship between personal virtual experiences and
learning. In contrast to prior research on virtual environments, the evaluation of
a personalized VR experience offered by a prototype is unique with but requires
methods for evaluating metrics. Our paper suggests the need for the exploration
of metrics for persuasive prototypes for virtual experiences.
Keywords: Education, Learning, Metrics, Virtual Reality, Prototypes.
1 Introduction
A personalized Virtual Reality (VR) experience can provide profound means of
knowledge transfer. Essentially, VR allows a user to take their own path through a
contextualized knowledge-base in a realistic (or non-realistic), natural (or virtual),
interactive way. Additionally, the virtual surroundings evoke psychological and phys-
ical reactions, that potentially have deeper impact than other forms of media.
Evaluating the illusive impact of a personal experience is difficult. Several metrics
have been created to evaluate various virtual experiences, including the Presence
Questionnaire and the Immersive Tendency Questionnaire. These types of surveys
have focused on a fairly structured or guided experience in a typical desktop envi-
ronment. As we move further into the 21st century, the personalized VR experience
has become more persuasive and open though self-guided or UI guided experiences.
1.1 Related Work
The use of VR and digital media seem to have created a “new realm of interaction”
[2] (p.132) in the 21st century. Even though, there are a variety of concerns from too
Copyright © 2018 held by the paper’s authors. Copying permitted for private and academic
purposes.
In: R. Orji, M. Kaptein, J. Ham, K. Oyibo, J. Nwokeji (eds.): Proceedings of the Personaliza-
tion in Persuasive Technology Workshop, Persuasive Technology 2018, Waterloo, Canada,
17-04-2018, published at http://ceur-ws.org
2 Personalizing Virtual Experiences: Metrics for Persuasive Prototypes
much screen time to logistical lack of devices or non-functioning issues [3-5], a deep-
er understanding of why and how to integrate VR into pedagogical practices is need-
ed. A growing number of educators are interested in the “interaction age” in which
students and teachers shift their expectations to adapt to the changing job market [2].
These concerns drive the need to explore 21st century learning in education, especial-
ly as it links to VR.
Today, many digital technologies are more closely linked to this personal experi-
ence of interaction. Technology and in particular some VR developments allow, en-
courage, and force interdisciplinary applications [1]. Rather than watching television
and increasing the amount of screen time for students, a warning from the American
Academy of Pediatrics (AAP), we need to utilize this persuasive technology by creat-
ing digitally literate learners who use technology to interact and find information.
Prior research in VR describes how we can use VR to promote skills and knowledge
through its immersive and interactive qualities [6].
Some researchers have begun to explore quasi-experimental ways of measuring
successful VR experiences through various knowledge pre- and post-tests, focusing
on measuring content knowledge [7, 8]. Other researchers have used surveys or ques-
tionnaires to measure the VR experience in general [9]. Still other researchers have
measured presence, immersion, and flow as a way of understanding immersion and
interaction, which can lead to learning [10, 11]. There are several survey question-
naires that have been developed and validated that would be appropriate for measur-
ing learning, such as the Presence Questionnaire and the Immersion Tendency Ques-
tionnaire [11, 12].
Presence is described as a “psychological state of being there mediated by an envi-
ronment that engages our senses, captures our attention, and fosters our active in-
volvement” [11] (p.298). Immersion is also a psychological state and can be charac-
terized as “perceiving oneself to enveloped by, included in, and interacting with an
environment that provides a continuous stream of stimuli and experiences” [11]
(p.299).
Another questionnaire that has potential to capture the subtle consequences of VR
focuses on flow. Flow is a state where “people feel involved in meaningful actions,
maintain a sense of control and stay focused on a goal” [10] (p.506). The flow experi-
ence “seems to occur only when a person is actively engaged in some form of clearly
specified interaction with the environment” [13] (p.43). Similar to presence and im-
mersion, flow is focuses on active engagement within an environment. For example,
Bressler and Bodzin [10] used a short flow state scale to measure flow in a post-
survey with students. These survey questionnaires could be the grounding for measur-
ing some 21st century experiences and could be combined with knowledge tests to
measure VR experiences.
2 Persuasive Prototypes
The persuasive prototype was built for an exhibit for the Royal British Columbia
(BC) Museum for children and young adults. The Royal BC Museum wants to invite
Personalizing Virtual Experiences: Metrics for Persuasive Prototypes 3
users along a personal journey back to BC in the mid 1800s. The experience includes
information about the era and the location including a “flyover” experience (Fig. 1)
along with an opportunity to “pan for gold”—meaning the user must, in VR, scoop
sand and water into a pan and gently agitate it to allow the gold to sink to the bottom
of the pan.
Fig. 1. Overview of gold panning canyon seen by participant in the “flyover”.
This prototype suggests several possibilities for VR in informal learning environ-
ments, like museums. Four major contributions of this persuasive prototype include:
1) increase interest/attendance of young adults to the Royal BC Museum, which is the
overall goal of the Royal BC Museum and can have dissemination potential for other
museums; 2) increase our understanding of the role of VR in education in the 21st
century and if this should be the next steps in educational technologies or if VR
should or should not be encouraged in schools or informal settings or both; 3) discov-
er how we learn when using media and technology devices, and in particular do im-
mersive 3-D technologies reinforce or suppress learning activities; and 4) refine met-
rics for measuring learning, presence, immersion, and flow in VR settings by continu-
ing to test the currently validated surveys and refine for research and educational pur-
poses. VR does not just mean playing games and watching videos with a funky head
mounted display. This misconception has led to a degradation of VR or use without
pedagogical support, which in turn has led many to not use VR at all.
A growing number of educators are interested in the “interaction age” in which
students and teachers shift their expectations to adapt to the changing job market [2].
Being technologically literate can be quite difficult as many presume that students
know technology just because they are “digital natives”. They know how to text or
Instagram or take a selfie, but many nuances of technology are not known to them.
Teachers increasingly encourage the exploration of different technologies through
critical thought. Part of the issue lies not in how to use technology to support tradi-
tional models of education, but rather to shift thinking with pedagogical integration.
With the rapid evolution of technology and the increasing pressures to use VR,
society has different expectations. It is not just the use of computers, or iPads or VR
4 Personalizing Virtual Experiences: Metrics for Persuasive Prototypes
headsets or other applications associated with this media, rather it is how we use these
devices pedagogically to encourage thinking and learning of hard and soft skills. Per-
haps most importantly, as researchers, the question is: how will we measure learning
to establish if VR enhances pedagogical practices?
In order to address some of these concerns, we need to evaluate the metrics used.
Combining the metrics of presence, immersion, and flow, we can have a better scope
of user’s individual experiences. However, these surveys have many overlaps in ques-
tions. For example, “I felt I was in control of what I was doing” and “How much were
you able to control the events”. This overlap needs to be addressed. In order to meas-
ure the persuasiveness of this technology, we propose using a combination of these
methods of the gold rush exhibit and evaluate the surveys using the following ques-
tions:
1) What metrics should be considered in the user survey?
2) How can the relevance of quantitative metrics be determined?
3) How does each metric weigh into the experience, and how are they related?
4) How can we quickly explore and navigate between quantitative and qualitative
results?
Using these questions for evaluation and reduced questionnaires (Fig. 2), we can
begin to get an understanding of the value of the proposed prototype. Customizing
validated metrics, identifying key factors, and efficiently exploring the general rela-
tionship between quantitative and qualitative results allowed us to explore subtle rela-
tionships and consequences.
Q1 How completely were you able to actively survey or search the environment using vision?
Q2 How involved were you in the virtual environment experience?
Q3 How quickly did you adjust to the virtual environment experience?
Q4 How much did the visual display quality interfere or distract you from performing assigned tasks
or required activities?
Q5 How much did the control devices (handheld) interfere with the performance of assigned tasks or
with other activities?
Q6 How well could you concentrate on the assigned tasks or required activities rather than the actual
VR mechanisms (headset/handles) used to perform those tasks or activities?
Q7 How much did the auditory (sound) aspects of the environment involve you?
Q8 How well could you move or manipulate objects in the virtual environment?
Q9 I was challenged and I felt I could meet the challenge.
Q10 How much did you lose track of normal time outside of the virtual experience?
Q11 Did you enjoy what you were doing?
Q12 Were you 'in the zone'?
Q13 How mentally alert do you feel at the present time?
Q14 How good are you at blocking out external distractions when you are involved in something?
Figure 2. Questions in our survey composed from questions from previous work.
Personalizing Virtual Experiences: Metrics for Persuasive Prototypes 5
We explored the factors addressed in the questions using brushing (or selecting) in
a Parallel Coordinate Graph (PCG). This visualization of the dataset allows us to ex-
plore how individual results vary (ids 1-60)(Fig. 3). We additionally splayed the val-
ues slightly, so that the individual results can be drilled into and explored for opportu-
nities to personalize the virtual experience. In this graph, values of 4 and above for
each of the “enjoy” and “in the zone” factors have been coloured green, values of 3
are blue, and below 3 in either category are red.
Figure 3: Parallel Coordinate Graph of factors from the survey questions.
Preliminary investigation as to why there is so much variation in these results re-
veal issues of expectations for realism, familiarity with controllers, motion sickness
and even game mechanic glitches.
3 Future Work
Figure 4. Urthecast satellite imagery of Port Alberni.
Our next and related project in this series will include satellite imagery for a more
realistic experience (Fig. 4), addressing the ways in which we failed to meet user
expectations in the first prototype. Our goal is to design a personalized persuasive
application for emergency preparedness. In this instance, tsunamis as historically
reported to have impacted Port Alberni in British Columbia. If successful, the experi-
ence will persuade users to improve their earthquake and tsunami preparedness in the
6 Personalizing Virtual Experiences: Metrics for Persuasive Prototypes
next generation, but we need to further explore how to personalize the experience for
maximum impact.
4 Conclusions
More research is needed to directly explore how to precisely measure presence,
immersion, and flow. Our VR prototype suggests the value proposition of persuasive
technologies developed to enhance personal experiences.
In terms of future work with the prototype, we intend to test with a larger (more
than 50) group of users to evaluation the metrics. Additionally, we would like to ad-
dress the level of interest and engagement suggested by the users. This will contribute
to the growing interest, knowledge, and research of VR in informal and formal learn-
ing settings.
Acknowledgement
The Tsunami project relies on modeling from scientists at Oceans Network Canada,
with support from IBM and satellite imagery from Urthecast.
References
1. Kereluik, K., et al., What knowledge is of most worth: Teacher knowledge for 21st century
learning. Journal of Digital Learning in Teacher Education, 2013. 29(4): p. 127-140.
2. Saavedra, A.R. and V.D. Opfer, Learning 21st-century skills requires 21st-century teaching. Phi
Delta Kappan, 2012. 94(2): p. 8-13.
3. Hanson, K. and B.E. Shelton, Design and development of virtual reality: Analysis of challenges
faced by educators. Journal of Educational Technology & Society, 2008. 11(1).
4. Huang, H.-M., Toward constructivism for adult learners in online learning environments.
British Journal of Educational Technology, 2002. 33(1): p. 27-37.
5. Huang, H.-M., U. Rauch, and S.-S. Liaw, Investigating learners’ attitudes toward virtual reality
learning environments: Based on a constructivist approach. Computers & Education, 2010.
55(3): p. 1171-1182.
6. Bricken, M. and C.M. Byrne, Summer students in virtual reality: A pilot study on educational
applications of virtual reality technology. 1992, University of Washington.
7. Chen, C.-T., Development and evaluation of senior high school courses on emerging
technology: A case study of a course on virtual reality. Turkish Online Journal of Educational
Technology - TOJET, 2012. 11(1): p. 46-59.
8. Hauptman, H., Enhancement of spatial thinking with virtual spaces 1.0. Computers &
Education, 2010. 54(1): p. 123-135.
9. Tcha-Tokey, K., et al., Propostion and validation of a questionnaire to measure the user
experience in immersive virtual environments. The International Journal of Virtual Reality,
2016. 16(1): p. 33-48.
Personalizing Virtual Experiences: Metrics for Persuasive Prototypes 7
10. Bressler, D.M. and A.M. Bodzin, A Mixed Methods Assessment of Students' Flow Experiences
during a Mobile Augmented Reality Science Game. Journal of Computer Assisted Learning,
2013. 29(6): p. 505-517.
11. Witmer, B.G., C.J. Jerome, and M.J. Singer, The factor structure of the presence questionnaire.
Presence: Teleoperators and Virtual Environments, 2005. 14(3): p. 298-312.
12. Witmer, B.G. and M.J. Singer, Measuring presence in virtual environments: A presence
questionnaire. Presence: Teleoperators and Virtual Environments, 1998. 7(3): p. 225-240.
13. Csikszentmihalyi, M., Flow: The psychology of optimal experience. 1990, New York, NY:
Harper and Row.