In these early days of educational Virtual Reality (VR) applications, it is critical to establish best practices for exploring quantitative and qualitative data revealing the subtle relationship between personal virtual experiences and learning. In contrast to prior research on virtual environments, the evaluation of a personalized VR experience offered by a prototype is unique with but requires methods for evaluating metrics. Our paper suggests the need for the exploration of metrics for persuasive prototypes for virtual experiences.
A personalized Virtual Reality (VR) experience can provide profound means of knowledge transfer. Essentially, VR allows a user to take their own path through a contextualized knowledge-base in a realistic (or non-realistic), natural (or virtual), interactive way. Additionally, the virtual surroundings evoke psychological and physical reactions, that potentially have deeper impact than other forms of media.
Evaluating the illusive impact of a personal experience is difficult. Several metrics have been created to evaluate various virtual experiences, including the Presence Questionnaire and the Immersive Tendency Questionnaire. These types of surveys have focused on a fairly structured or guided experience in a typical desktop environment. As we move further into the 21st century, the personalized VR experience has become more persuasive and open though self-guided or UI guided experiences. 1.1
The use of VR and digital media seem to have created a “new realm of interaction” [2] (p.132) in the 21st century. Even though, there are a variety of concerns from too much screen time to logistical lack of devices or non-functioning issues [3-5], a deeper understanding of why and how to integrate VR into pedagogical practices is needed. A growing number of educators are interested in the “interaction age” in which students and teachers shift their expectations to adapt to the changing job market [2]. These concerns drive the need to explore 21st century learning in education, especially as it links to VR.
Today, many digital technologies are more closely linked to this personal experience of interaction. Technology and in particular some VR developments allow, encourage, and force interdisciplinary applications [1]. Rather than watching television and increasing the amount of screen time for students, a warning from the American Academy of Pediatrics (AAP), we need to utilize this persuasive technology by creating digitally literate learners who use technology to interact and find information. Prior research in VR describes how we can use VR to promote skills and knowledge through its immersive and interactive qualities [6].
Some researchers have begun to explore quasi-experimental ways of measuring successful VR experiences through various knowledge pre- and post-tests, focusing on measuring content knowledge [7, 8]. Other researchers have used surveys or questionnaires to measure the VR experience in general [9]. Still other researchers have measured presence, immersion, and flow as a way of understanding immersion and interaction, which can lead to learning [10, 11]. There are several survey questionnaires that have been developed and validated that would be appropriate for measuring learning, such as the Presence Questionnaire and the Immersion Tendency Questionnaire [11, 12].
Presence is described as a “psychological state of being there mediated by an environment that engages our senses, captures our attention, and fosters our active involvement” [11] (p.298). Immersion is also a psychological state and can be characterized as “perceiving oneself to enveloped by, included in, and interacting with an environment that provides a continuous stream of stimuli and experiences” [11] (p.299).
Another questionnaire that has potential to capture the subtle consequences of VR focuses on flow. Flow is a state where “people feel involved in meaningful actions, maintain a sense of control and stay focused on a goal” [10] (p.506). The flow experience “seems to occur only when a person is actively engaged in some form of clearly specified interaction with the environment” [13] (p.43). Similar to presence and immersion, flow is focuses on active engagement within an environment. For example, Bressler and Bodzin [10] used a short flow state scale to measure flow in a postsurvey with students. These survey questionnaires could be the grounding for measuring some 21st century experiences and could be combined with knowledge tests to measure VR experiences. 2
The persuasive prototype was built for an exhibit for the Royal British Columbia (BC) Museum for children and young adults. The Royal BC Museum wants to invite users along a personal journey back to BC in the mid 1800s. The experience includes information about the era and the location including a “flyover” experience (Fig. 1) along with an opportunity to “pan for gold”—meaning the user must, in VR, scoop sand and water into a pan and gently agitate it to allow the gold to sink to the bottom of the pan.
This prototype suggests several possibilities for VR in informal learning environments, like museums. Four major contributions of this persuasive prototype include: 1) increase interest/attendance of young adults to the Royal BC Museum, which is the overall goal of the Royal BC Museum and can have dissemination potential for other museums; 2) increase our understanding of the role of VR in education in the 21st century and if this should be the next steps in educational technologies or if VR should or should not be encouraged in schools or informal settings or both; 3) discover how we learn when using media and technology devices, and in particular do immersive 3-D technologies reinforce or suppress learning activities; and 4) refine metrics for measuring learning, presence, immersion, and flow in VR settings by continuing to test the currently validated surveys and refine for research and educational purposes. VR does not just mean playing games and watching videos with a funky head mounted display. This misconception has led to a degradation of VR or use without pedagogical support, which in turn has led many to not use VR at all.
A growing number of educators are interested in the “interaction age” in which students and teachers shift their expectations to adapt to the changing job market [2]. Being technologically literate can be quite difficult as many presume that students know technology just because they are “digital natives”. They know how to text or Instagram or take a selfie, but many nuances of technology are not known to them. Teachers increasingly encourage the exploration of different technologies through critical thought. Part of the issue lies not in how to use technology to support traditional models of education, but rather to shift thinking with pedagogical integration.
With the rapid evolution of technology and the increasing pressures to use VR, society has different expectations. It is not just the use of computers, or iPads or VR headsets or other applications associated with this media, rather it is how we use these devices pedagogically to encourage thinking and learning of hard and soft skills. Perhaps most importantly, as researchers, the question is: how will we measure learning to establish if VR enhances pedagogical practices?
In order to address some of these concerns, we need to evaluate the metrics used. Combining the metrics of presence, immersion, and flow, we can have a better scope of user’s individual experiences. However, these surveys have many overlaps in questions. For example, “I felt I was in control of what I was doing” and “How much were you able to control the events”. This overlap needs to be addressed. In order to measure the persuasiveness of this technology, we propose using a combination of these methods of the gold rush exhibit and evaluate the surveys using the following questions: 1) What metrics should be considered in the user survey? 2) How can the relevance of quantitative metrics be determined? 3) How does each metric weigh into the experience, and how are they related? 4) How can we quickly explore and navigate between quantitative and qualitative results?
Using these questions for evaluation and reduced questionnaires (Fig. 2), we can begin to get an understanding of the value of the proposed prototype. Customizing validated metrics, identifying key factors, and efficiently exploring the general relationship between quantitative and qualitative results allowed us to explore subtle relationships and consequences.
Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12 Q13 Q14
How completely were you able to actively survey or search the environment using vision? How involved were you in the virtual environment experience? How quickly did you adjust to the virtual environment experience? How much did the visual display quality interfere or distract you from performing assigned tasks or required activities? How much did the control devices (handheld) interfere with the performance of assigned tasks or with other activities? How well could you concentrate on the assigned tasks or required activities rather than the actual VR mechanisms (headset/handles) used to perform those tasks or activities? How much did the auditory (sound) aspects of the environment involve you? How well could you move or manipulate objects in the virtual environment? I was challenged and I felt I could meet the challenge.
How much did you lose track of normal time outside of the virtual experience? Did you enjoy what you were doing? Were you 'in the zone'? How mentally alert do you feel at the present time?
How good are you at blocking out external distractions when you are involved in something?
We explored the factors addressed in the questions using brushing (or selecting) in a Parallel Coordinate Graph (PCG). This visualization of the dataset allows us to explore how individual results vary (ids 1-60)(Fig. 3). We additionally splayed the values slightly, so that the individual results can be drilled into and explored for opportunities to personalize the virtual experience. In this graph, values of 4 and above for each of the “enjoy” and “in the zone” factors have been coloured green, values of 3 are blue, and below 3 in either category are red.
Preliminary investigation as to why there is so much variation in these results reveal issues of expectations for realism, familiarity with controllers, motion sickness and even game mechanic glitches. 3
Our next and related project in this series will include satellite imagery for a more realistic experience (Fig. 4), addressing the ways in which we failed to meet user expectations in the first prototype. Our goal is to design a personalized persuasive application for emergency preparedness. In this instance, tsunamis as historically reported to have impacted Port Alberni in British Columbia. If successful, the experience will persuade users to improve their earthquake and tsunami preparedness in the next generation, but we need to further explore how to personalize the experience for maximum impact.
More research is needed to directly explore how to precisely measure presence, immersion, and flow. Our VR prototype suggests the value proposition of persuasive technologies developed to enhance personal experiences.
In terms of future work with the prototype, we intend to test with a larger (more than 50) group of users to evaluation the metrics. Additionally, we would like to address the level of interest and engagement suggested by the users. This will contribute to the growing interest, knowledge, and research of VR in informal and formal learning settings.
Acknowledgement The Tsunami project relies on modeling from scientists at Oceans Network Canada, with support from IBM and satellite imagery from Urthecast.
Kereluik, K., et al., What knowledge is of most worth: Teacher knowledge for 21st century learning. Journal of Digital Learning in Teacher Education, 2013. 29(4): p. 127-140. Saavedra, A.R. and V.D. Opfer, Learning 21st-century skills requires 21st-century teaching. Phi Delta Kappan, 2012. 94(2): p. 8-13.
Hanson, K. and B.E. Shelton, Design and development of virtual reality: Analysis of challenges faced by educators. Journal of Educational Technology & Society, 2008. 11(1). Huang, H.-M., Toward constructivism for adult learners in online learning environments. British Journal of Educational Technology, 2002. 33(1): p. 27-37.
Huang, H.-M., U. Rauch, and S.-S. Liaw, Investigating learners’ attitudes toward virtual reality learning environments: Based on a constructivist approach. Computers & Education, 2010. 55(3): p. 1171-1182.
Bricken, M. and C.M. Byrne, Summer students in virtual reality: A pilot study on educational applications of virtual reality technology. 1992, University of Washington.
Chen, C.-T., Development and evaluation of senior high school courses on emerging technology: A case study of a course on virtual reality. Turkish Online Journal of Educational Technology - TOJET, 2012. 11(1): p. 46-59.
Hauptman, H., Enhancement of spatial thinking with virtual spaces 1.0. Computers & Education, 2010. 54(1): p. 123-135.
Tcha-Tokey, K., et al., Propostion and validation of a questionnaire to measure the user experience in immersive virtual environments. The International Journal of Virtual Reality, 2016. 16(1): p. 33-48. 10. 11. 12. 13.