Game based learning is popularly being adapted as a learning method that promotes student engagement. Such a system typically lies within non-immersive Virtual Reality system. In this research, we raise concerns that heavy focus on gamification might come at cost of quality learning. Extending to the same we explore three research questions centered around current challenges in such learning system, do such system follow principles of learning and what design implications can help such system ensure learning. We used a popular learning theory (Merrill's First Principles of Learning) as the foundation of this research. We identified three key challenges and proposed four design guidelines to address those challenges. The design guidelines were validated by con- ducting a user-study on a prototype system. The results show substantial positive implication of design guidelines on learning.
With advancements in technology, many educators are using technology enabled learning [
While the use of virtual reality (VR) in game-based learning systems (GLEs) has been shown to
have numerous benefits, such as increased motivation and engagement among learners, we raise a
concern that a heavy focus on gamification may come at the cost of quality learning. Gamification,
while effective in motivating learners, may place more emphasis on the game mechanics rather than the
educational content, leading to a potential loss of depth and breadth in learning. In other words, while
the use of games and other interactive elements can be engaging, they may not necessarily facilitate a
deep understanding of the subject matter or lead to the acquisition of meaningful skills and knowledge.
Moreover, it is important to consider that the use of VR in GLEs is still a relatively new field, and its
effectiveness and limitations are still being explored. To investigate the effectiveness of VR-enabled
game-based learning systems (GLEs) in promoting learning outcomes, we conducted a study that aimed
to identify the challenges associated with such systems considering established principles of learning
in the literature. Specifically, we utilized Merrill’s First Principle of Learning (MPL) [
In summary, we explore the following three research questions: • RQ1: What are the current challenges faced by learners using 2D non- immersive VR based learning systems to learn • RQ2: Does current 2D non-immersive VR based learning follow pedagogical principle of learning? • RQ3: What design implications can be considered to create a learning system that ensures learning even with gamification?
As stated in section above, we used Merrill’s First Principle of Learning (MPL) to assess the
instructional design quality and to thereby identify challenges with the current example system. We
used the five MPL to: set parameters to assess the current example system, propose design guidelines
and assess the prototype system. The five principles are summarized below [
To explore our first research question about the current problem faced by learners using 2D nonimmersive VR based learning systems, we conducted a user-study where experienced Computer Science students assess the learning provided by an existing system that teaches bubble sort. 3.1.
We recruited 5 Computer Science students aged in the range of 21-24. Of these, 4 were male and 1 was female. All the participants were full time registered students in University of British Columbia at the time of this study with 4 in Undergrad program and 1 in Graduate program. We chose Computer Science students as we considered having prior knowledge of the subject is an important skill to provide feedback on a learning system teaching that concept. No compensation was provided to the participants. 3.2.
The existing system that we use to assess the challenges is called VisuAlgo[
We carried out a moderated exploratory usability test in 5 different sessions with each session focused on one participant. Each session started with us telling participants about the VisuAlgo system and instructing them to explore the bubble sort module to assess the quality of learning providers. General observations were made like time taken and general behavior while the participants explored the system. After the study, the participants were asked to fill in a survey that gathered data about their experience with the system. The parameters in this survey were set in close relationship with the MPL. Table 1 shows the questions that were asked in this survey (excluding the demographics question).
After the survey we also collected qualitative data from the participant via a semi-structured interview. The questions asked were: • • • • •
In your opinion, are these systems effective? Were you able to easily understand how to navigate and learn via the system? What challenges did you face? How was the overall experience?
Any suggestions to improve the experience? 3.4.
As per the general observation carried out by us, it took 3 minutes 42 seconds for an average participant to complete the study. For a long time, participants looked confused on what to do. They spent a lot of time hovering around the system and made unnecessary clicks before actually carrying out the supposed task. In addition, the problem description on the system word heavy and participants weren’t looking interested and decided to skip through those. Based on the survey response, 80% of people reported to have had previous experience interacting with 2D interactive based learning systems. Table 2 presents the mean reported value for question 2(a) to 2(e) by the 5 participants along with minimum, maximum and variance.
The mean values for all the 5 parameters belong to the lower half of the likert scale (1-5) representing disappointment of user experience with the system. Moreover, all parameters also have a lower variance that suggests a more consistent opinion by the users. The qualitative data was analyzed by creating an affinity diagram (Fig. 1). The process of creating an affinity diagram involves grouping and organizing vast amounts of data into clusters or categories that share common relationships or themes. The green cluster in the figure represents positive feedback while the red represents negative feedback. In nutshell, while the users found such a learning approach to be fun and interesting, they struggled with reading word-heavy texts and felt a need for some kind of assessment or feedback to know whether they learnt.
• C1: Unclear and non-interactive problem description - One of the key challenges observed was that the participants felt the need for an improved problem description that is not word heavy and expresses the motivation of learning. This is associated with the Problem and Activation component of MPL. • C2: No demonstration provided on how to carry out the task or how to learn from the system Participants found demonstration or direction of task to be an important feature of a learning system. Otherwise, it was confusing for them on how to actually use the system to learn. This is associated with the Demonstrations component of MPL. • C3: Missing feedback or learning assessment - After exploring the system, participants felt the need for some kind of assessment that can help them know how much they have learnt or if they learnt. This is associated with the Application and Integration component of MPL.
To provide design solutions to solve the above listed challenges and ensure learning is promoted in relation to MPL, we propose a set of four guidelines (Table 3).
We conducted an evaluation to test our set of design guidelines. To do this, we designed a prototype system on Figma and conducted a user study to assess the quality of that system, again in relation to MPL. 5.1.
To design the prototype system, we followed the steps of design thinking frame- work [
To validate the prototype, we conducted two user studies on the prototype system. The first study was the second half of within-subject study to compare the learning quality assessment by the Computer Science students between the old VisuAlgo system and the new prototype system. While the other study was a standalone user study conducted on a set of non-CS students on the new prototype system, who didn’t have any prior knowledge of bubble sort. This was done to assess whether they were able to learn. bubble sort from the prototype system.
For the within-subject CS user study, we had the same CS students as in the challenge identification user study. (Aged 21-24; 4M, 1F; 4 Undergrad, 1 Graduate). The other set consisting of non-CS students were aged 20-26 with 2 male and 3 female participants. Among them 2 were graduate students in Engineering and Education. While 3 were undergraduate students in Statistics, Psychology and Food sciences. No compensation was provided to the participants. 5.3.
The procedure for both the CS and non-CS participant set was the same. We carried out individual user sessions of moderated exploratory usability tests. The session started with we giving out a general overview of the system and telling users that they have to use the system to learn bubble sort, a basic number sorting algorithm. After the study, the participants were to fill a survey that gathered data about their experience with the system. The parameters in this survey were set in close relationship with the MPL. The questions asked were the same as in the challenges identification user study as listed in Table 1. To the CS students, we asked an additional question to compare their assessment of both the systems (VisuAlgo and new prototype). The question asked them to compare the systems on three parameters: Problem description and motivation, Task demonstration and learning and Feedback & Quiz. They were given 3 options for each parameter: “Better in the previous system”, “Better in the new system” and “Can’t Compare”. 5.4.
The result obtained from the CS learning assessment test and non-CS learning experience tests are shown in Figure 3(a) and 3(b) respectively.
For the additional comparison questions asked to the CS set: all of them responded with a prototype to be a better system for “Problem description and motivation” and “Feedback & Quiz”. While for “Task demonstration and learning”, 4 responded prototype to better and 1 responded VisuAlgo to be better.
MPL acted as a strong foundation of this research. Throughout this research study, we could maintain coherency from challenges to design guidelines to the validation study based on the principles of learning in MPL. Our qualitative data from the challenge identification study helped us identify key challenges with the current non-immersive VR-based learning system. As it can be seen in Figure 3(a), adapting the design guidelines improved the learning assessment of the participants substantially from the VisuAlgo to the prototype system. For all the 5 parameters the mean scores for the prototype system is better than VisuAlgo. Specifically, there is a high margin difference in mean of the “overall” parameter. Similarly, Figure 3(b) shows that the non-CS participants were able to learn about bubblesort through this system. The results from the validation user study gave us key findings thereby validating our design guidelines.
Some of the limitation observed in this study are: • Limited generalizability: This study only focused on one algorithm, bubble sort, which may limit the generalizability of the findings to other algorithms or topics. Further research should investigate the effectiveness of the pro- posed system on a wider range of topics to enhance generalizability. • Non-matching user groups: Both user groups, CS and non-CS, did not exactly match the userpersona that the prototype was based on. The results may have been influenced by individual differences between the participants, such as their prior knowledge, learning styles, and cognitive abilities. • Lack of baseline score: Figure 3(b) does not provide a baseline score to com- pare the non-CS group’s learning experience scores. Without a comparative threshold, it is difficult to assess the significance of those mean scores. Future studies should consider including a baseline score to provide a better understanding of the non-CS participants’ learning experience.
Our first research question aimed to identify the current challenges in 2D non- immersive VR-based learning systems. Through our qualitative analysis in the challenge identification user study, we successfully brought out challenges based on user experience. The second research question was to assess whether the cur- rent system follows the principles of learning. Our quantitative analysis in the challenge identification user study successfully assessed this across five parameters derived from MPL, a popular learning theory. Our third research question sought to provide design implications that promote learning rather than gamification. We proposed design guidelines for non-immersive VRbased learning systems that follow the principles of learning and validated them through a prototype. The results of our usability study for the prototype validate the effectiveness of our design guidelines. 8. References