Virtual reality (VR) technologies have been sporadically used for teaching and learning in Higher Education Institutes (HEIs) for years. As these pedagogical VR initiatives are mainly considered isolated islands of innovation [1], they are sparingly supported by dedicated learning analytics (LA). Following the COVID-19 pandemic, the growing general interest in immersive VR [2] surged with the rise of the Metaverse. The application of innovative technologies for educational purposes usually lags behind business-related applications. However, shortly after the emergence of the Metaverse, the term 'Metaversity' was coined both as a commercial educational course platform [3], and as a general term describing Metaverse for HEI [4] [5]. The Metaversity even drew the attention of Forbes Business magazine, with a title asking: "Will 2023 be the year of the Metaversity?" [6].
Lately, interest in the Metaverse and the Metaversity somewhat plummeted due to the emergence of generative AI, especially, Chat-GPT, but mainly because the metaverse is not yet ready for ubiquitous use. Nevertheless, as immersive capabilities mature and become embedded in daily life, HEI will need to adopt the Metaversity as it is expected to become an important instrument for teaching and learning. Therefore, HEI that strategically value educational technologies must prepare for the Metaversity in advance.
The Metaversity combines extended reality technologies and new formats of academic content. It allows for both synchronic and asynchronic learning while generating a new level of data on students, which contains not only their actions within the virtual learning environment (VLE), but also their biometrics while engaging with learning activities within the Metaversity. This major change in scope and volume of data, challenges LA to design new performance measures to support creating, operating, and managing learning and teaching in the Metaversity while taking into consideration ethics and general data governance standards such as GDPR.
Designing appropriate LA that will provide adequate performance measures for instructors and students is essential for building effective learning environments in the Metaversity. Our approach is based on the Theory of Constraints (TOC) thinking process, which defines the system's goal, and derives all further actions accordingly [7]. The goal of LA is to improve the design of learning environments [8] by connecting LA to pedagogical intent [9]. The goal of the Metaversity learning environment is to augment student learning with virtual experiences. Therefore, LA should focus data collection and analysis on measuring student learning experiences in the Metaversity.
While the Metaversity is expected to increase student engagement, it may have unintended negative consequences on student learning [10], may not be appropriately designed, and might even lead to unintended biometrical leakage of students' data to the platform or software providers. Hence, the purpose of this research-in-progress is to develop a student evaluation framework for measuring student achievements while performing academic tasks in the Metaversity, or afterward. Furthermore, we provide an initial proof-of-concept by demonstrating the applicability of the proposed student evaluation framework to a Geology course that includes a major VR element in the form of an interactive 360 o Video and 3D images as part of a VR Geological field trip of the Ramon stream.
The Metaversity is strategically important to the Open University of Israel (OUI). We draw on our experience with the early adoption of Zoom, to better prepare the gradual process of adopting the building blocks of the future Metaversity. Specifically, we take into consideration the essential need to integrate student assessment back into our VLE. This new form of data can be assimilated with the existing LA that is already embedded [11]. The OUI started experimenting with Zoom for online lectures in 2014. When the COVID-19 pandemic outburst in 2020, about a third of our students studied online via Zoom, while the others learned face-to-face in classes all over the country. As a result of our pedagogical and technical experience with Zoom, we were able to smoothly transfer all our instructors and students to remote teaching and learning via Zoom. Nowadays, we have 700 hours of recorded Zoom sessions per day.
Within the context of HEI, Strategic innovation is aimed at creating an enduring (competitive) added value in comparison to other educational institutions [12]. The decision on the assimilation of Zoom back in 2014 was considered a form of strategic innovation at the OUI. Currently over 80% of The OUI's 50,000 students study from afar while less than 20% prefer traditional face-toface learning in physical classes that are spread throughout Israel. This shift in students' preferences post the COVID-19 pandemic was the main trigger for the Metaversity initiative and its acceptance as a strategic innovation for the OUI in 2022. Additionally, current research shows that it's getting harder to retain student attention and create engagement [13] [14]. The Metaversity may be one of the main options to significantly improve the student learning experience. During the maturation phases of the Metaversity, we have time to explore new opportunities to identify effective immersive VR elements and embed them in current learning environments, even though they do not yet provide the full Metaversity learning experience. Finally, within a few years, we expect the Metaversity, with its immersive VR devices to become a common learning enabler, much like today's prevalence of laptops and mobile phones. When the Metaversity becomes widespread, a few years from now, we shall be ready.
In preparation for the future of the Metaversity, we identified three gaps: the pedagogical design of VR learning experiences and environments [2], teaching in the Metaverse [15], and ethics and technical aspects of collection, extraction, and integration of identified and personalized VR learning data into the VLE [10]. This research in progress contributes to bridging the first gap by developing a student evaluation model that suggests four types of student evaluation in the Metaversity.
The student evaluation framework is designed to compare parallel ways of implementing the assessment of VR learning. The four levels of the student evaluation model serve as a gradual implementation process, starting from the first simple type, and advancing to more complex assessment types. Choosing the appropriate type of assessment is dependent on the pedagogical intentions of the course instructor as well as on the availability of resources and technological maturity.
The four levels of the student evaluation framework in the Metaversity are: Instructional designers can refine their approaches and continually improve the learning experience for all learners. Therefore, this level is not directed at individual learners but rather at optimizing the VR learning experiences for the entire course population. Aggregated and anonymized data that is not directly associated with individual students inflicts minimal restrictions from a data governance perspective. Analyzing pedagogical intent and achievements collectively by grouping data cohorts, courses, or other non-identifiable categories prevents any potential breaches of privacy while ensuring compliance with regulations.
We define four types of assessment for the same task, which is identifying several geological phenomena. Usually, this scenario occurs on-site in one of Israel's most exquisite landscapes, the Ramon stream. It is the world's largest "erosion cirque", located in the southern area of Israel that is called the Negev desert. Over the years, we encountered accessibility challenges that called for novel ways of teaching and learning about these unique geographical attributes that serve as an integral part of the learning goals. Our student evaluation model contains four assessment types.
The comparison among these four different assessment types is aimed at choosing the most appropriate one for this task. As we are currently in the developmental stages of this learning VR environment, this paper presents the four types of assessment but does not analyze the results of the comparison. These are expected to be available at a later stage.
The first type of VR learning assessment occurs outside of the VR experience. This means that while the actual learning is experienced within the VR environment, students are required to log into their VLE for the assessment and fill out a knowledge quiz. For this task, we used several high-resolution pictures of 3D objects that were taken from several key locations within the Ramon stream. Students were asked to recognize the highlighted phenomena, such as those marked on the picture in Figure 1. Photogrammetry models (digital twins) used 'Reality Capture', for building realistic models based on a collection of images. 'Sketch Up' allows 3D Modelling to complete the photogrammetry model of the current state with suggested reconstructions of lost parts. Three-dimensional editing and addition of info-graphics and visual information using 'Blender'. Outside the VR environment students are asked to complete a quiz in the 'Moodle' VLE.
Figure 2 shows an example of embedded student assessment and feedback. Students are asked to participate in a 360 o interactive VR geological field trip. First, students hear the instructor's micro lecture, which refers to the geological unique landscape. Second, students 'roam' the virtual stream, and encounter questions. Pressing the question mark, a multiple-choice question is virtually presented, and the student marks an answer. Finally, the student receives immediate feedback. This form of assessment may be used for self-practice or as part of the graded formative assessment. Both cases require the identification of students and the collection of their answers inside the VR environment. These achievements are then transferred back to the VLE by using common standards such as xAPI [16] or SCORM [17]. '3D Vista' is used to add interactive questions and information layers to the 3D models. Although 3DVista has a native LMS integration, at this time security and privacy concerns have not been mitigated. Meanwhile, this platform is used for formative assessment and self-practice.
VR provides learners with ample opportunities to interact with virtual objects and tasks that need to be completed within the VR learning experiences. Applications demonstrating these principles can simulate real-life scenarios and replicate dangerous environments or costly tasks. Assessment of mastering these experiences can serve as reliable evidence for real-life competencies. This type of assessment resembles a game-like environment that is constantly capturing data while asking users to gradually demonstrate progress through a set of bite-size tasks [18]. In educational settings, badges can be used to visually depict task completion and learning progress. Figure 3 depicts an example of experience assessment along the Ramon stream. As with traditional assessment within the VR, each student should be identified, the performance data should be collected and transferred back to the VLE. We use 'Unity' to combine the elements from the first level into interactive experiences, this environment acts as a game-creation program. Unity enables the animation of 3D elements, and their set reactions when touched. The next step is to export items from 'Unity' and/or 'Blender' and, to place them with the whole model into 'Spatial'. As far as we have thus far been able to assess Spatial is unusual in allowing high-level VR interactions with multiple participants. We added to the Spatial environment interactive VR spaces and Social VR meeting points.
This type of assessment allows instructional designers to analyze their pedagogical intent in comparison to the actual achievements of their students within the VR learning environment. This type of assessment is not directed at the individual learner but rather at all students as a whole. One example of this type of assessment can be a heat map that captures the eye gaze [19] of participants and provides scaled visual feedback on points of interest. Naturally, pedagogical instructors should ask learners to investigate these points of interest and compare these requirements to actual learning using the heatmap. Alternatively, capturing a student's gaze on a specific point of interest or a small-scale area can lead to an activation of a trigger of completion. Figure 4 depicts an example of comparing users' heat map, to the expected points of interest.
HEI that seek to prepare themselves for the future Metaversity as strategic innovation need to take into consideration three gaps: pedagogical design [2], teaching in the Metaverse [15], and ethics and technical consideration of data governance [10]. This paper presents four types of assessment as preparation for the first gap. While the Metaversity provides diverse new types of data that may be transformed by LA to valuable performance measures, we suggest adopting a gradual approach. This means that LA from VR should focus initially on assessment of understandable performance measures and limit the data collection solely to these initial learning goals.
The new forms of data and visual analytics that were extracted from wearable devices, were the main catalysts for the formation of the LA community back in 2011 [20]. We expect the Metaversity to bring to HEI the same scale of innovation in the near future. This can be another dramatic leap for LA with the scope and richness of learning data that will be available. However, ethical, and specifically privacy considerations should be built to match the dark side of the Metaversity [10]. This need is intensified given that these new types of big data resource, can lead to a high level of fidelity that matches the common biometrical identity [21]. The four types of assessment we presented mitigated these concerns by limiting the collection, measurement and LA specifically to learning goals and to match the specific outlined pedagogical design.
We would like to thank Beni Zaks, Head of Learning Experience at the Open University of Israel, who as a gifted VR and 3D artist, developed the VR environments and all the 3D objects. We are grateful to the course instructors: Dr. Yael Levenson and Dr. Yoni Israeli for sharing their content expertise and for piloting the new Metaverse and VR environment with their students.