The Metaversity is a general term describing the emerging extended reality environments universities use for teaching and learning in the metaverse. Combining virtual reality technologies and academic content, the Metaversity generates a unique level of detail on students' data, which contains not only their actions within the virtual learning environment but also their biometrics while engaging in the Metaversity. This major change in scope and volume of data, challenges learning analytics (LA) to design new performance measures to support creating, operating, and managing learning and teaching in the Metaversity. Within these LA challenges, we focus on student evaluation in the Metaversity. Our approach is based on the Theory of Constraints (TOC) thinking process, which defines the system's goal, and derives all further actions accordingly. Therefore, this research-in-progress suggests a four-level evaluation model for measuring student achievements while performing academic tasks in the Metaversity, or afterward (external assessment; traditional assessment within the VR; experience assessment within the VR; aggregated assessment within the VR). We use a Geology course to demonstrate our student evaluation framework.
Virtual reality (VR) technologies have been sporadically used for teaching and learning in Higher
Education Institutes (HEIs) for years. As these pedagogical VR initiatives are mainly considered
isolated islands of innovation [
Lately, interest in the Metaverse and the Metaversity somewhat plummeted due to the emergence of generative AI, especially, Chat-GPT, but mainly because the metaverse is not yet ready for ubiquitous use. Nevertheless, as immersive capabilities mature and become embedded in daily life, HEI will need to adopt the Metaversity as it is expected to become an important instrument for teaching and learning. Therefore, HEI that strategically value educational technologies must prepare for the Metaversity in advance.
The Metaversity combines extended reality technologies and new formats of academic content. It allows for both synchronic and asynchronic learning while generating a new level of data on students, which contains not only their actions within the virtual learning environment 0000-0002-3010-827X (A. Winer); 0000-0001-7991-4646 (N. Geri) © 2023 Copyright for this paper by its authors.
Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR Workshop Proceedings (CEUR-WS.org) (VLE), but also their biometrics while engaging with learning activities within the Metaversity. This major change in scope and volume of data, challenges LA to design new performance measures to support creating, operating, and managing learning and teaching in the Metaversity while taking into consideration ethics and general data governance standards such as GDPR.
Designing appropriate LA that will provide adequate performance measures for instructors and students is essential for building effective learning environments in the Metaversity. Our approach is based on the Theory of Constraints (TOC) thinking process, which defines the system’s goal, and derives all further actions accordingly [7]. The goal of LA is to improve the design of learning environments [8] by connecting LA to pedagogical intent [9]. The goal of the Metaversity learning environment is to augment student learning with virtual experiences. Therefore, LA should focus data collection and analysis on measuring student learning experiences in the Metaversity.
While the Metaversity is expected to increase student engagement, it may have unintended negative consequences on student learning [10], may not be appropriately designed, and might even lead to unintended biometrical leakage of students' data to the platform or software providers. Hence, the purpose of this research-in-progress is to develop a student evaluation framework for measuring student achievements while performing academic tasks in the Metaversity, or afterward. Furthermore, we provide an initial proof-of-concept by demonstrating the applicability of the proposed student evaluation framework to a Geology course that includes a major VR element in the form of an interactive 360o Video and 3D images as part of a VR Geological field trip of the Ramon stream.
The Metaversity is strategically important to the Open University of Israel (OUI). We draw on our experience with the early adoption of Zoom, to better prepare the gradual process of adopting the building blocks of the future Metaversity. Specifically, we take into consideration the essential need to integrate student assessment back into our VLE. This new form of data can be assimilated with the existing LA that is already embedded [11]. The OUI started experimenting with Zoom for online lectures in 2014. When the COVID-19 pandemic outburst in 2020, about a third of our students studied online via Zoom, while the others learned face-to-face in classes all over the country. As a result of our pedagogical and technical experience with Zoom, we were able to smoothly transfer all our instructors and students to remote teaching and learning via Zoom. Nowadays, we have 700 hours of recorded Zoom sessions per day.
Within the context of HEI, Strategic innovation is aimed at creating an enduring (competitive) added value in comparison to other educational institutions [12]. The decision on the assimilation of Zoom back in 2014 was considered a form of strategic innovation at the OUI. Currently over 80% of The OUI’s 50,000 students study from afar while less than 20% prefer traditional face-toface learning in physical classes that are spread throughout Israel. This shift in students' preferences post the COVID-19 pandemic was the main trigger for the Metaversity initiative and its acceptance as a strategic innovation for the OUI in 2022. Additionally, current research shows that it’s getting harder to retain student attention and create engagement [13] [14]. The Metaversity may be one of the main options to significantly improve the student learning experience. During the maturation phases of the Metaversity, we have time to explore new opportunities to identify effective immersive VR elements and embed them in current learning environments, even though they do not yet provide the full Metaversity learning experience. Finally, within a few years, we expect the Metaversity, with its immersive VR devices to become a common learning enabler, much like today's prevalence of laptops and mobile phones. When the Metaversity becomes widespread, a few years from now, we shall be ready.
In preparation for the future of the Metaversity, we identified three gaps: the pedagogical
design of VR learning experiences and environments [
The student evaluation framework is designed to compare parallel ways of implementing the assessment of VR learning. The four levels of the student evaluation model serve as a gradual implementation process, starting from the first simple type, and advancing to more complex assessment types. Choosing the appropriate type of assessment is dependent on the pedagogical intentions of the course instructor as well as on the availability of resources and technological maturity.
The four levels of the student evaluation framework in the Metaversity are: 1. External Assessment in the VLE post the VR learning experience - The primary goal of this stage is to evaluate the knowledge and understanding that learners have acquired during their VR learning experience. The quiz scores can serve as an indicator of the VR content's quality and effectiveness. From the perspective of LA data collection, this stage relies on existing quiz formats that are already present in the VLE platform. The identity of students and their privacy are kept without any additional effort. 2. Traditional Assessment within the VR Learning Experience - By replicating traditional assessment formats in situ within VR, the goal of the second level is to provide students with immediate feedback on their learning experiences. This stage also strives to enrich the learning process by providing real-time guidance, reinforcing comprehension, and fostering engagement, thus contributing to enhanced knowledge retention and skill development. Anonymous Data Collection can balance the need for real-time assessment feedback with data governance standards. Personally Identifiable Information (PII) can be minimized by using pseudonyms or unique identifiers. This approach helps protect students' privacy while still providing valuable insights. Students' scores can be extracted from the VR environment and uploaded to the VLE using a conversion table.
level is to bridge the gap between theoretical knowledge and practical application, ensuring learners can apply their skills in authentic contexts and demonstrate real-life competencies. This level is particularly relevant for simulating real-life scenarios, including potentially hazardous or costly situations. The game-like environment can capture learning data while users progressively demonstrate their proficiency through bite-sized tasks. Basic engagement can use the level two data governance mechanism. However, in cases where biometrical identity can be deducted from the learner's experiences, an additional layer of privacy is needed. This level requires students to sign a Consent-Based Data Capture form, before engaging with the VR experiences. Instructors should clearly communicate the purpose of data capture and how it will be used to enhance student learning. The data governance of the VR learning environment should align the level of data collection with students' preferences. When students opt out, only non-biometric data can be collected. Collecting data solely for research purposes without any relevance to the actual learning of students is to be minimized or avoided altogether.
level is to analyze the effectiveness of the VR learning environment as a whole. It assesses how well the instructional design aligns with the collective achievements of all students. Instructional designers can refine their approaches and continually improve the learning experience for all learners. Therefore, this level is not directed at individual learners but rather at optimizing the VR learning experiences for the entire course population. Aggregated and anonymized data that is not directly associated with individual students inflicts minimal restrictions from a data governance perspective. Analyzing pedagogical intent and achievements collectively by grouping data cohorts, courses, or other non-identifiable categories prevents any potential breaches of privacy while ensuring compliance with regulations.
We define four types of assessment for the same task, which is identifying several geological phenomena. Usually, this scenario occurs on-site in one of Israel's most exquisite landscapes, the Ramon stream. It is the world's largest "erosion cirque", located in the southern area of Israel that is called the Negev desert. Over the years, we encountered accessibility challenges that called for novel ways of teaching and learning about these unique geographical attributes that serve as an integral part of the learning goals. Our student evaluation model contains four assessment types. The comparison among these four different assessment types is aimed at choosing the most appropriate one for this task. As we are currently in the developmental stages of this learning VR environment, this paper presents the four types of assessment but does not analyze the results of the comparison. These are expected to be available at a later stage.
The first type of VR learning assessment occurs outside of the VR experience. This means that while the actual learning is experienced within the VR environment, students are required to log into their VLE for the assessment and fill out a knowledge quiz. For this task, we used several high-resolution pictures of 3D objects that were taken from several key locations within the Ramon stream. Students were asked to recognize the highlighted phenomena, such as those marked on the picture in Figure 1.
Photogrammetry models (digital twins) used ‘Reality Capture’, for building realistic models based on a collection of images. ‘Sketch Up’ allows 3D Modelling to complete the photogrammetry model of the current state with suggested reconstructions of lost parts. Three-dimensional editing and addition of info-graphics and visual information using ‘Blender’. Outside the VR environment students are asked to complete a quiz in the 'Moodle' VLE.
Figure 2 shows an example of embedded student assessment and feedback. Students are asked to participate in a 360o interactive VR geological field trip. First, students hear the instructor’s micro lecture, which refers to the geological unique landscape. Second, students ‘roam’ the virtual stream, and encounter questions. Pressing the question mark, a multiple-choice question is virtually presented, and the student marks an answer. Finally, the student receives immediate feedback. This form of assessment may be used for self-practice or as part of the graded formative assessment. Both cases require the identification of students and the collection of their answers inside the VR environment. These achievements are then transferred back to the VLE by using common standards such as xAPI [16] or SCORM [17]. '3D Vista' is used to add interactive questions and information layers to the 3D models. Although 3DVista has a native LMS integration, at this time security and privacy concerns have not been mitigated. Meanwhile, this platform is used for formative assessment and self-practice.
VR provides learners with ample opportunities to interact with virtual objects and tasks that need to be completed within the VR learning experiences. Applications demonstrating these principles can simulate real-life scenarios and replicate dangerous environments or costly tasks. Assessment of mastering these experiences can serve as reliable evidence for real-life competencies. This type of assessment resembles a game-like environment that is constantly capturing data while asking users to gradually demonstrate progress through a set of bite-size tasks [18]. In educational settings, badges can be used to visually depict task completion and learning progress. Figure 3 depicts an example of experience assessment along the Ramon stream. As with traditional assessment within the VR, each student should be identified, the performance data should be collected and transferred back to the VLE.
We use ‘Unity’ to combine the elements from the first level into interactive experiences, this environment acts as a game-creation program. Unity enables the animation of 3D elements, and their set reactions when touched. The next step is to export items from ‘Unity’ and/or ‘Blender’ and, to place them with the whole model into ‘Spatial’. As far as we have thus far been able to assess Spatial is unusual in allowing high-level VR interactions with multiple participants. We added to the Spatial environment interactive VR spaces and Social VR meeting points.
This type of assessment allows instructional designers to analyze their pedagogical intent in comparison to the actual achievements of their students within the VR learning environment. This type of assessment is not directed at the individual learner but rather at all students as a whole. One example of this type of assessment can be a heat map that captures the eye gaze [19] of participants and provides scaled visual feedback on points of interest. Naturally, pedagogical instructors should ask learners to investigate these points of interest and compare these requirements to actual learning using the heatmap. Alternatively, capturing a student's gaze on a specific point of interest or a small-scale area can lead to an activation of a trigger of completion. Figure 4 depicts an example of comparing users' heat map, to the expected points of interest.
'3D Vista' supports the collection of eye-tracking inputs. Heatmaps monitor and track the traffic inside the virtual tour. Places that attracted most of the students' attention were colored in red.
HEI that seek to prepare themselves for the future Metaversity as strategic innovation need to
take into consideration three gaps: pedagogical design [
The new forms of data and visual analytics that were extracted from wearable devices, were the main catalysts for the formation of the LA community back in 2011 [20]. We expect the Metaversity to bring to HEI the same scale of innovation in the near future. This can be another dramatic leap for LA with the scope and richness of learning data that will be available. However, ethical, and specifically privacy considerations should be built to match the dark side of the Metaversity [10]. This need is intensified given that these new types of big data resource, can lead to a high level of fidelity that matches the common biometrical identity [21]. The four types of assessment we presented mitigated these concerns by limiting the collection, measurement and LA specifically to learning goals and to match the specific outlined pedagogical design.
We would like to thank Beni Zaks, Head of Learning Experience at the Open University of Israel, who as a gifted VR and 3D artist, developed the VR environments and all the 3D objects. We are grateful to the course instructors: Dr. Yael Levenson and Dr. Yoni Israeli for sharing their content expertise and for piloting the new Metaverse and VR environment with their students.