=Paper=
{{Paper
|id=Vol-3667/LAVRLAK24_paper_4
|storemode=property
|title=Towards Learning Analytics for Student Evaluation in the Metaversity
|pdfUrl=https://ceur-ws.org/Vol-3667/LAVRLAK24_paper_4.pdf
|volume=Vol-3667
|authors=Amir Winer,Nitza Geri
|dblpUrl=https://dblp.org/rec/conf/lak/WinerG24
}}
==Towards Learning Analytics for Student Evaluation in the Metaversity==
https://ceur-ws.org/Vol-3667/LAVRLAK24_paper_4.pdf
Towards Learning Analytics for Student Evaluation in the
Metaversity
Amir Winer1,* , Nitza Geri1,*
1 The Open University of Israel, Raanana, Israel
Abstract
The Metaversity is a general term describing the emerging extended reality environments universities
use for teaching and learning in the metaverse. Combining virtual reality technologies and academic
content, the Metaversity generates a unique level of detail on students’ data, which contains not only
their actions within the virtual learning environment but also their biometrics while engaging in the
Metaversity. This major change in scope and volume of data, challenges learning analytics (LA) to design
new performance measures to support creating, operating, and managing learning and teaching in the
Metaversity. Within these LA challenges, we focus on student evaluation in the Metaversity. Our
approach is based on the Theory of Constraints (TOC) thinking process, which defines the system’s goal,
and derives all further actions accordingly. Therefore, this research-in-progress suggests a four-level
evaluation model for measuring student achievements while performing academic tasks in the
Metaversity, or afterward (external assessment; traditional assessment within the VR; experience
assessment within the VR; aggregated assessment within the VR). We use a Geology course to
demonstrate our student evaluation framework.
Keywords
Extended reality, Metaversity, performance measurement, learning analytics, student evaluation,
Theory of Constraints (TOC), higher education 1
1. Introduction
Virtual reality (VR) technologies have been sporadically used for teaching and learning in Higher
Education Institutes (HEIs) for years. As these pedagogical VR initiatives are mainly considered
isolated islands of innovation [1], they are sparingly supported by dedicated learning analytics
(LA). Following the COVID-19 pandemic, the growing general interest in immersive VR [2] surged
with the rise of the Metaverse. The application of innovative technologies for educational
purposes usually lags behind business-related applications. However, shortly after the
emergence of the Metaverse, the term ‘Metaversity’ was coined both as a commercial educational
course platform [3], and as a general term describing Metaverse for HEI [4] [5]. The Metaversity
even drew the attention of Forbes Business magazine, with a title asking: “Will 2023 be the year
of the Metaversity?” [6].
Lately, interest in the Metaverse and the Metaversity somewhat plummeted due to the
emergence of generative AI, especially, Chat-GPT, but mainly because the metaverse is not yet
ready for ubiquitous use. Nevertheless, as immersive capabilities mature and become embedded
in daily life, HEI will need to adopt the Metaversity as it is expected to become an important
instrument for teaching and learning. Therefore, HEI that strategically value educational
technologies must prepare for the Metaversity in advance.
The Metaversity combines extended reality technologies and new formats of academic
content. It allows for both synchronic and asynchronic learning while generating a new level of
data on students, which contains not only their actions within the virtual learning environment
Joint Proceedings of LAK 2024 Workshops, co-located with 14th International Conference on Learning Analytics and
Knowledge (LAK 2024), Kyoto, Japan, March 18-22, 2024.
∗ Corresponding author
amirwi@openu.ac.il (A. Winer); nitzage@openu.ac.il (N. Geri)
0000-0002-3010-827X (A. Winer); 0000-0001-7991-4646 (N. Geri)
© 2023 Copyright for this paper by its authors.
Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR Workshop Proceedings (CEUR-WS.org)
CEUR
ceur-ws.org
Workshop ISSN 1613-0073
Proceedings
(VLE), but also their biometrics while engaging with learning activities within the Metaversity.
This major change in scope and volume of data, challenges LA to design new performance
measures to support creating, operating, and managing learning and teaching in the Metaversity
while taking into consideration ethics and general data governance standards such as GDPR.
Designing appropriate LA that will provide adequate performance measures for instructors
and students is essential for building effective learning environments in the Metaversity. Our
approach is based on the Theory of Constraints (TOC) thinking process, which defines the
system’s goal, and derives all further actions accordingly [7]. The goal of LA is to improve the
design of learning environments [8] by connecting LA to pedagogical intent [9]. The goal of the
Metaversity learning environment is to augment student learning with virtual experiences.
Therefore, LA should focus data collection and analysis on measuring student learning
experiences in the Metaversity.
While the Metaversity is expected to increase student engagement, it may have unintended
negative consequences on student learning [10], may not be appropriately designed, and might
even lead to unintended biometrical leakage of students' data to the platform or software
providers. Hence, the purpose of this research-in-progress is to develop a student evaluation
framework for measuring student achievements while performing academic tasks in the
Metaversity, or afterward. Furthermore, we provide an initial proof-of-concept by demonstrating
the applicability of the proposed student evaluation framework to a Geology course that includes
a major VR element in the form of an interactive 360o Video and 3D images as part of a VR
Geological field trip of the Ramon stream.
2. The Metaversity at the Open University of Israel
The Metaversity is strategically important to the Open University of Israel (OUI). We draw on our
experience with the early adoption of Zoom, to better prepare the gradual process of adopting
the building blocks of the future Metaversity. Specifically, we take into consideration the essential
need to integrate student assessment back into our VLE. This new form of data can be assimilated
with the existing LA that is already embedded [11]. The OUI started experimenting with Zoom for
online lectures in 2014. When the COVID-19 pandemic outburst in 2020, about a third of our
students studied online via Zoom, while the others learned face-to-face in classes all over the
country. As a result of our pedagogical and technical experience with Zoom, we were able to
smoothly transfer all our instructors and students to remote teaching and learning via Zoom.
Nowadays, we have 700 hours of recorded Zoom sessions per day.
Within the context of HEI, Strategic innovation is aimed at creating an enduring (competitive)
added value in comparison to other educational institutions [12]. The decision on the assimilation
of Zoom back in 2014 was considered a form of strategic innovation at the OUI. Currently over
80% of The OUI’s 50,000 students study from afar while less than 20% prefer traditional face-to-
face learning in physical classes that are spread throughout Israel. This shift in students'
preferences post the COVID-19 pandemic was the main trigger for the Metaversity initiative and
its acceptance as a strategic innovation for the OUI in 2022. Additionally, current research shows
that it’s getting harder to retain student attention and create engagement [13] [14]. The
Metaversity may be one of the main options to significantly improve the student learning
experience. During the maturation phases of the Metaversity, we have time to explore new
opportunities to identify effective immersive VR elements and embed them in current learning
environments, even though they do not yet provide the full Metaversity learning experience.
Finally, within a few years, we expect the Metaversity, with its immersive VR devices to become
a common learning enabler, much like today's prevalence of laptops and mobile phones. When
the Metaversity becomes widespread, a few years from now, we shall be ready.
In preparation for the future of the Metaversity, we identified three gaps: the pedagogical
design of VR learning experiences and environments [2], teaching in the Metaverse [15], and
ethics and technical aspects of collection, extraction, and integration of identified and
personalized VR learning data into the VLE [10]. This research in progress contributes to bridging
the first gap by developing a student evaluation model that suggests four types of student
evaluation in the Metaversity.
3. The Student Evaluation Framework
The student evaluation framework is designed to compare parallel ways of implementing the
assessment of VR learning. The four levels of the student evaluation model serve as a gradual
implementation process, starting from the first simple type, and advancing to more complex
assessment types. Choosing the appropriate type of assessment is dependent on the pedagogical
intentions of the course instructor as well as on the availability of resources and technological
maturity.
The four levels of the student evaluation framework in the Metaversity are:
1. External Assessment in the VLE post the VR learning experience - The primary goal
of this stage is to evaluate the knowledge and understanding that learners have acquired
during their VR learning experience. The quiz scores can serve as an indicator of the VR
content's quality and effectiveness. From the perspective of LA data collection, this stage relies
on existing quiz formats that are already present in the VLE platform. The identity of students
and their privacy are kept without any additional effort.
2. Traditional Assessment within the VR Learning Experience - By replicating
traditional assessment formats in situ within VR, the goal of the second level is to provide
students with immediate feedback on their learning experiences. This stage also strives to
enrich the learning process by providing real-time guidance, reinforcing comprehension, and
fostering engagement, thus contributing to enhanced knowledge retention and skill
development. Anonymous Data Collection can balance the need for real-time assessment
feedback with data governance standards. Personally Identifiable Information (PII) can be
minimized by using pseudonyms or unique identifiers. This approach helps protect students'
privacy while still providing valuable insights. Students' scores can be extracted from the VR
environment and uploaded to the VLE using a conversion table.
3. Experience assessment within the VR learning experience – The primary goal of this
level is to bridge the gap between theoretical knowledge and practical application, ensuring
learners can apply their skills in authentic contexts and demonstrate real-life competencies.
This level is particularly relevant for simulating real-life scenarios, including potentially
hazardous or costly situations. The game-like environment can capture learning data while
users progressively demonstrate their proficiency through bite-sized tasks. Basic engagement
can use the level two data governance mechanism. However, in cases where biometrical
identity can be deducted from the learner's experiences, an additional layer of privacy is
needed. This level requires students to sign a Consent-Based Data Capture form, before
engaging with the VR experiences. Instructors should clearly communicate the purpose of data
capture and how it will be used to enhance student learning. The data governance of the VR
learning environment should align the level of data collection with students' preferences.
When students opt out, only non-biometric data can be collected. Collecting data solely for
research purposes without any relevance to the actual learning of students is to be minimized
or avoided altogether.
4. Aggregated Assessment within the VR Learning Experience - the goal of the fourth
level is to analyze the effectiveness of the VR learning environment as a whole. It assesses how
well the instructional design aligns with the collective achievements of all students.
Instructional designers can refine their approaches and continually improve the learning
experience for all learners. Therefore, this level is not directed at individual learners but rather
at optimizing the VR learning experiences for the entire course population. Aggregated and
anonymized data that is not directly associated with individual students inflicts minimal
restrictions from a data governance perspective. Analyzing pedagogical intent and
achievements collectively by grouping data cohorts, courses, or other non-identifiable
categories prevents any potential breaches of privacy while ensuring compliance with
regulations.
4. Implementing the Evaluation Framework in a Geology Course
We define four types of assessment for the same task, which is identifying several geological
phenomena. Usually, this scenario occurs on-site in one of Israel's most exquisite landscapes, the
Ramon stream. It is the world's largest "erosion cirque", located in the southern area of Israel that
is called the Negev desert. Over the years, we encountered accessibility challenges that called for
novel ways of teaching and learning about these unique geographical attributes that serve as an
integral part of the learning goals. Our student evaluation model contains four assessment types.
The comparison among these four different assessment types is aimed at choosing the most
appropriate one for this task. As we are currently in the developmental stages of this learning VR
environment, this paper presents the four types of assessment but does not analyze the results of
the comparison. These are expected to be available at a later stage.
4.1. External Assessment in the VLE post the VR learning experience
The first type of VR learning assessment occurs outside of the VR experience. This means that
while the actual learning is experienced within the VR environment, students are required to log
into their VLE for the assessment and fill out a knowledge quiz. For this task, we used several
high-resolution pictures of 3D objects that were taken from several key locations within the
Ramon stream. Students were asked to recognize the highlighted phenomena, such as those
marked on the picture in Figure 1.
Figure 1: A 3D Object viewed by a 360o viewer outside VR with multiple-choice assessment in
the VLE
Photogrammetry models (digital twins) used ‘Reality Capture’, for building realistic models
based on a collection of images. ‘Sketch Up’ allows 3D Modelling to complete the photogrammetry
model of the current state with suggested reconstructions of lost parts. Three-dimensional
editing and addition of info-graphics and visual information using ‘Blender’. Outside the VR
environment students are asked to complete a quiz in the 'Moodle' VLE.
4.2. Traditional Assessment within the VR learning experience
Figure 2 shows an example of embedded student assessment and feedback. Students are asked
to participate in a 360o interactive VR geological field trip. First, students hear the instructor’s
micro lecture, which refers to the geological unique landscape. Second, students ‘roam’ the virtual
stream, and encounter questions. Pressing the question mark, a multiple-choice question is
virtually presented, and the student marks an answer. Finally, the student receives immediate
feedback. This form of assessment may be used for self-practice or as part of the graded formative
assessment. Both cases require the identification of students and the collection of their answers
inside the VR environment. These achievements are then transferred back to the VLE by using
common standards such as xAPI [16] or SCORM [17].
Figure 2: Student assessment and feedback within the VR learning experience
'3D Vista' is used to add interactive questions and information layers to the 3D models.
Although 3DVista has a native LMS integration, at this time security and privacy concerns have
not been mitigated. Meanwhile, this platform is used for formative assessment and self-practice.
4.3. Experience assessment within the VR learning experience
VR provides learners with ample opportunities to interact with virtual objects and tasks that need
to be completed within the VR learning experiences. Applications demonstrating these principles
can simulate real-life scenarios and replicate dangerous environments or costly tasks.
Assessment of mastering these experiences can serve as reliable evidence for real-life
competencies. This type of assessment resembles a game-like environment that is constantly
capturing data while asking users to gradually demonstrate progress through a set of bite-size
tasks [18]. In educational settings, badges can be used to visually depict task completion and
learning progress. Figure 3 depicts an example of experience assessment along the Ramon
stream. As with traditional assessment within the VR, each student should be identified, the
performance data should be collected and transferred back to the VLE.
Figure 3: Experience assessment within the VR learning experience
We use ‘Unity’ to combine the elements from the first level into interactive experiences, this
environment acts as a game-creation program. Unity enables the animation of 3D elements, and
their set reactions when touched. The next step is to export items from ‘Unity’ and/or ‘Blender’
and, to place them with the whole model into ‘Spatial’. As far as we have thus far been able to
assess Spatial is unusual in allowing high-level VR interactions with multiple participants. We
added to the Spatial environment interactive VR spaces and Social VR meeting points.
4.4. Aggregated Assessment within the VR learning experience
This type of assessment allows instructional designers to analyze their pedagogical intent in
comparison to the actual achievements of their students within the VR learning environment.
This type of assessment is not directed at the individual learner but rather at all students as a
whole. One example of this type of assessment can be a heat map that captures the eye gaze [19]
of participants and provides scaled visual feedback on points of interest. Naturally, pedagogical
instructors should ask learners to investigate these points of interest and compare these
requirements to actual learning using the heatmap. Alternatively, capturing a student's gaze on a
specific point of interest or a small-scale area can lead to an activation of a trigger of completion.
Figure 4 depicts an example of comparing users' heat map, to the expected points of interest.
Figure 4: Aggregated Student eye gazes heat map within a VR experience
'3D Vista' supports the collection of eye-tracking inputs. Heatmaps monitor and track the
traffic inside the virtual tour. Places that attracted most of the students' attention were colored
in red.
5. Conclusion and future work
HEI that seek to prepare themselves for the future Metaversity as strategic innovation need to
take into consideration three gaps: pedagogical design [2], teaching in the Metaverse [15], and
ethics and technical consideration of data governance [10]. This paper presents four types of
assessment as preparation for the first gap. While the Metaversity provides diverse new types of
data that may be transformed by LA to valuable performance measures, we suggest adopting a
gradual approach. This means that LA from VR should focus initially on assessment of
understandable performance measures and limit the data collection solely to these initial
learning goals.
The new forms of data and visual analytics that were extracted from wearable devices, were
the main catalysts for the formation of the LA community back in 2011 [20]. We expect the
Metaversity to bring to HEI the same scale of innovation in the near future. This can be another
dramatic leap for LA with the scope and richness of learning data that will be available. However,
ethical, and specifically privacy considerations should be built to match the dark side of the
Metaversity [10]. This need is intensified given that these new types of big data resource, can lead
to a high level of fidelity that matches the common biometrical identity [21]. The four types of
assessment we presented mitigated these concerns by limiting the collection, measurement and
LA specifically to learning goals and to match the specific outlined pedagogical design.
Acknowledgments
We would like to thank Beni Zaks, Head of Learning Experience at the Open University of Israel,
who as a gifted VR and 3D artist, developed the VR environments and all the 3D objects. We are
grateful to the course instructors: Dr. Yael Levenson and Dr. Yoni Israeli for sharing their
content expertise and for piloting the new Metaverse and VR environment with their students.
References
[1] Avidov-Ungar, O., Shamir-Inbal, T., & Blau, I. (2022). Typology of digital leadership roles
tasked with integrating new technologies into teaching: Insights from metaphor analysis.
Journal of Research on Technology in Education, 54(1), 92-107.
https://doi.org/10.1080/15391523.2020.1809035
[2] Radianti, J., Majchrzak, T. A., Fromm, J., & Wohlgenannt, I. (2020). A systematic review of
immersive virtual reality applications for higher education: Design elements, lessons
learned, and research agenda. Computers & Education, 147, 103778.
https://doi.org/10.1016/j.compedu.2019.103778
[3] https://themetaversity.app/#home
[4] Kshetri, N. (2023) : Metaverse in higher education institutions: Drivers and effects, 32nd
European Conference of the International Telecommunications Society (ITS): "Realising the
digital decade in the European Union – Easier said than done?", Madrid, Spain, 19th - 20th
June 2023, International Telecommunications Society (ITS), Calgary.
http://hdl.handle.net/10419/277991
[5] Wang, M., Yu, H., Bell Z., & Chu, X. (2022). Constructing an edu-metaverse ecosystem: A new
and innovative framework. IEEE Transactions on Learning Technologies, 15(6), 685-696.
https://doi.org/10.1109/TLT.2022.3210828
[6] Follows, T. (2023). Will 2023 be the year of the Metaversity? Forbes.
https://www.forbes.com/sites/traceyfollows/2023/03/13/will-2023-be-the-year-of-the-
metaversity/?sh=6e2c1a932f56
[7] Goldratt, E. M., & Cox, J. (1986). The goal: A process of ongoing improvement. Croton-on-
Hudson, NY: North River Press.
[8] Winer, A., & Geri, N. (2019). Learning analytics performance improvement design (LAPID) in
higher education: Framework and concerns. Online Journal of Applied Knowledge
Management, 7(2), 41-55. https://doi.org/10.36965/OJAKM.2019.7(2)41-55
[9] Lockyer, L., Heathcote, E., & Dawson, S. (2013). Informing pedagogical action: Aligning
learning analytics with learning design. American Behavioral Scientist, 57(10), 1439-1459.
https://doi.org/10.1177/0002764213479367
[10] Carter, M., & Egliston, B. (2023). What are the risks of virtual reality data? Learning analytics,
algorithmic bias and a fantasy of perfect data. New media & society, 25(3), 485-504.
https://doi.org/10.1177/14614448211012794
[11] Winer, A., & Geri, N. (2023). Connecting pedagogical design to intervention: A large scale
implementation of embedded learning analytics. Companion Proceedings of the 13th
International Conference on Learning Analytics and Knowledge (LAK23). March 13-17,
2023. University of Texas at Arlington, Arlington, Texas, USA (pp. 49-52).
https://www.solaresearch.org/core/lak23-companion-proceedings/
[12] Kallenberg, T. (2007). Strategic innovation in HE: The roles of academic middle managers.
Tertiary Education and Management, 13, 19-33.
[13] Hews, R., McNamara, J., & Nay, Z. (2022). Prioritising lifeload over learning load:
Understanding postpandemic student engagement. Journal of University Teaching &
Learning Practice, 19(2), 128-146. https://doi.org/10.53761/1.19.2.9
[14] Neuwirth, L. S., Jović, S., & Mukherji, B. R. (2021). Reimagining higher education during and
post-COVID-19: Challenges and opportunities. Journal of Adult and Continuing Education,
27(2), 141-156. https://doi.org/10.1177/1477971420947738
[15] Zhang, Y., Yang, S., Hu, X., & He, P. (2022). Metaverse Teaching Overview. In 2022 12th
International Conference on Information Technology in Medicine and Education (ITME) v
(pp. 104-109). IEEE.
[16] Mangina, E., McGirl, T., Ryan, G., Murphy, J., & McAuliffe, F. (2022). Experience API (xAPI) for
Virtual Reality (VR) Education in Medicine. In Learning with Technologies and Technologies
in Learning: Experience, Trends and Challenges in Higher Education (pp. 335-359). Cham:
Springer International Publishing.
[17] Zhang, Y., Wang, Y., Fan, G., Song, Y., & Hu, Y. (2023). Multimodal teaching analytics: the
application of SCORM courseware technology integrating 360-degree panoramic VR in
historical courses. Scientific Reports, 13(1), 18893.
[18] Oyelere, A. S., Agbo, F. J., & Oyelere, S. S. (2023). Formative evaluation of immersive virtual
reality expedition mini-games to facilitate computational thinking. Computers & Education:
X Reality, 2, 100016.
[19] Lenart, C., Ahadian, P., Yang, Y., Suo, S., Corsello, A., Kosko, K., & Guan, Q. (2023, October).
Gaze Analysis System for Immersive 360° Video for Preservice Teacher Education. In
Proceedings of the 31st ACM International Conference on Multimedia (pp. 8608-8616).
[20] Duval, E. (2011). Attention please! Learning analytics for visualization and recommendation.
In Proceedings of the 1st international conference on learning analytics and knowledge (pp.
9-17).
[21] Garrido, G. M., Nair, V., & Song, D. (2023). SoK: Data Privacy in Virtual Reality. arXiv preprint
arXiv:2301.05940.