Learning in Virtual Reality ofers various ways to make the learning process interactive, but the implementation of such features is complex, time-consuming and expensive. In order to evaluate the eficiency of interactive tasks, a Learning Analytics dashboard, presented in this paper, was created for both teachers/educators and content creators. The dashboard presents data from a study with diferent interactive/immersive and non-interactive/non-immersive variants of a learning application for the rendering pipeline, a showcase topic from computer graphics. The dashboard has been implemented with transferability in mind by using xAPI as a data format and can thus be easily transferred to other contexts.
In recent years, the integration of Virtual Reality (VR) into educational settings has gained
more and more interest [
With Learning Analytics, we can go beyond the goal of providing learners with feedback on their performance using superficial descriptive analytics, a gap identified by Susnjak et al. [3]. Rather it is possible to give teachers and content creators interesting insights into the behavior of learners by visualizing learning traces in a Learning Analytics dashboard (LAD). This way, we will enable the creation of content that can be adapted to the actual needs of learners and teachers, create actionable insights for educators and gain fundamental knowledge about learning (behaviour) in VR environments, which should also be generalizable to other use cases.
The assumption that interactive tasks enhance learning outcomes is not new (see [4] and [5]), but we still investigate the design of learning environments. Interactive learning is the basis for many VR educational applications, e.g. to develop spatial abilities [6]. However, the gap persists in the uncertainty surrounding which interactive tasks are most efective in conveying educational content. Addressing this question, we introduce a LAD designed explicitly for educators and content creators (persons creating educational material without teaching them directly). This dashboard serves as a tool for investigating the impact of interactivity in VR learning applications.
While the primary focus is on understanding the efects of interactivity, the learning environment, the data and the study also embark on a nuanced exploration by collecting data in both immersive VR and a desktop VR version without the goal of achieving a media comparison, see [7]. The comparative analysis within the LAD facilitates a comprehensive examination of the two variants, presenting usability issues but also the nuances and similarities between immersive and desktop VR. The dashboard allows for the testing of interactive scenarios of varying complexity, as the application is engineered to be modular.
In essence, this paper presents a LAD to investigate the usage of VR in education. Primary stakeholders are teachers and content creators who are interested in the relationship between interactivity and learning outcomes. The insights gained from this study contribute not only to the refinement of existing VR educational practices but also pave the way for innovative, informed, and economical advancements in future applications.
Related work to this can be examined from diferent angles, primarily from the perspective of Learning Analytics in VR, but also from the perspective of interactive learning and dashboard design. Related work in the field of computer graphics education is summarized in Heinemann et al. [8]. To summarize the preliminary work, it can be said that very little use has been made of VR as a technology in teaching, but many approaches in computer graphics are interactive and practical.
Multimodal Learning Analytics and VR Learning Analytics in Virtual Reality can be based on various data sources. The technology itself can be used to collect information beyond "simple log data" (often referred to as clickstreams in traditional learning environments), for example, movement data. This means that VR learning applications can, in contrast to classic setups, be evaluated without external sensors using multimodal Learning Analytics (MMLA), see [9]. Oucaichi et al. state that VR is an ideal space to capture learners’ behaviour and body movement, but they question whether VR is an ideal learning space [10]. Besides this, the general analysis (as well as the collection, pre-processing, annotation and interpretation) of multimodal data remains a challenge [11]. According to Oucaichi et al. [10], VR is one of the still rather small emerging technologies considered in MMLA research. This work provides approaches to answer the gap regarding the success of VR as a learning medium [10].
Learning Analytics Dashboards There are many studies on dashboards in general, as they are an important tool for involving people in the decision-making process for learning activity and data analysis [12]. Initially, visualizations and predictions were the focus of research. Currently, a research focus is to extend the dashboards to include processes that involve stakeholders and, for example, multimodal data [12]. In addition to current research topics, gaps identified in research on LADs, for example, include the lack of integration of JEDI concepts (justice, equity, diversity, and inclusion) [13]. Including JEDI concepts comes with its own challenges and opportunities; we’ll discuss mainly the theme of software development resources.
Learning Analytics Data Format One of the possible data formats specialized for tracking learner activities is the eXperience API (xAPI) specification. To briefly introduce xAPI here, the reason for using this specification is the following: we aim to leverage the eXperience API (xAPI) specification across our work in various forms and flavours of Learning Analytics. Through the use of xAPI statements, user activities are articulated using the syntax: an actor performs a specific verb on an object/activity. The incorporation of xAPI extensions helps to provide additional information about the learning scenario (termed context extension), details regarding the target activity (or object) (activity extension), or additional information concerning task progress (result extensions) [14]. However, xAPI has a degree of freedom on how to use it and isolated solutions are often created. To achieve a more consistent usage standard, it is possible to use xAPI profiles and registries. Ehlenz et al. explore the perspectives aforded by existing and former xAPI registries, especially in an academic area [15]. Possible solutions like an xAPI Registry are already implemented; see [16]. The xAPI Registry was institutionalized to provide a profound basis for the scientific application of xAPI across various disciplines. Developers and researchers have the opportunity to suggest modifications, participate in the maintenance or add new definitions through either GitLab or the web interface. The registry is implemented humanfriendly for interdisciplinary usage, but also implements straightforward machine-readable interfaces. Furthermore, each unique identifier (IRI in xAPI lingo) for definitions is also a valid pointer to a readable version of all stored metadata for both machines and humans, making the data itself even more accessible and hopefully self-explaining.
The research prototype used for this study is the modular application Rendering Pipeline eXperience in Virtual Reality (RePiX VR) [8], demonstration link (YouTube). RePiX VR is an interactive, immersive learning scenario which focuses on teaching the nine core stages (explained in [17]) of the rendering pipeline, a process of converting a 3D environment into a 2D image. It can be used either in a VR environment, using a head-mounted display, or on a computer (desktop VR).
One goal of the application is to address challenges for teachers, such as the didactic reduction for the diferent target groups, the dificult visualization of the procedural, basic concepts especially for individual steps of the pipeline - and the balance between details of the individual steps of the pipeline and the overview of the overall process. Especially at an abstract level, it is dificult to involve students and provide interactions that promote learning actively.
The learning objectives of the simulation are located at diferent taxonomy levels and thus
enable both a gentle learning curve for beginners and an in-depth examination for advanced
learners (revision of Bloom’s taxonomy [
To achieve the goal of gathering data for our learning applications, we created the framework
OmiLAXR (Open and modular integration of Learning Analytics in XR) [
The OmiLAXR framework focuses on supporting an easy integration of xAPI in XR applications (in Unity). We created components and concepts for enabling the automatized gathering of VR environment activities by a set of tracking systems, e.g. interacting with an object, eye tracking data, movement and head gestures (nodding and shaking). By integrating OmiLAXR in our research object RePiX VR, we achieved a rich collection of user behaviour for our study (explained next).
In October 2022, the RePiX VR project was tested with 128 (VR) + 159 (desktop) students who
were participating in the computer graphics lecture of the RWTH Aachen University, resulting
in 120 complete data records for the immersive VR variant, 56 complete data sets for the desktop
variant. The high number of incomplete data sets in the desktop version is due to the voluntary
completion of questionnaires and duplicate registrations from the circle of VR users (to study
for the exam). To gain some insights into knowledge acquisition, each participant had to
answer surveys directly before and after the learning experience (pre- and post-test design). The
content-related questions were the same in both surveys. Still, the second survey also included
the UXIVE questionnaire ([
To gain some insights into the efects of interactivity while learning, the application, texturing, lighting, and rasterization stage had two versions, an interactive and a non-interactive one; see Table 1. To create a controlled environment, six diferent application modes were created, where each game mode had diferent interactive stages. Since the interactive versions of each Variant
Application
Texturing
Lighting
Rasterization marks the interactive variant of a stage. stage are more time-consuming than the non-interactive ones, each participant was shown two interactive and two non-interactive stages.
The non-interactive version of the application stage shows an animation of how each variable of a matrix changes the appearance of an object. The interactive version builds on the noninteractive version. After showing the animation, the learner should use the gained knowledge about matrices to move a sun, earth, and moon as they would move in a solar system.
The non-interactive texturing stage describes how a texture is stored in a 2D texture map and then applied to a 3D object. In the interactive version of the texturing stage, the learner can use a virtual spray can, which allows him to spray in diferent colours on the 3D object or the 2D texture map. If the participant colours the 3D object, the tint is applied to the texture, and vice versa.
The non-interactive version of the lighting stage gives a short simulation of what happens in a scenario without lighting by turning of the light in the whole learning scenario. The interactive version of the lighting stage shows an object and gives the learner flashlights. Through the lfashlights, the learner can light the object from diferent angles with diferent light sources. The learner can also switch and compare the efects of diferent shading models (Flat shading, Gouraud shading, and Phong shading).
The non-interactive rasterization stage explains how, from the 3D scenario, a colour for each screen pixel is calculated so that a monitor can show a 3D scenario. In the interactive version of the rasterization stage, the learner can create triangles and modify the pixel density of a screen. Through those changes, the learner can experience how the created triangles appear on a monitor.
Research Question The dashboard is created to give educators and content creators insights into two research questions: (1) In which aspects difer the learning experiences when using immersive Virtual Reality instead of 2D desktop applications? and (2) In which aspects difer the learning experiences for interactive scenarios instead of behaviouristic informative texts or animations?.
Indicators To create a dashboard that could answer the given research questions, the xAPI statements were filtered and pre-processed to create indicators. For both research questions, the indicators time spent per stage, quantity of specific interactions , task score, and survey score were used. For the diference between the learning experiences in VR compared to the desktop application, further indicators of immersion and health evaluation from the UXIVE questionnaire were of interest.
Data Analysis The data pre-processing was done using Jupyter Notebook1 and the Python library NumPy2.
The data of the xAPI statements that were collected during the RePiX VR experience was exported into a JSON file format. The xAPI statements contain an extension with the timestamp, which enables the calculation of the diference between the timestamp when a stage started and it finished. For each interaction, an xAPI statement was sent. Therefore, the question of how often each interaction was tracked, an indicator was created that counted how often each verb of the xAPI statement was sent. For each interactive task in the RePiX VR application that was solved correctly, the learner gained a score. The score is higher the faster a task is solved. The final task score is the average of all scores gained during the learning experience.
The survey data was used in a CSV data format export from LimeSurvey3. Data from the UXIVE was used for the survey score, health evaluation, and immersion evaluation. The survey consists of questions that are either positively connoted, e.g. "Please indicate to what extent you agree with each statement: [ I enjoyed the experience so much that I was energetic.]", or negatively connoted, e.g. "Please indicate to what extent you agree with each statement: [During the experience my eyes hurt.]". For each question, the learner should give a score using an 8-point Likert Scale.
Furthermore, the survey score was calculated by taking the diference of the average points for all positively and negatively connoted questions. The health and immersion evaluation was done by calculating the score for each question related to the topic of health or immersion. Dashboard Implementation After the analysis and pre-processing of the data, the LAD was developed (active URL in Appendix A). The pre-processing results were exported into a JSON ifle, which was then imported into the dashboard. The dashboard was used to create interactive visualizations and was developed using Vue.js4 for the frontend development and Apexcharts5 for the visualizations.
Dashboard Visualization This section presents the finished dashboard. The dashboard consists of three diferent pages: overview (Figure 1), position analysis (Figure 3), and comparison (Figure 2). Each page presents the research questions on the left side, while the top right corner is used as a menu bar to switch between the diferent pages. Below the research question, an explanation for the selected box plots is given. Each box plot only shows data of the participants 1https://jupyter.org/, accessed: 14.12.2023 2https://numpy.org/, accessed: 14.12.2023 3https://www.limesurvey.org/, accessed: 14.12.2023 4https://vuejs.org/, accessed: 14.12.2023 5https://apexcharts.com/, accessed: 14.12.2023 that were between 5% and 95% of the data set, and their quantiles presenting the students from 30% to 70% of the data set. The visualizations for immersion and health evaluation use only the participants’ data between 40% to 60% of the data because the data for these two visualizations are widely spread, and the selected data still provides a good approximation of the average participant. Next, each view provides general data information about the study. First of all, it shows for which group which of the stages was interactive (like Table 1). Also, a distribution of how many data sets for each group exist is given. Lastly, information about how many participants did not finish the study is given.
In addition to the information that each page of the dashboard provides, the overview page (Figure 1) also provides global filters on the right side. These filters are used to change the visualizations interactively. For example, when deselecting the checkbox of Group A, the collected data of Group A will no longer be considered for the current visualizations.
Through the development process, it was found that this approach makes it dificult to compare the visualizations of diferent filter combinations, which is the reason why the comparison page was added. The comparison page (Figure 2) is split into two halves. Each half of the page has its local filter, independently configuring the visualizations on the same half. Therefore, visualizations with diferent filters are shown directly beside each other and make a direct and adaptive comparison of the data set possible. Using the comparison page makes it possible to answer the research questions from educators and content creators.
On the last page, the position analysis was added to analyze the movements inside the application. The goal was to create a heat map to indicate the most common position in a single time step, The overall position distribution is shown on the dashboard in Figure 3. This visualization gives teachers an impression beyond the interaction data as to whether diferent perspectives were used to solve the three-dimensional tasks and to check certain objects. Position data can also help to distinguish users who moved themselves using the physical VR area (or WASD keyboard navi- Figure 3: The position page of the LAD provides gation) versus learners who preferred to tele- a distribution of the position in the port. virtual environment of all participants in the RePiX VR study.
Results To get answers to the given research questions, diferent filters were set on the implemented dashboard. The first research question (1) In which aspects difer the learning experiences when using Virtual Reality instead of 2D desktop applications?, can be investigated by selecting the VR filter on one side of the comparison page and the desktop filter on the other side of the comparison page. The filter selection showed that RePiX VR was perceived as more immersive, the survey score was higher, and the participants moved more in the VR environment. At the same time, the VR environment resulted in more health issues, a lower task score, and fewer interactions than the desktop environment.
For the second research question (2) In which aspects difer the learning experiences for interactive scenarios instead of behaviouristic informative texts?, the only quantitative indicator was that participants spent more time in interactive than non-interactive scenarios.
However, these are only the results that can be taken directly from the quantitative data. Educators and content creators can already reflect on the teaching application RePiX VR at a meta-level. This study already provided us with some valuable insights showing usability issues and further steps. When quantitative data is also (manually) integrated, it shows that the combination of data, like the time needed with assessment, is feasible and sensible. Interactive tasks can thereby be used to provide learners with in-app feedback.
Another publication, which analyzed quantitative data from an earlier study, has already
shown that a reliable text-to-speech system for the guiding robot is useful to provide feedback
to the learner without having to rely on text boxes [
This paper presents a dashboard in closer detail, which can be used to evaluate various aspects
of the VR experience. By using it to reflect on the learning application, the dashboard itself is
implicitly put to the test. As indicated in the description, the basic structure of the dashboard can
also be transferred to learners. Necessary adjustments include anonymization and highlighting
one’s own performance, as well as a study on the connection between user behaviour and
outcome, which can be used to generate feedback. Check Van Leeuwen et al. for more difering
aspects [
Other transfers are also feasible but are beyond the scope of this paper.
One of the key educational challenges for usage poses the duration of the VR experience, so
introducing interactivity at every stage is not the solution. Educators should emphasize selected
learning objectives and create a guided tour which fits the curriculum. One option is to split
up the VR experience or ofer elective excursions for content beyond the core curriculum, e.g.
concerning the light stage or in combination with other applications like Virtual Ray Tracer 2.0
[
Too many interactive tasks risk losing the bigger picture, meaning that the sum of
interactive tasks on basal levels of learning goal taxonomies fades the higher-level goals into the
background. Like reading code line by line without grasping the algorithm itself, learners
could lose themselves in individual steps without developing an understanding of the rendering
pipeline as a whole - based on the theory of the block model, an educational model for program
comprehension, which also argues on diferent dimensions of knowledge in computer science,
see [
An open question remains about the efectiveness of the didactical setup (single usage): Can
this application be used for learning in groups, engaging in a collaborative reflection process,
or is the individual experience suficient, especially regarding the principles of self-regulated
learning [
The evaluation and the insights generated by the dashboard should, for now, be considered as an individual case study (with no claims for generalizability), as the user of the dashboard has to know both the content being taught as well as the VR application itself. Therefore, it is only used by a small number of persons yet. Besides, interpreting the presented data is a complex process [11], deriving actions for the didactic decisions is a reflective process. Consequently, only actual educators or educational content creators are viable focus groups for evaluating the dashboard.
In the future, this dashboard could be extended not only by providing new filters and visualizations but also by customization possibilities based on the well-structured xAPI format.
Another possibility is to transfer it to diferent use cases while keeping the same interfaces, data collectors, and indicators. And thereby fulfilling the criteria for shared software resources, cross-border collaborations and adoption of open-source software, see [13]. This is particularly easy due to the standardized data specification xAPI and the standardized interfaces provided by OmiLAXR. Incorporating new filters and additional information could further incorporate JEDI into the dashboard, e.g. by highlighting information certain students have in common [13]. In our case, it could be possible that there is a cluster of students with shared experiences in prior VR usage.
Furthermore, we found that the application can be a valuable resource for students apart from the accompanying integration during the course, e.g. as interactive preparation for the ifnal exams, which is an insight from the qualitative data we collected in the survey.
In this paper, we presented a study and the first attempt to create a meaningful dashboard for
educators using the RePiX VR application. We tracked user behaviour (object interaction, eye
tracking, movement, head gestures). We conducted a pre- and post-test before and after the
learning experience to gain more insights about interactivity and learning efects. In the study,
we distinguished between diferent application variants (with diferent stages, interactive or
not). Finally, the PPDAC (Problem, Plan, Data, Analysis and Conclusion) cycle was used to
create an interactive LA dashboard; see [
• Dashboard code and online demo instance • VR Learning Application (RePiX VR) code and website