In recent years, the integration of Virtual Reality (VR) into educational settings has gained more and more interest [1]. This surge in popularity is not without reason; on one hand, the technology gets affordable. On the other hand, the immersive and interactive nature of VR provides a unique learning experience. As educators and content creators explore the growing possibilities of VR in education, the need for assessment, design guidelines, best practices, and effective tools to assess and optimize these experiences becomes increasingly important, e.g. Ansone et al. discussing the need for a pedagogic framework for Usability in VR [2].
With Learning Analytics, we can go beyond the goal of providing learners with feedback on their performance using superficial descriptive analytics, a gap identified by Susnjak et al. [3]. Rather it is possible to give teachers and content creators interesting insights into the Joint Proceedings of LAK 2024 Workshops, co-located with 14th International Conference on Learning Analytics and Knowledge (LAK 2024), Kyoto, Japan, March 18-22, 2024. ⋆ You can use this document as the template for preparing your publication. We recommend using the latest version behavior of learners by visualizing learning traces in a Learning Analytics dashboard (LAD). This way, we will enable the creation of content that can be adapted to the actual needs of learners and teachers, create actionable insights for educators and gain fundamental knowledge about learning (behaviour) in VR environments, which should also be generalizable to other use cases.
The assumption that interactive tasks enhance learning outcomes is not new (see [4] and [5]), but we still investigate the design of learning environments. Interactive learning is the basis for many VR educational applications, e.g. to develop spatial abilities [6]. However, the gap persists in the uncertainty surrounding which interactive tasks are most effective in conveying educational content. Addressing this question, we introduce a LAD designed explicitly for educators and content creators (persons creating educational material without teaching them directly). This dashboard serves as a tool for investigating the impact of interactivity in VR learning applications.
While the primary focus is on understanding the effects of interactivity, the learning environment, the data and the study also embark on a nuanced exploration by collecting data in both immersive VR and a desktop VR version without the goal of achieving a media comparison, see [7]. The comparative analysis within the LAD facilitates a comprehensive examination of the two variants, presenting usability issues but also the nuances and similarities between immersive and desktop VR. The dashboard allows for the testing of interactive scenarios of varying complexity, as the application is engineered to be modular.
In essence, this paper presents a LAD to investigate the usage of VR in education. Primary stakeholders are teachers and content creators who are interested in the relationship between interactivity and learning outcomes. The insights gained from this study contribute not only to the refinement of existing VR educational practices but also pave the way for innovative, informed, and economical advancements in future applications.
Related work to this can be examined from different angles, primarily from the perspective of Learning Analytics in VR, but also from the perspective of interactive learning and dashboard design. Related work in the field of computer graphics education is summarized in Heinemann et al. [8]. To summarize the preliminary work, it can be said that very little use has been made of VR as a technology in teaching, but many approaches in computer graphics are interactive and practical.
Reality can be based on various data sources. The technology itself can be used to collect information beyond "simple log data" (often referred to as clickstreams in traditional learning environments), for example, movement data. This means that VR learning applications can, in contrast to classic setups, be evaluated without external sensors using multimodal Learning Analytics (MMLA), see [9]. Oucaichi et al. state that VR is an ideal space to capture learners' behaviour and body movement, but they question whether VR is an ideal learning space [10]. Besides this, the general analysis (as well as the collection, pre-processing, annotation and interpretation) of multimodal data remains a challenge [11]. According to Oucaichi et al. [10], VR is one of the still rather small emerging technologies considered in MMLA research. This work provides approaches to answer the gap regarding the success of VR as a learning medium [10].
Learning Analytics Dashboards There are many studies on dashboards in general, as they are an important tool for involving people in the decision-making process for learning activity and data analysis [12]. Initially, visualizations and predictions were the focus of research. Currently, a research focus is to extend the dashboards to include processes that involve stakeholders and, for example, multimodal data [12]. In addition to current research topics, gaps identified in research on LADs, for example, include the lack of integration of JEDI concepts (justice, equity, diversity, and inclusion) [13]. Including JEDI concepts comes with its own challenges and opportunities; we'll discuss mainly the theme of software development resources.
Learning Analytics Data Format One of the possible data formats specialized for tracking learner activities is the eXperience API (xAPI) specification. To briefly introduce xAPI here, the reason for using this specification is the following: we aim to leverage the eXperience API (xAPI) specification across our work in various forms and flavours of Learning Analytics. Through the use of xAPI statements, user activities are articulated using the syntax: an actor performs a specific verb on an object/activity. The incorporation of xAPI extensions helps to provide additional information about the learning scenario (termed context extension), details regarding the target activity (or object) (activity extension), or additional information concerning task progress (result extensions) [14]. However, xAPI has a degree of freedom on how to use it and isolated solutions are often created. To achieve a more consistent usage standard, it is possible to use xAPI profiles and registries. Ehlenz et al. explore the perspectives afforded by existing and former xAPI registries, especially in an academic area [15]. Possible solutions like an xAPI Registry are already implemented; see [16]. The xAPI Registry was institutionalized to provide a profound basis for the scientific application of xAPI across various disciplines. Developers and researchers have the opportunity to suggest modifications, participate in the maintenance or add new definitions through either GitLab or the web interface. The registry is implemented humanfriendly for interdisciplinary usage, but also implements straightforward machine-readable interfaces. Furthermore, each unique identifier (IRI in xAPI lingo) for definitions is also a valid pointer to a readable version of all stored metadata for both machines and humans, making the data itself even more accessible and hopefully self-explaining.
The research prototype used for this study is the modular application Rendering Pipeline eXperience in Virtual Reality (RePiX VR) [8], demonstration link (YouTube). RePiX VR is an interactive, immersive learning scenario which focuses on teaching the nine core stages (explained in [17]) of the rendering pipeline, a process of converting a 3D environment into a 2D image. It can be used either in a VR environment, using a head-mounted display, or on a computer (desktop VR).
One goal of the application is to address challenges for teachers, such as the didactic reduction for the different target groups, the difficult visualization of the procedural, basic conceptsespecially for individual steps of the pipeline -and the balance between details of the individual steps of the pipeline and the overview of the overall process. Especially at an abstract level, it is difficult to involve students and provide interactions that promote learning actively.
The learning objectives of the simulation are located at different taxonomy levels and thus enable both a gentle learning curve for beginners and an in-depth examination for advanced learners (revision of Bloom's taxonomy [18]). The target dimensions of the application include typical declarative factual knowledge, learning the terminology and acquiring conceptual and procedural knowledge at various taxonomy levels. Examples of the range of learning objectives are conceivable from "Learners name the results of the individual steps of the pipeline" to "Learners can compare different lighting algorithms on a visual level and make a reasoned decision". In order to address learners with heterogeneous knowledge as intuitively as possible, the application was designed as a guided tour in which a robot serves as a contact person and teacher.
To achieve the goal of gathering data for our learning applications, we created the framework OmiLAXR (Open and modular integration of Learning Analytics in XR) [19]. This framework was implemented in several development cycles and was based on requirements derived from experiences gained from various applications and use cases. For the tracking, we chose to use the xAPI specification and an xAPI Registry; see [16].
The OmiLAXR framework focuses on supporting an easy integration of xAPI in XR applications (in Unity). We created components and concepts for enabling the automatized gathering of VR environment activities by a set of tracking systems, e.g. interacting with an object, eye tracking data, movement and head gestures (nodding and shaking). By integrating OmiLAXR in our research object RePiX VR, we achieved a rich collection of user behaviour for our study (explained next).
In October 2022, the RePiX VR project was tested with 128 (VR) + 159 (desktop) students who were participating in the computer graphics lecture of the RWTH Aachen University, resulting in 120 complete data records for the immersive VR variant, 56 complete data sets for the desktop variant. The high number of incomplete data sets in the desktop version is due to the voluntary completion of questionnaires and duplicate registrations from the circle of VR users (to study for the exam). To gain some insights into knowledge acquisition, each participant had to answer surveys directly before and after the learning experience (pre-and post-test design). The content-related questions were the same in both surveys. Still, the second survey also included the UXIVE questionnaire ( [20]) with questions related to 10 concepts of VR experiences, like usability, flow, and presence questions.
To gain some insights into the effects of interactivity while learning, the application, texturing, lighting, and rasterization stage had two versions, an interactive and a non-interactive one; see Table 1. To create a controlled environment, six different application modes were created, where each game mode had different interactive stages. Since the interactive versions of each
Overview of all six different interaction modes/groups. The marks the interactive variant of a stage.
stage are more time-consuming than the non-interactive ones, each participant was shown two interactive and two non-interactive stages.
The non-interactive version of the application stage shows an animation of how each variable of a matrix changes the appearance of an object. The interactive version builds on the noninteractive version. After showing the animation, the learner should use the gained knowledge about matrices to move a sun, earth, and moon as they would move in a solar system.
The non-interactive texturing stage describes how a texture is stored in a 2D texture map and then applied to a 3D object. In the interactive version of the texturing stage, the learner can use a virtual spray can, which allows him to spray in different colours on the 3D object or the 2D texture map. If the participant colours the 3D object, the tint is applied to the texture, and vice versa.
The non-interactive version of the lighting stage gives a short simulation of what happens in a scenario without lighting by turning off the light in the whole learning scenario. The interactive version of the lighting stage shows an object and gives the learner flashlights. Through the flashlights, the learner can light the object from different angles with different light sources. The learner can also switch and compare the effects of different shading models (Flat shading, Gouraud shading, and Phong shading).
The non-interactive rasterization stage explains how, from the 3D scenario, a colour for each screen pixel is calculated so that a monitor can show a 3D scenario. In the interactive version of the rasterization stage, the learner can create triangles and modify the pixel density of a screen. Through those changes, the learner can experience how the created triangles appear on a monitor.
This section will explain the LAD and present insights from the study.
Indicators To create a dashboard that could answer the given research questions, the xAPI statements were filtered and pre-processed to create indicators. For both research questions, the indicators time spent per stage, quantity of specific interactions, task score, and survey score were used. For the difference between the learning experiences in VR compared to the desktop application, further indicators of immersion and health evaluation from the UXIVE questionnaire were of interest.
The data pre-processing was done using Jupyter Notebook1 and the Python library NumPy 2 .
The data of the xAPI statements that were collected during the RePiX VR experience was exported into a JSON file format. The xAPI statements contain an extension with the timestamp, which enables the calculation of the difference between the timestamp when a stage started and it finished. For each interaction, an xAPI statement was sent. Therefore, the question of how often each interaction was tracked, an indicator was created that counted how often each verb of the xAPI statement was sent. For each interactive task in the RePiX VR application that was solved correctly, the learner gained a score. The score is higher the faster a task is solved. The final task score is the average of all scores gained during the learning experience.
The survey data was used in a CSV data format export from LimeSurvey 3 . Data from the UXIVE was used for the survey score, health evaluation, and immersion evaluation. The survey consists of questions that are either positively connoted, e. Furthermore, the survey score was calculated by taking the difference of the average points for all positively and negatively connoted questions. The health and immersion evaluation was done by calculating the score for each question related to the topic of health or immersion.
After the analysis and pre-processing of the data, the LAD was developed (active URL in Appendix A). The pre-processing results were exported into a JSON file, which was then imported into the dashboard. The dashboard was used to create interactive visualizations and was developed using Vue.js 4 for the frontend development and Apexcharts 5 for the visualizations.
This section presents the finished dashboard. The dashboard consists of three different pages: overview (Figure 1), position analysis (Figure 3), and comparison (Figure 2). Each page presents the research questions on the left side, while the top right corner is used as a menu bar to switch between the different pages. Below the research question, an explanation for the selected box plots is given. Each box plot only shows data of the participants that were between 5% and 95% of the data set, and their quantiles presenting the students from 30% to 70% of the data set. The visualizations for immersion and health evaluation use only the participants' data between 40% to 60% of the data because the data for these two visualizations are widely spread, and the selected data still provides a good approximation of the average participant. Next, each view provides general data information about the study. First of all, it shows for which group which of the stages was interactive (like Table 1). Also, a distribution of how many data sets for each group exist is given. Lastly, information about how many participants did not finish the study is given.
In addition to the information that each page of the dashboard provides, the overview page (Figure 1) also provides global filters on the right side. These filters are used to change the visualizations interactively. For example, when deselecting the checkbox of Group A, the collected data of Group A will no longer be considered for the current visualizations.
Through the development process, it was found that this approach makes it difficult to compare the visualizations of different filter combinations, which is the reason why the comparison page was added. The comparison page (Figure 2) is split into two halves. Each half of the page has its local filter, independently configuring the visualizations on the same half. Therefore, visualizations with different filters are shown directly beside each other and make a direct and adaptive comparison of the data set possible. Using the comparison page makes it possible to answer the research questions from educators and content creators. On the last page, the position analysis was added to analyze the movements inside the application. The goal was to create a heat map to indicate the most common position in a single time step, The overall position distribution is shown on the dashboard in Figure 3. This visualization gives teachers an impression beyond the interaction data as to whether different perspectives were used to solve the three-dimensional tasks and to check certain objects. Position data can also help to distinguish users who moved themselves using the physical VR area (or WASD keyboard navigation) versus learners who preferred to teleport.
To get answers to the given research questions, different filters were set on the implemented dashboard. The first research question (1) In which aspects differ the learning experiences when using Virtual Reality instead of 2D desktop applications?, can be investigated by selecting the VR filter on one side of the comparison page and the desktop filter on the other side of the comparison page. The filter selection showed that RePiX VR was perceived as more immersive, the survey score was higher, and the participants moved more in the VR environment. At the same time, the VR environment resulted in more health issues, a lower task score, and fewer interactions than the desktop environment.
For the second research question (2) In which aspects differ the learning experiences for interactive scenarios instead of behaviouristic informative texts?, the only quantitative indicator was that participants spent more time in interactive than non-interactive scenarios.
However, these are only the results that can be taken directly from the quantitative data.
Educators and content creators can already reflect on the teaching application RePiX VR at a meta-level. This study already provided us with some valuable insights showing usability issues and further steps. When quantitative data is also (manually) integrated, it shows that the combination of data, like the time needed with assessment, is feasible and sensible. Interactive tasks can thereby be used to provide learners with in-app feedback.
Another publication, which analyzed quantitative data from an earlier study, has already shown that a reliable text-to-speech system for the guiding robot is useful to provide feedback to the learner without having to rely on text boxes [21]. This study confirms that people with usability problems take longer and wish for more help in the free text answers, which can be implemented most simply, for example, via timers.
This paper presents a dashboard in closer detail, which can be used to evaluate various aspects of the VR experience. By using it to reflect on the learning application, the dashboard itself is implicitly put to the test. As indicated in the description, the basic structure of the dashboard can also be transferred to learners. Necessary adjustments include anonymization and highlighting one's own performance, as well as a study on the connection between user behaviour and outcome, which can be used to generate feedback. Check Van Leeuwen et al. for more differing aspects [22].
Other transfers are also feasible but are beyond the scope of this paper. One of the key educational challenges for usage poses the duration of the VR experience, so introducing interactivity at every stage is not the solution. Educators should emphasize selected learning objectives and create a guided tour which fits the curriculum. One option is to split up the VR experience or offer elective excursions for content beyond the core curriculum, e.g. concerning the light stage or in combination with other applications like Virtual Ray Tracer 2.0 [23] or Rayground [24].
Too many interactive tasks risk losing the bigger picture, meaning that the sum of interactive tasks on basal levels of learning goal taxonomies fades the higher-level goals into the background. Like reading code line by line without grasping the algorithm itself, learners could lose themselves in individual steps without developing an understanding of the rendering pipeline as a whole -based on the theory of the block model, an educational model for program comprehension, which also argues on different dimensions of knowledge in computer science, see [25].
An open question remains about the effectiveness of the didactical setup (single usage): Can this application be used for learning in groups, engaging in a collaborative reflection process, or is the individual experience sufficient, especially regarding the principles of self-regulated learning [26].
The evaluation and the insights generated by the dashboard should, for now, be considered as an individual case study (with no claims for generalizability), as the user of the dashboard has to know both the content being taught as well as the VR application itself. Therefore, it is only used by a small number of persons yet. Besides, interpreting the presented data is a complex process [11], deriving actions for the didactic decisions is a reflective process. Consequently, only actual educators or educational content creators are viable focus groups for evaluating the dashboard.
In the future, this dashboard could be extended not only by providing new filters and visualizations but also by customization possibilities based on the well-structured xAPI format.
Another possibility is to transfer it to different use cases while keeping the same interfaces, data collectors, and indicators. And thereby fulfilling the criteria for shared software resources, cross-border collaborations and adoption of open-source software, see [13]. This is particularly easy due to the standardized data specification xAPI and the standardized interfaces provided by OmiLAXR. Incorporating new filters and additional information could further incorporate JEDI into the dashboard, e.g. by highlighting information certain students have in common [13].
In our case, it could be possible that there is a cluster of students with shared experiences in prior VR usage.
Furthermore, we found that the application can be a valuable resource for students apart from the accompanying integration during the course, e.g. as interactive preparation for the final exams, which is an insight from the qualitative data we collected in the survey.
In this paper, we presented a study and the first attempt to create a meaningful dashboard for educators using the RePiX VR application. We tracked user behaviour (object interaction, eye tracking, movement, head gestures). We conducted a pre-and post-test before and after the learning experience to gain more insights about interactivity and learning effects. In the study, we distinguished between different application variants (with different stages, interactive or not). Finally, the PPDAC (Problem, Plan, Data, Analysis and Conclusion) cycle was used to create an interactive LA dashboard; see [27]. Although the dashboard delivers a good overview of our study, the interpretation needs some expertise in the learning content. As a result, our research prototype has been identified as a valuable additional learning material for students of the computer graphics course at our university and further steps for the development of the application and didactic decisions were made. The collected data, in combination with our dashboard, also emphasized xAPI as a useful data format for realizing LA dashboards because of its flexibility and expandability.
The application, data and all sources are available via • Dashboard code and online demo instance • VR Learning Application (RePiX VR) code and website