Immersive virtual reality (IVR) enables the delivery of educational content in situations where traditional in-person instruction would be dangerous, impossible, counterproductive, or simply too expensive [1]. Not surprisingly, there has been a steady increase in research interest, investigating the promise and pitfalls of VR in education [2].
Besides the situational benefits, another important strength of IVR is hardware related. Modern consumer IVR headsets are equipped with an array of various built-in sensors. Originally designed to enable and enhance the experience of immersive games, they can also be exploited for the purpose of gathering real-time user data that can be related to the learning process and outcome. For example, positional data can provide insight about learning outcome [3], cognitive load [4], and social interactions [5].
One particular sensor that has been previously hardly accessible but is finding its way into consumer devices is eye tracking. Put simply, video-based eye trackers emit infrared/nearinfrared light and utilize the resulting corneal reflections and their spatial relation to the center of the pupil to estimate eye gaze vectors [6]. In combination with IVR, eye tracking offers unprecedented opportunities to study human behavior and cognition [7]. IVR allows creating highly realistic and controlled environments, and modern game engines make it relatively easy to record gaze directions and areas of interest (AOI) compared to mobile eye tracking systems that track gaze in the real world. It has also been demonstrated that eye trackers integrated into IVR headsets achieve sufficient levels to reliably identify the current fixation location, provided that the gaze targets of interest are not in close proximity [8].
Consequently, the value of eye tracking in IVR could be demonstrated across a wide range of tasks. More specifically, eye tracking was shown to enhance user interactions in IVR, for example object selection [9] or typing on a virtual keyboard [10]. In the context of education and training, it is important to note that eye tracking can be used to infer cognitive load [11], joint attention of learners [12], and the distribution of teachers' visual attention in the classroom [13]. This opens up many possibilities, ranging from personalized IVR learning experiences to enhanced performance feedback for learners and teachers, respectively.
However, despite the promising research findings, eye tracking is still underrepresented in practical settings outside a scientific context. It is conceivable that the higher cost of IVR headsets with integrated eye-trackers make these devices less accessible for educational use cases. This is even more relevant in the case of collaborative learning, where a classroom would need to be equipped with a higher number of head-mounted displays.
Therefore, this study set out to investigate whether the position and orientation (i.e. pose) of an IVR headset offers a viable approximation of eye gaze. The research question was driven by the idea that, provided head pose (hereafter referred to as head gaze) and eye gaze align sufficiently well, the former could be used to substitute the latter, therefore offering a low-cost alternative to IVR headsets with integrated eye trackers.
Despite the high practical relevance, little research exists to date that studied whether head gaze can sufficiently approximate eye gaze in IVR. However, there is a recent study that argued that head gaze can indeed serve as a proxy for eye gaze in the context of human-robot interaction [14] when the aim is to teach a (virtual) robot about a person's intent, i.e. what object a person is intending to interact with. Similarly, head gaze has proven useful in a scenario involving the collaboration with a virtual agent [15]. In this study, the use of bidirectional head gaze between human participants and a virtual character was shown to have a similar positive effect on task performance as bidirectional gaze using eye tracking.
In the same vein, a few studies from the field of social psychology have utilized head gaze as a proxy for social eye contact. For example, one study investigated how participants interacted with a virtual physician during a simulated clinical visit [16]. The authors reported that the emotional state of the participants influenced the amount of eye contact they made during the conversation with the physician. Another IVR study tracked nonverbal behavior of participants in a virtual classroom and found different patterns of head movement depending on the level of self-reported social anxiety. Participants with higher level of anxiety exhibited more lateral head movement, indicating increased room scanning behavior compared to participants with low levels of anxiety [17].
Both studies made the implicit assumption that users are mostly looking straight ahead when wearing an IVR headset, thus exhibiting little eye-in-head motion range. Although it has shown to be useful to approximate eye tracking with head tracking [15,14], it is noteworthy that users' eye movements in IVR can show quite substantial deviations from the forward direction of the head pose. Sidenmark and Gellersen investigated the coordination of eye, head, and body movements during gaze shifts [18]. They found that smaller gaze shifts of 25°visual angle or less are predominantly performed with the eyes and without much contribution from the head or torso. However, they also reported large inter-individual differences between users in terms of the eyes' motion range, varying from 20°to 70°visual angle. In line with these findings, another study recently found a high correspondence between eye and head movements in IVR, leading to an accuracy of 75% for AOI with an angular size of 25°, with a substantial drop in accuracy when the AOI were smaller [19].
Taken together, the existing literature shows initial evidence that head gaze can be successfully utilized to approximate eye gaze in IVR, provided that careful attention is directed towards the design of the virtual objects (i.e., AOI). However, we are not aware of studies that investigated the practicability of these findings in applied settings of learning or training. Therefore, the aim of the study was twofold. First, we aimed to evaluate the similarity between eye and head gaze in a dynamic virtual teaching scenario. Based on the previous findings, we hypothesized that we would observe a high correspondence of head pose and eye gaze in a sparsely furnished IVR training environment (i.e., a scene with predominantly large AOI). Second, we investigated whether we could use a machine learning algorithm to successfully predict the correct eye gaze targets based on the head gaze plus additional positional tracking data recorded from the IVR headset and corresponding hand controllers.
Forty-five pre-service teachers (PSTs) at a large metropolitan university in South Africa participated in the study. The sample consisted of third-year undergraduates from the Department of Science and Technology education.
As an integral part of their third-year curriculum, PSTs are practicing their teaching skills by conducting several microlessons throughout their studies. Microlessons are defined as short lesson presentations, typically revolving around a single, tightly defined topic [20]. The goal of microlessons is to develop the PSTs' pedagogical skills in a safe environment and to teach them how to reflect on their own behavior.
In the context of this study, PSTs chose one of sixteen topics from the subjects of biology, physics and chemistry. Their task was to prepare the lesson and deliver it using a learnercentered, inquiry-based teaching strategy inside the IVR environment. In an inquiry-based science classroom, the teacher is seen as a facilitator, who provides ample opportunities for learners to actively engage in the learning process [21,22]. The study was approved by the local ethics committee of the University of Johannesburg.
The IVR application was co-designed with five teacher educators to ensure the alignment with inquiry-based teaching. Hence, the IVR classroom was set up in the Unity Game Engine with two types of tables 1) a main table to accommodate students during the introductory and closing phases of the microlessons, and 2) three separate breakout tables, where groups of two to three students could collaborate on a given task. The classroom was also equipped with a whiteboard for slide presentations and drawings, and a flipchart for displaying quiz results. Furthermore, there was a teacher's podium that hosted a control panel to manipulate various classroom functions (e.g., controlling the slides, starting a quiz, etc.). Importantly, it could also be used to select, spawn, and move 3D objects as well as students between tables. For illustration, a screenshot of the IVR environment is depicted in Figure 1.
Before the participants delivered their microteaching lesson, they received a brief training about the IVR classroom including a short hands-on experience to familiarize them with the available tools and objects of the IVR classroom. Then, the PSTs carried out the teaching exercise while changing roles after each microlesson. For example, in a group of four PSTs A, B, C, and D, PST A would first take on the role of teacher, while the other three PSTs would assume the role of students. After a maximum of 15 minutes allocated for the microlesson, they would change roles and repeat the procedure until each PST had completed their lesson. Later, the PSTs received individual feedback on their teaching behavior from their educator based on a recording of the lesson and a learning analytics dashboard.
From a total of 51 microlessons held, we selected 23 based on the following criteria: functional version of the application able to record the eye rotation data (11 excluded sessions), minimum duration of 5 minutes (17 excluded), and 2 lessons were excluded due to the reusage of the same login for a teacher and a student in the same microlesson. This filtering resulted in excluding 6 out of 24 participants as teachers from the dataset. Eye gaze data was only collected for the teacher roles because these participants wore a Meta Quest Pro headset with integrated eye tracker as opposed to the Meta Quest 2 headsets for the student role. PSTs in the teacher role received feedback on their eye gaze behavior via the learning analytics dashboard after the lesson. Eye tracking was not available for the student roles due to the limited availability and higher cost of the eye tracking enabled IVR headsets. Raw data was collected automatically from each device during the run of the microlessons. The collected sensors included pose data (position and rotation) from the IVR headset and both hand controllers, rotation for both eyes and the head and the objects where the user was gazing at. These are summarized in Table 1. These sensory data were collected independently for each user and sensor, with different sampling rate for each sensor -10 Hz for positional data and 20 Hz for the eye tracking data. Hence, the sensory data needed to be synchronized first. This was done by creating 50ms time windows and taking a) the minimum value inside each window time frame for the positional and rotational data and b) the union of all objects present in the eye and head gazing data. Each collected row represents a 50ms long window from a microlesson. Concatenating data from all microlessons generated N=439,749 rows.
We modeled the problem as a multi-class classification. The target variable was the object that was detected as being gazed at by the user's eyes. Before training the model, the dataset had to be cleaned. This included the removal of eye rotation values outside the reported field-of-view of the IVR headset (representing recording failures) and saccades. Several approaches exist to identify fixations and saccades, respectively. We used an Area-of-Interest Identification algorithm, which defines a fixation as a group of consecutive gaze points that fall within the same target area [23]. Groups that did not span a minimum duration of 100 ms were regarded as saccades and excluded from the analysis. This filtering resulted in a reduced dataset of N=186,864 rows.
The gazed objects can distinguish between specific users, but since these users were different across sessions, a class User was created to represent all the users. This processing resulted in a dataset with N=338,040 rows with 13 different classes1 , recorded for 18 users in 23 microlessons, with the average duration 16 minutes (min=5, max=28) and four other students on average being present apart from the teacher (min=1, max=5).
For the modeling, we utilized a Random Forest classifier. This selection was motivated by Random Forest being often one of the best classifiers in Learning Analytics data [24], but also in a recent study on VR data collected from an educational application [25]. We used the implementation from the R ranger package [26]. Due to the large dataset, we did not perform hyperparameter tuning and for the same reason, we used the version of training without replacement and training on a .632 fraction of the data, setting the seed=123, and leaving all other parameters default.
We used 5-fold cross validation (i.e. always 80% of the dataset with 20% left for testing) with the split was stratified by the class distribution. The random forest model (further referred as "CLASSIFIER") was compared to two baseline models. 1) Model "FLOOR" represents a naive classifier that classifies all the instances to the majority class. 2) Model "HEAD GAZE" represents a model that is using the gazed object as derived from the head gaze. All the metrics are reported as mean and standard deviation across the 5 testing folds.
For both the machine learning model and the baselines, we report usual metrics for a multiclass classification, i.e. precision, recall and F1-score, which were averaged across all the classes using a) the weighted and b) macro average. We focus more on the weighted average because we think it is important to consider the distribution of the classes.
The results of both average types are depicted in Figure 2. We see that the performance using only the baseline model "FLOOR" is very poor, both for the weighted and macro average. The only value above .20 is a weighted-average recall, due to classifying the largest class affecting the weighted average more than the macro. The "HEAD GAZE" baseline performs better than "FLOOR" on all the metrics, showing promising direction. However all the values are below .35, which is still very poor performance.
On the other hand, both averages reveal a steep increase in the performance for the "CLASSI-FIER" over both of the baselines, with the F1-score for the macro-average .7568 and the weighted average .8637. The higher values of the weighted average are caused by a better performance on the larger classes. This is expected, as some of the minor classes might not have a sufficient representation in the dataset to produce good results.
A similar picture about the improvement of the machine learning model compared to the baseline appears from Figure 3, depicting the heatmaps for the "HEAD GAZE" and "CLASSIFIER" predictors. While the baseline model matrix is quite scattered, and full of misclassifications for almost every class, the "CLASSIFIER" model reveals a more pronounced diagonal line indicating higher precision on all classes. For example, it is apparent that the "HEAD GAZE", misclassified many objects as "User". This would indicate that a teacher is indeed paying closer attention to students, a laudable feature of student-centered teaching. These false-positives for a "User" are significantly reduced for the "CLASSIFIER" model. Still, the classifier is far from perfect, especially because of the many misclassifications for the two largest classes "Room Wall" and "Room Floor".
Investigating the use of positional tracking in a microteaching scenario, we found a low correspondence of gaze targets inferred from eye tracking and the forward orientation of an IVR headset, that is, comparing eye gaze and head gaze. This result suggests that head gaze alone does not sufficiently approximate eye gaze, which is in contrast to previous reports claiming that the former can be used as proxy for the latter [15,14]. However, our finding is consistent with the notion that people exhibit substantial eye gaze deviations from the head forward direction with large inter-individual differences regarding the eye-in-head motion range [18]. Moreover, it is noteworthy that contrary to controlled, experimental studies, we investigated positional tracking in an applied training scenario. Teaching a microlesson in the IVR environment entailed dynamic motion in terms of changing between different locations ("teleporting") and the handling of various interactive tools and objects.
Nevertheless, we could demonstrate the usefulness of positional tracking data in IVR. Although head gaze matched the eye gaze only poorly, submitting the positional data of the IVR headset and hand controllers to a Random Forest classifier, the model was able to predict the fixations of the eye tracking with high precision and recall. More specifically, the results indicated a .8637 ± .0020 F1-score for weighted-average of the random forest. Compared to the F1-score of .2867 ± .0017 for the baseline model with head gaze, this represents an improvement of .5770.
Despite the promising results regarding the usefulness of positional tracking data to predict actual eye gaze during teaching in IVR, it is important to discuss potential limitations of our approach to the data analysis. Although we trained and evaluated our classifier on different data samples, both datasets contained data from the same individuals. It is therefore conceivable that the resulting predictive performance is inflated, i.e., higher than if the model had been evaluated on new participants. This holds particularly true as people show significant interindividual differences in eye movement behavior [18], which would make the prediction of new participants' behavior challenging. However, it is also important to emphasize that the PSTs rotate roles in the teaching exercise. Therefore, it can be considered adequate to train an algorithm on the PSTs' data in a session when they are wearing an eye tracking enabled headset, and use that model to make inferences about their gaze in the other sessions.
Another potential limitation to note is that classical random forest classifiers do not generate high-quality models on correlated data [27]. This stems from the violated assumption of independent and identically distributed when dealing with longitudinal data. Therefore, a future direction of our research is to use a more computationally intensive model (e.g. a long short-term memory (LSTM) recurrent neural network) designed to handle time-series data.
Finally, we would like to point out that, to our knowledge, no established, independent estimates of the Meta Quest Pro's eye tracking performance exist to date. A preliminary study found an accuracy of 1.652º and a precision of 0.699º standard deviation, which is comparable to other IVR devices with integrated eye tracking [28]. However, the authors of the study also point out to be careful when interpreting fixation results. Their word of caution is related to the findings that the validity and reliability of eye tracking in IVR is influenced by many interacting factors, e.g. the placement of visual targets close or far from the periphery or vision correction [8]. Generally speaking, there is always a certain uncertainty involved in eye tracking research in the absence of an external reference measurement. This is not a specific limitation of this study but rather a general problem of eye tracking research.
For future direction, we are planning to corroborate and validate our findings by a) employing a more adequate machine learning model (see above) and b) investigating how well our results transfer from the teacher to the student role. For this purpose, we are planning to equip students with eye tracking enabled devices too. This would allow us to train and test an ML algorithm on different sessions of the same PST in different roles. Showing that the good predictive performance power of the positional data generalizes across different sessions could have farreaching practical implications for teacher education. It would equip PSTs and their educators with sophisticated, non-obtrusive ways to measure the attentional focus of teacher and students during a teaching exercise. For example, it could be used to make inferences about the PSTs' ability to distribute their attention to all students equally, and to make them aware of how their behavior compares to that of experienced teachers [13]. Generally, visualizing the attentional focus can greatly contribute to augmenting the feedback the PSTs receive from their peers and educators, therefore improving this central component of teacher training [20].
In this study, we investigated to what extent we can approximate the eye gazed objects by a) head gaze and b) a random forest classifier trained using the combination of position and rotation data. This is in the context of an IVR classroom of PSTs practicing their lesson. We found an added benefit of the machine learning model, which showed a good performance, opposed to using only the rather poor results of the pure head gaze approximation. These results are promising as they suggest that in some contexts, using cheaper devices might be sufficient to estimate the eye gaze of IVR users, and enable analytics possible currently only on expensive devices with eye tracking.
We would like to thank Lucas Martinic and Ferhan Özkan from XR Bootcamp GmbH for programming the VR application and Deian Popic for creating the 3D assets used in the study. The study was partially funded by a Higher Ed XR Innovation grant from the Tides Foundation.