Estimation of Dificulty When Reading VR-based Educational Comics Using Gaze, Facial Movement, Heart Rate, and Electroencephalography⋆

Abstract

Using the metaverse in education can improve motivation, presence, engagement, immersion, interest, and performance. However, automated recognition of progress or learning dificulties is still a challenge. In this study, we search for relevant biometric features that can be used to estimate the user's self-reported dificulty level of educational comic books (manga) materials in Virtual Reality (VR). Educational manga has been an efective learning tool for a wide range of fields. It has unique spatial features that are unavailable in traditional approaches such as browsing history or log data analysis. Our approach uses facial expressions, electrocardiogram (ECG), and electroencephalogram (EEG) data, in addition to eye gaze, which has been used previously, to estimate dificulty when reading VR-based educational manga. We measured learners' data while reading manga materials in a VR space and estimated the learner's self-reported dificulty at two levels (challenging and accessible) using Support Vector Machine (SVM) and Random Forest (RF) algorithms. As a result, the RF approach using facial expressions and ECG features achieved an accuracy of 0.97 and an F1 score of 0.94 on the two-level dificulty estimation task.

Keywords

• eol>Dificulty Estimation
• Biosignals
• Virtual Reality
• Afective Computing

1. Introduction

3. Data set

vtiiosnuahleelpxsplaosrsaetsisolneaarnnderrse’acdoimngpr[e1h0e]n.sEioyne ignazeeduincafotiromnaal- $/01 '()*+" %2!"#'$()%&*'+#()#*+%2 '()* , '()*+" ,&&-'+(#)+*%-+#&()#*+ '()* , smeatttiinognsto[1e1s]t.imSaaktea mthoetodificeutlatyl. luevseedloefyree-atrdainckgimnga ningfaor- - materials in a VR environment [ 8 ]. .

Facial expression information has been used to eval- ! "& " & $ " & $ # # # % uate learner engagement using machine learning techniques [ 12 ]. Nakamura et al. estimated the dificulty of an Figure 1: The experiment consists of three phases: baseline, English word test using a combination of facial features reading, and annotation. Note that the manga sample in this and system operation logs [ 13 ]. figure is a dummy; it was not used in the actual experiment.

Heart rate information obtained from ECG has been used as a physiological method for internal boundary conditions [ 9, 14, 15 ]. Heart rate increases with task perceived dificulty level of each panel in the annotation demands and memory load [ 9 ]. HRV, the variability of the phase.

RR interval of the heartbeat, is used to assess stress [ 16 ].

RMSSD of HRV, unafected by respiration [ 17 ], correlates 3.3. Procedure with parasympathetic activity and stress. The 20th and 80th percentiles of the HRV data are used to estimate Participants were briefed on the experiment’s purpose, the state of stress and relaxation. pNN20 and pNN50 the VR environment, and the data collection process. reflect vagus nerve [ 17 ]. These indices are often adequate They were informed that eye gaze, facial expression, for estimating a learner’s self-reported dificulty when heart rate, and EEG information would be recorded durreading manga materials in a VR environment. ing the learning process. A questionnaire was adminis

Previous studies on EEG-based cognitive load as- tered to determine the participants’ prior knowledge of sessment have focused on and frequency bands. immunology and VR experience. Afterward, participants wave activity is linked to frontal lobe activity and put on the HMD and practiced operating the interface working memory capacity, while wave activity is re- in the VR environment using manga materials diferent lated to cognitive attention, arousal, and memory capac- from those used in the experiment. ity [ 18, 19 ]. Studies measuring EEG in VR environments Next, the participants wore the ECG, followed by the have reported increased cognitive load in such environ- EEG. Gel was used to adjust the impedance to 30 kΩ ments [ 20, 19 ]. or lower. Afterward, the participants were asked to put on the HMD again and confirm that eye gaze and face tracking were working correctly. 3.4. Apparatus

We conducted a data collection experiment in a VR space using educational manga about immunology to develop a model for dificulty estimation.

We used an HTC VIVE Pro Eye and an HP VR Backpack

G2 with an i7-8850H processor and NVIDIA GeForce 2080 GDDR6 8GB in the experiment (Fig. 2). The software 3.1. Participants consisted of Unity version 2019.2.3.f1. The HMD’s eyeNine students (eight males, one female, 22-24 years old, tracker and SRanipal 1.3.2.0 SDK were used to collect eyemean 23 years old, standard deviation 0.3) with no prior tracking information. The VIVE Facial Tracker collected knowledge of immunology were recruited. Two partici- facial expression information as 38 diferent blend shapes, pants with missing data were excluded from the analysis. including lips, chin, and cheeks, at 60 Hz with a response This study was approved by our institution’s IRB. time of 10 ms. Shimmer31 was used to collect ECG. Two Shimmer terminals were used for the measurement and synchronization with Unity. An EEGo sports2, a portable 3.2. Task mobile EEG machine, was used for the EEG measurement. The 32 wet electrodes of the EEG cap were configured in the 10-20 system. EEG data were recorded via the EEGo sports software.

There were three phases: baseline, reading, and annotation (see Fig 1). Initially, a dark screen was displayed for three minutes to establish a baseline, during which the participants were asked to relax. Then, they learned the contents by reading the immunology manga material in a VR space during the reading phase. After that, they went through the same material and labeled the

1https://shimmersensing.com/product/shimmer3-ecg-unit-2/ 2https://www.ant-neuro.com/products/eego_sports

VIVE Facial Tracker were taken with electrodes attached following the standard five-lead placement method. The electrodes used for the EEG measurements were Fz, F3, F4, Cz, C3, C4, POz, Pz, P3, P4, M1, and M2, arranged according to the international 10-20 method. After the caps were attached, the gel was inserted, and the impedance was confirmed to be less than 30 kΩ before measurement began.

4. Analysis

Shimmer

Features were extracted from sliding windows separately

Figure 2: Image showing an experiment setup. Participants per modality [ 23 ]. The window slid in 0.5-second steps wore the VIVE Pro Eye, the VIVE Facial Tracker, the EEG Cap, for all modalities. and Shimmer3. Facial Expression Features. BlendShapes associated with face region movements were used. The resting state was set to zero. We calculated the minimum, average, 3.5. Stimuli and maximum values for each five-second window by summing the values of 38 BlendShapes. We also calcuWe used a manga entitled “Understanding Immunol- lated the change in the total value and used the minimum, ogy through Manga”, published by Ohmsha [ 21 ], which average, and maximum values as the feature values. The granted permission for us to use the manga material. reason for using this method is that learners are likely to Ohmsha’s Manga “de Wakaru series” is widely recog- make lip thrusting or mouthing movements when readnized for its easy-to-understand explanations and manga ing a dificult passage, even if they do not have a visibly introducing various fields such as science, electricity/- sad or frustrated expression. electronics, machinery, and architecture. Chapters 1-3 ECG Features. NeuroKit2 [ 24 ] was used to read the (76 pages) were used in this experiment. ECG data, clean the signal, and perform peak detection. The window size was 20 seconds. Features were output 3.6. Measurements for each window. Based on previous work, the features used were mean heart rate, HRV minimum, maximum, 3.6.1. Dificulty Annotations mean, standard deviation, RMSSD, 20th and 80th perThe time participants spent per panel within one page centiles, pNN20, and pNN50. was not constant due to the diverse dificulty levels of the EEG Features. The EEGo sports has an output of panels. Thus, we asked them to annotate their perceived 0.5∼ 30Hz band-pass filter. The output data included dificulty of each panel at three levels: “easy,” “dificult but Trigger information sent from Unity. Then, the noise understandable,” and “dificult and not understandable.” was removed using Independent Component Analysis They could record the dificulty levels using a ray-casting (ICA) in MNE-Python [25], and Morlet wavelets were interaction to mark dificult vignettes while reading. To used to perform time-frequency analysis at 3∼ 13Hz. The avoid diferences in the definitions of each label among average frequency power of the wave (4∼ 8Hz) and the participants, before the experiment, participants were the wave (8∼ 12Hz) were calculated. The window size asked to rate a panel as “dificult but understandable” if was then set to two seconds, and the power of the they understood the manga after reading it; and “dificult and waves within the window was calculated. We but not understandable” if they did not understand the hypothesize that activity is higher and activity is manga contents. If they were unsure, the participants lower when the learner perceives dificulty. were asked to rate it as “dificult but understandable.” Classification. To validate the proposed method, This paper reports the results of estimating the user’s we constructed a model with two classes, “easy” and self-reported dificulty based on the “easy” and “other” “dificult,” employing Support Vector Machine (SVM) classifications. After each chapter, the NASA TLX Ques- and Random Forest (RF) algorithms. We fitted subjecttionnaire was used to assess general workload. independent models using leave-one-subject-out crossvalidation. Sixteen diferent combinations were com3.6.2. Physiological Data pared in the subject-independent model, each consisting of either the presence or absence of eye gaze, facial exThirty-eight diferent BlendShapes corresponding to the pression, electrocardiogram, and EEG features. Accuracy, lips, chin, cheeks, and other body parts were collected for Precision, Recall, and F1 scores were used to evaluate facial expression information [ 22 ]. ECG measurements performance.

5. Results

Self-reported workload. Table 1 shows the results of each questionnaire, including the dificulty reported in each chapter. For simplicity, the Likert scale is considered an interval scale. It was observed that the dificulty level of Chapter 1 was the lowest. The dificulty level of Chapters 2 and 3 increased because the chapters contained more content and specialized vocabulary as they progress. The level of fatigue was also rated higher in the last chapters.

Number of Labels for Learner’s self-reported Evaluation. Table 2 shows the percentage of the learner’s self-reported evaluation labels per panel and per window of the collected manga teaching materials. While only 8.12% of the panels were labeled as “dificult,” this proportion increased to 30.0% when the time unit was taken into account. This suggests that participants spent more time on panels that were perceived as “dificult.”

Accuracy of Learner’s Self-reported Dificulty Estimation. When all features were used for each participant, the results were as follows: accuracy of 77%, F1 score of 77% for SVM, accuracy of 94%, and F1 score of 94% for RF. All combinations of biometric information were analyzed to identify the most relevant features for dificulty estimation. Accuracy, Precision, Recall, and F1 scores are shown in Table 3. Accuracy, F1 score, and Recall were 96.5%, 94.0%, and 91.2%, respectively, when facial expression and electrocardiograms were combined, and precision was 97.8%, the highest accuracy when gaze, facial expression, and electrocardiograms were combined.

6. Discussion and Conclusion

We investigated eye gaze, facial expression, electrocar

diogram, and electroencephalogram (EEG) features to estimate the learner’s self-reported dificulty level in reading educational manga. We collected data while reading educational manga in a VR environment and evaluated the estimation accuracy using time-based SVM and RF. Accuracy and F1 scores reached 0.965 and 0.940, respectively, using the RF approach with facial expressions and electrocardiogram features. The estimation accuracy was improved when the proposed method was used for facial expressions and electrocardiograms and decreased when EEG features were used.

Compared to the results obtained by estimating gaze features alone, the combination of gaze and facial expression and gaze and electrocardiogram improved the accuracy of dificulty estimation and F1 score. However, both estimation accuracy and F1 score were lower for the combination of gaze and EEG. Estimation accuracy and F1 score were higher when all features were used except EEG features. This is probably because the EEG signalto-noise ratio was low when simultaneously wearing an HMD. Future work should improve head-movementrelated noise reduction on the EEG. Combining facial expression and ECG features produced the highest estimation accuracy and F1 score. This indicates that the ECG Numbers in parentheses denote standard deviation (SD). ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

Acc. features are important for estimating dificulty. These two features can be collected more quickly than others and are independent of learning materials. Therefore, they can be easily applied to adaptive learning systems in VR environments, not limited to educational manga materials.

NeuroKit2: A python toolbox for neurophysiologi

cal signal processing, Behavior Research Methods 53 (2021) 1689–1696. [25] A. Gramfort, M. Luessi, E. Larson, D. A. Engemann, D. Strohmeier, C. Brodbeck, R. Goj, M. Jas, T. Brooks, L. Parkkonen, M. S. Hämäläinen, MEG and EEG data analysis with MNE-Python, Frontiers in Neuroscience 7 (2013) 1–13.

References

[1] X. Chen , D. Zou , H. Xie , F. L. Wang , Metaverse in education: Contributors, cooperations, and research themes, IEEE Transactions on Learning Technologies ( 2023 ).

[2] M. Wang , H. Yu , Z. Bell , X. Chu , Constructing an edu-metaverse ecosystem: A new and innovative framework , IEEE Transactions on Learning Technologies 15 ( 2022 ) 685 - 696 .

[3] R. Heradio , L. De La Torre , D. Galan , F. J. Cabrerizo , E. Herrera-Viedma , S. Dormido , Virtual and remote labs in education: A bibliometric analysis , Computers & Education 98 ( 2016 ) 14 - 38 .

[4] R. Heradio , L. de la Torre, S. Dormido, Virtual and remote labs in control education: A survey , Annual Reviews in Control 42 ( 2016 ) 1 - 10 .

[5] L. A. Mamolo , Development of digital interactive math comics (dimac) for senior high school students in general mathematics, Cogent Education 6 ( 2019 ) 1689639 .

[6] I. Fuse , S. Okabe , Computer ethics education using video and manga teaching materials: learning efects and the order of using teaching materials , Transactions of Japanese Society for Information and Systems in Education 27 ( 2010 ) 327 - 336 .

[7] P. García , A. Amandi , S. Schiafino , M. Campo , Evaluating bayesian networks' precision for detecting students' learning styles , Computers & Education 49 ( 2007 ) 794 - 808 .

[8] K. Sakamoto , S. Shirai , N. Takemura , J. Orlosky , H. Nagataki , M. Ueda , Y. Uranishi , H. Takemura , Subjective dificulty estimation of educational comics using gaze features , IEICE Transactions on Information and Systems E106.D ( 2023 ) 1038 - 1048 .

[9] R. L. Charles , J. Nixon , Measuring mental workload using physiological measures: A systematic review , Applied ergonomics 74 ( 2019 ) 221 - 232 .

[10] S. P. Marshall , The index of cognitive activity: Measuring cognitive workload , in: Proceedings of the IEEE 7th conference on Human Factors and Power Plants , IEEE, 2002 , pp. 7 - 7 .

[11] C. Lima Sanches , O. Augereau , K. Kise , Estimation of reading subjective understanding based on eye gaze analysis , PLOS ONE 13 ( 2018 ) 1 - 16 .

[12] J. Whitehill , Z. Serpell , Y. C. Lin , A. Foster , J. R. Movellan , The faces of engagement: Automatic recognition of student engagement from facial expressions , IEEE Transactions on Afective Computing 5 ( 2014 ) 86 - 98 .

[13] K. Nakamura , K. Kakusho , M. Murakami , M. Minoh , Estimating learners' subjective impressions of the dificulty of course materials by observing their faces in e-learning, The IEICE transactions on information and systems (D) 93 ( 2010 ) 568 - 578 (in Japanese).

[14] H.-G. Kim , E.-J. Cheon , D.-S. Bai, Y. H. Lee , B.- H. Koo , Stress and heart rate variability: a metaanalysis and review of the literature , Psychiatry investigation 15 ( 2018 ) 235 .

[15] N. Meshkati , Heart rate variability and mental workload assessment , in: Advances in psychology, volume 52 , Elsevier , 1988 , pp. 101 - 115 .

[16] S. Delliaux , A. Delaforge , J.-C. Deharo , G. Chaumet, Mental workload alters heart rate variability, lowering non-linear dynamics , Frontiers in physiology 10 ( 2019 ) 565 .

[17] S. Laborde , E. Mosley , J. F. Thayer , Heart rate variability and cardiac vagal tone in psychophysiological research-recommendations for experiment planning, data analysis, and data reporting , Frontiers in psychology 8 ( 2017 ) 213 .

[18] L. J. Castro-Meneses , J.-L. Kruger , S. Doherty , Validating theta power as an objective measure of cognitive load in educational video , Educational Technology Research and Development 68 ( 2020 ) 181 - 202 .

[19] S. Baceviciute , A. Mottelson , T. Terkildsen , G. Makransky, Investigating representation of text and audio in educational vr using learning outcomes and eeg , in: Proceedings of the 2020 CHI conference on human factors in computing systems , 2020 , pp. 1 - 13 .

[20] G. Makransky , T. S. Terkildsen , R. E. Mayer , Adding immersive virtual reality to a science lab simulation causes more presence but less learning , Learning and instruction 60 ( 2019 ) 225 - 236 .

[21] H. Kawamoto , The Manga Guide to Immunology, Ohmsha, 2014 .

[22] P. Joshi , W. C. Tien , M. Desbrun , F. Pighin , Learning controls for blend shape based realistic facial animation , in: ACM Siggraph 2006 Courses , 2006 , pp. 17 - es .

[23] F. Dollack , K. Kiyokawa , H. Liu, M. PerusquiaHernandez, C. Raman, H. Uchiyama , X. Wei , Ensemble Learning to Assess Dynamics of Afective Experience Ratings and Physiological Change , in: 2023 11th International Conference on Afective Computing and Intelligent Interaction Workshops and Demos (ACIIW) , 2023 , pp. 1 - 8 .

[24] D. Makowski , T. Pham , Z. J. Lau , J. C. Brammer , F. Lespinasse , H. Pham , C. Schölzel , S. H. A. Chen ,