The paper discusses AR/VR/MR/XR technologies in learning namely their influence/ opportunity and risks' mitigation. Main aspects are as follows: methodology (factors influencing a student's cybersickness in AR/VR/MR/XR, the improved model of the cognitive activity in synthetic learning environment). It has been developed the technique and ICT to study psychophysiological changes in normal and stressed conditions. The experimentation results demonstrated that decrease in myocardial tension index under cognitive performance conditions in immersive activity over time of observation was more significant and this fact could be accounted in measurement of influence of the synthetic environment on students, as well as the technique to measure AR/VR/MR influence. The technique proposed by the authors is based on modified ICT and used in previous research: to assess influence of AR/VR/MR/XR as changes of short cognitive/perceptual tests (3 minutes before the work and afterwards) with registration of physiological indices informative in our research.
in particular, and relevant digital competencies in general [
Purpose. To develop the model of risks related to AR/VR/MR/XR technologies in learning, as well as ICT structure and technique to study influence of these technologies at the learner performance and cybersickness. 2
The life cycle of technologies, innovative technological products and services will only accelerate. In such conditions it is natural that scientists connect their hopes for creation of the positive integrated reality under conditions of convergence of physical and virtual learning environments. In these conditions, the role of research and development in the field of educational applications of ICT, in particular in general secondary education, remains extremely important. Corporate learning has pioneered such applications, drawing on best practices in VR and AR, artificial intelligence, including the use of chat bots, knowledge bases, including video content creation, micro-learning and mobile learning. These processes push the evolution of tools, forms and methods of teaching in general education as well.
We adhere to the opinion that the potential of the information and educational environment
saturated with digital technologies, first of all, should be considered from the standpoint of the development
of cognitive activity of learning subjects. If students learn information images, including real natural
phenomena and processes, through experimentation with various digital tools and technologies
(simulations, computer simulations, virtual and augmented reality, etc.), it will provide creative activity in
an integrated (real and virtual) learning environment, will affect the cognitive motivation of students,
will contribute to the formation of appropriate digital competencies [
An important role in identifying areas for possible transformation of the education system is shared access to new digital technologies. According to world experts, the tools of virtual reality VR, augmented reality AR and augmented virtuality AV, mixed MR and extended reality XR (the latter includes all previous ones as well) are developing at the highest rate. The fourth industrial revolution is accompanied by the transition of production and, consequently, education in a synthetic environment of activity (both production and training). Education reform requires the accelerated implementation of augmented reality tools into the educational process, as well as the preparation of future employees to interact with artificial intelligence systems, as well as robotic systems.
The difference between these terms and means is that they are different combinations of real (RR) and virtual reality, forming variants of an immersive learning environment in which perception is the result of a synthesis of consciousness and feeling [15, p.16]. The most widespread over time from this spectrum are augmented, virtual, mixed and extended realities, for which the following types are distinguished:
supplemented - market-oriented AR, location-based AR, overlay-based AR, projection-based
AR; all these types can be used in the educational process [
Augmented reality includes the whole spectrum, from "complete real" to "complete virtual" in the
concept of the continuum of reality-virtuality, proposed by Paul Milgram. With the development of
technical means of virtualization of reality and means of registration of indicators of human sensory
systems and the impact on them there is a specialization and separation of new types of immersive
environments. For example, SR (substitutional reality), 360 virtual reality (or 360 VR, which allows to
observe the object from any angle), etc. It can be expected that the range of synthetic "realities" will
further expand, accounting the possible relationships of human sensory systems, cognitive models of
activity and real reality [
It is important that the synthetic environment is not natural for humans, and its impact on his/her mental and physiological processes remains insufficiently studied. Until now, among the factors influencing the synthetic environment with different forms of virtualization on a person and his health, it is proposed to distinguish the following:
1) personal: internal (inherent in man) - hereditary, gender, age, ethnic; physiological - interpupillary distance, flicker fusion frequency threshold, postural stability, strength and mobility of nervous processes, plasticity, cardiovascular system, vestibular apparatus; mental - consciousness, cognitive features, spatial operations, thinking; health - diseases, disorders of the visual system; 2) technological: optical factors; reflection factors; factors related to spatial tracking; sound factors; factors related to the form factor;
3) operational – adaptation, the degree of control, head movements, general visual (other sensors, in addition) flow, speed linear and rotational acceleration, speed of self-movement, density and height of the visual scene above the terrain, brightness level, vection (illusion of self-movement), duration, cognitive workload. The model of influencing factors demonstrates their systemic nature (Figure 1). adaptation the degree of control head movements general visual flow speed linear and rotational acceleration speed of self-movement density and height of the visual scene above the terrain brightness level vection duration cognitive workload
O p e r a t i o n a l optical factors reflection factors factors related to spatial tracking
sound factors factors related to the form factor
T e c h n o l o g i c al P e r s o n a l
hereditary gender age ethnic
Physiological interpupillary distance flicker fusion frequency threshold postural stability strength and mobility of nervous processes plasticity cardiovascular system vestibular apparatus
consciousness cognitive features spatial operations thinking
For learning, the most significant factors are that relate to the person features/abilities and operational ones.
There is increasing interest in determining any risks of using such technology and any aftereffect
from exposure. But the duration of the aftereffects from virtual environment (VE) exposure are not
well studied to date even after 20-years study [
But there is only a few studies that aimed to assess the physiological “cost” of cognitive work in
VR. The investigation of VR-induced aftereffects on various basic cognitive abilities and its
relationship with cybersickness. Previous studies suggest an adverse effect of VR exposure on simple reaction
times. Aftereffects on other basic cognitive abilities have rarely been studied. The authors propose a
general aftereffect of VR exposure on reaction times that is only slightly related to subjective degrees
of cybersickness. Taken together, however, usage of VR systems, even if inducing moderate levels of
cybersickness, leads only to minor decrements of cognitive performance [
VR technologies make learning more visual, enable trainees to be activated, and more fully engage
them in the learning process. These technologies facilitate and simplify the collaboration of people
who are at a distance, who can meet with the help of AR, prepare joint documents, lead projects and
perform many other work almost as effectively as with personal contact in real world. Teachers and
students have the opportunity to use virtual laboratories to study the world around them, develop
skills, as well as to demonstrate their development and automated assessment [
Cognitive activity in VR is the brightest manifestation of the immersive environment for education/training. Virtual Reality glasses, helmets, 360° panoramic cameras and screens, as well as CAVEs give new opportunity for learning, and have some specific features in comparison with more traditional digital devices: If in the real world the user interacts with the digital world through a "window“ (computers, tablets and mobile gadgets) observing what is happening "from the outside“, "immersion" is the condition in which the user loses awareness of the facts actually occurring in an artificial world. If in the real world the user involves almost only vision (other sensors are not involved or they supply non-observational information), in “immersion”, the user experiences the virtual world with involved his/her senses and is able to interact with the virtual environment. Sensory useful (target) load approaches 100% and requires 100% attention and concentration, regardless of the significance of the task. Sensory "hunger" (with monotonous operator activity) can be replaced by sensory exhaustion.
The question is what could be VR/AR/MR consequences for health? How safe are virtual reality technologies for human health? The long-term consequences of using these technical means are not yet clear. But it is already obvious that they are invading the work of the human body. And we are talking not only about the curvature of the spine due to the prolonged wearing of a heavy device on the head, but also about the impact on the user's eyes. The headset forms a wide field of image; it is a rather complex device that interferes with the normal operation of the visual apparatus. A systematic study of the impact of immersions in virtual reality on human health in general and on his mental health is still an open question [21]. Medical and physiological studies over the past quarter century have shown [22] that immersing a person in a specially designed virtual reality can significantly affect his/her mental health (treatment of depression, elimination of alcohol dependence and other mental disorders. However, all these studies are still quite fragmented, and the proposed methods require a qualified psychotherapist that is not applicable in education/training [23].
Further research of the problem should focus on the detailed development of types of threats to participants in the educational process, as well as methods of counteraction. A special point should be resistance to cyber-hazards, which can use the experience of training emergent industries’ operators primarily diagnosing the current state of the person and necessary adjustments in order to optimize its activities. 2.2
As it was stated [24], VR/AR/MR/XR are the brightest manifestation of the immersive environment for education/training, and have some specific features in comparison with more traditional networks/devices, and the object of activity (mental) is not external in relation to the human, but internal one [25]. The model of the cognitive work (P. Anokhin’s model of the activity functional system that has been developed by authors for cognitive activity) can be specified for the case of VR/AR, where activation of the sensory (affector) inputs (Figure 2) happen not so from the outer environment, as from the Virtual Act Program without activating the Act Program mechanism (as in physical activity).
In other words, the chain “Act acceptor – Act program – Act – Object – Result” in traditional activity transforms into the chain “Act acceptor – Virtual act program – Cognitive object – Cognitive result”. The most important feature of such a process is that all system elements and their interactions are part of the human organism, i.e., its internal environment, in contrast with physical (or miksted) activity, where its object is located outside the organism and activity takes place in whole or in part of the external environment. We believe, that this model can explain, why such regulation can deplete and imbalance the body: Act Program works in coordination with the Act Acceptor and Substratum needed for the normal life and activity. But lack of the signals from the Act Program cannot activate the general feedback from activity, only simulating it at the neurological level [25, p.352].
Another difference is influence of the Virtual act program on the Decision making block, because both blocks and the Cognitive objects are parts of the same cognititve process. Besides, the Virtual act program can influence on the Afferent inputs activating sensors participated in the particular cognitive activity according to the work task.
As a result, all neurophysiological subsystems (blocks in the schema) are more active the more immersive is the task performance. That is why the control of operational parameters in AR/VR/MR/XR is so important to mitigate cybersickness, especially in learning process that can take hours of a human activity, in general. 3
To date, it was confirmed that changes in efficiency of immersive cognitive activity with digital
units had strongly individual nature in day-to-day performance of the same complexity that could be
explained by the internal (physiological) and external influences on a human.This notion has been
confirmed by the high relationship between indicators of test performance (rate and reliability of
problem solving) and physiological indices. Only the simultaneous activation of both paths (energy and
information) provides higher level of optimization of educational/training capacity - quality education
[
It should be noted that the psychological/psychophysiological aspects of cybersecurity, inherent in the human factor in human-machine systems, can be significantly enhanced in the synthetic environment, which is currently an unexplored area in general and in education in particular [26]. On the other hand, the use of models for predicting the effectiveness of learning (included in adaptive systems) allows to assess and predict the effectiveness of augmented and virtual reality for synthetic learning environment [27] and human integration in general [28], epecially accounting modern trends in the ICT evolution [29]. To realize such an approach, the methodology described above was used as a basis to apply appropriate ICT that provides methodical maintenance of cybersickness risks’ measurement and assessment, test generation and control, data storing, data analysis and necessary service for researchers. 3.1
In our previous research the cybersickness has been modeling in experimental study of cognitive activity with measuring of its psychophysiological response to a human focused cognitive test performance (high motivated, without external disturbances) with and without time pressure [25]. In addition, the technique included measurement of electropuncture diagnostics’ indices by Nakatani (24 regular points and 3 stress’ points), as well measurement of lipid metabolism using sweat collection before and after the test session for each subject. Subjects included 28 males of 18-40 year old.
The experiments differed by the test (permutation of random non-repeating digits from 0 to 9 in ascending order) workload: training Е1 (60 min); Е5 - free rate ("«auto-pace"), and Е6 - fixed rate calculated as average by results of the appropriate test performance in Е1. The E6 could be evaluated as the model of immersive cognitive test performance (high motivation and time “pressure”, because according to our previous study the individually “average” rate of task exposure was more difficult than twice as slow or as fast). Duration of the test performance session was 3-hours long.
As indices of physiological “cost” of activity and the human state we registered a heart rate HR and blood pressure (systolic BPs, diastolic BPd) by means of the cardio-monitor "Solveig". The indices HR, BPs and BPd we registered during 10 min prior to the tests beginning (index “0”) and 10 min after finishing (relaxation), as well as every 5 min during the test activity [25, p.352]. Clear changes appeared in physiological response: increase of the level of low density lipids, energy balance (Electropuncture), heart rate (myocardial tension index after R. Baevsky) and blood pressure. The data received have demonstrated quite individual nature of changes in time.
To study that phenomenon, we carried out the similar experiments, but with another analysis of data stored. The method of computer data processing was aimed to analyse physiological changes at different phases of test performance that would correspond phases of a human performance after Egorov&Zagriadskii. Because diastolic blood pressure has been revealed as the most informative (sensitive to the cognitive workload) physiological index, BPd was averaged for consequent 20-minutes intervals and was represented at the “phase plane” (vector diagram where one point corresponded to one 20-minutes interval). Vizualization of the physiological changes on the phase plane (Figure 3) has confirmed that first 20…40 minutes of tests performance are accompanied by the higher systolic pressure and/or at the end of the performance (this corresponds the phase of the “end gust effect” after Egorov&Zagriadskii). But the first phase could have individual features from viewpoint of time structure, if to analyze it in more detailed way (f.e., minut-by-minut or in 5-minute intervals) that could be significant accounting that VR/AR activity longer 40 minutes were not studied, and “normal” academic hour is 45 minutes.
The next step of analysis was carried out to study myocardial tension index’s variation (after R. Baevsky) at the 1st 45-minutes interval of test performance (that could be considered as equivalent of the academic hour) with 5-minut consequent phases. We compared physiological workload in experiments E5 and E6 again. If in previous analysis with 20-minutes phases some individual differences were registered, it has been revealed a clear trend in subjects’ myocardial tension between experiments E5 and E6: the higher level of tension under “harder” conditions (experiment with time “pressure”) during first 20 minutes (four phases) of test performance (Figure 4).
In other words, decrease in myocardial tension index under cognitive performance conditions in immersive activity over time of observation was more significant and this fact could be accounted in measurement of influence of the synthetic environment on students.
The main difficulty in measuring influence of AR/VR/MR/XR on a human cognitive performance is to differentiate psychophysiological effects of usual cognitive workload and synthetic “realities”, especially in the education process. This is a general problem of assessment of Associations Between Digital Technology Engagement and Mental Health/efficiency Problems. Even more, according to the study [30], there is no evidence that associations between adolescents’ digital technology engagement and mental health problems have increased over. These results are based on observation of 430,561 participants (adolescents) over period 1991-2019 in the United States and United Kingdom. But that study did not take into account that everyday environment of todays’ young people is digital with related specifics. That is why measurement of possible influence of the synthetic environment should be provided by means the same or similar digital tools.
The technique proposed by the article authors is based on modified ICT described above and used in our research. But we plan to assess influence of AR/VR/MR/XR as changes of short cognitive/perceptual tests (3 minutes before the work and afterwords) with registration of physiological indices informative in our research.
Thest performance is controlled by appropriate ICT in on-line or off-line modes. 4
Augmented, virtual, mixed and extended realities become the part of a human everyday life in all areas of life and activity. It is important that the synthetic environment is not natural for humans, and its impact on his/her mental and physiological processes remains insufficiently studied. Factors influencing a student’s cybersickness in AR/VR/MR/XR can be considered as: personal (internal, inherent in human; physiological; mental; health); technological (technical means, ergonomic); operational (adaptation, the degree of control, head movements, general visual flow, linear and rotational acceleration, speed of self-movement, brightness level, vection (illusion of self-movement, duration, cognitive workload).
It has been developed the theoretical scheme of the functional system of learning activity as a further development of the TFS model for learning. It was highlighted that all neurophysiological subsystems are more active the more immersive is the task performance. That was why the control of operational parameters in AR/VR/MR/XR was so important to mitigate cybersickness, especially in learning process that could take hours of a human activity, in general.
Decrease in physiological indices (myocardial tension index and blood pressure) under cognitive performance conditions has been revealed in immersive activity over time of observation, and it was more significant than in “normal” conditions. Those results were confirmed by use of the technique “phase plane” proposed by authors.
That fact could be accounted in measurement of influence of the synthetic environment on students. Application of developed ICT provides methodical maintenance of cybersickness risks’ measurement and assessment, test generation and control, data storing, data analysis and necessary service for researchers. Respectively, it was developed the technique to assess influence of AR/VR/MR/XR as changes of short cognitive/perceptual tests (3 minutes before the work and afterwords) with registration of informative physiological indices. Future work is planned to provide evidence of such a technique efficiency for learning. 5 [21] R. Mukamal, Are virtual reality headsets safe for eyes?, 2017. URL: https://www.aao.org/eyehealth/tips-prevention/arevirtual-realityheadsets-safe-eyes. [22] D. Freeman, S. Reeve, A. Robinson, A. Ehlers, Virtual reality in the assessment, understanding, and treatment of mental health disorders, Psychological Medicine, Vol. 47, Iss. 14, 2017, 2393-2400. https://doi.org/10.1017/ S003329171700040X. [23] A. Yu. Uvarov, Virtual Reality Technologies in Education, Science and School, № 4, 2018, 198-117. [24] G. Riva, S. Wiederhold, A. Chirico, D. Di Lernia, F. Mantovani, A. Gaggioli, Brain and virtual reality: What do they have in common and how to exploit their potential, Annual Review of Cybertherapy and Telemedicine, 16, 2018, 3–7. [25] O. Pinchuk, O. Burov, S. Ahadzhanova, V. Logvinenko, Y. Dolgikh, T. Kharchenko, A. Shabalin, VR in Education: Ergonomic Features and Cybersickness, in: S. Nazir S., T. Ahram, W. Karwowski (eds.), Advances in Human Factors in Training, Education, and Learning Sciences, AHFE 2020, Advances in Intelligent Systems and Computing, vol. 1211, Springer, Cham, 2020, 350-355. https://doi.org/10.1007/978-3-030-50896-8_50. [26] V. Y. Bykov, O. Y. Burov, N. P. Dementievska, Cybersecurity in digital educational environment, Inf. Technol. Learn.
Tools, 70(2), 2019, 313-331. https://doi.org/10.33407/itlt.v82i2. [27] O. Spirin, O. Burov, Models and applied tools for prediction of student ability to effective learningin: 14th International Conference on ICT in Education, Research and Industrial Applications. Integration, Harmonization and Knowledge Transfer, CEUR-WS, V. 2104, 2018, 404-411. [28] E. Lavrov, N. Pasko, O. Siryk, O. Burov, M. Natalia, Mathematical Models for Reducing Functional Networks to Ensure the Reliability and Cybersecurity of Ergatic Control Systems, in: Proceedings 15th International Conference on Advanced Trends in Radioelectronics, Telecommunications and Computer Engineering, TCSET 2020, 2002, 179–184 . [29] O. Burov, V. Bykov, S. Lytvynova, ICT evolution: from single computational tasks to modeling of life, in: Proceedings of the 16th International Conference on ICT in Education, Research and Industrial Applications, Integration, Harmonization and Knowledge Transfer, Vol-2393, 2020, pp. 170-177. http://ceur-ws.org/Vol-2393/paper_353.pdf. [30] M. Vuorre, A. Orben, Andrew K. Przybylski, There Is No Evidence That Associations Between Adolescents’ Digital Technology Engagement and Mental Health Problems Have Increased, Clinical Psychological Science, First Published 3 May 2021. https://doi.org/10.1177/2167702621994549.