Conceptualising immersive multimodal environments for psychomotor skills training. Khaleel Asyraaf Mat Sanusia, Deniz Irenb and Roland Klemkea,b a Cologne Game Lab, TH Köln, Cologne, Germany b Open Universiteit, Heerlen, The Netherlands Abstract The coordination of psychomotor skills requires deliberate practice and techniques, all of which are typically taught in a physical setting, where instructions and timely feedback are given by the teachers. However, doing so remotely is commonly inefficient and ineffective, therefore, hindering the learner's progress. Sensors and immersive technologies enable the collection of multimodal data and the creation of immersion, respectively. These technologies have been widely used to further improve the learning outcome, especially in the psychomotor domain. In this paper, we present our research on designing an immersive training environment for remote psychomotor skill training and investigating how such an environment can be used for training skills in different psychomotor domains. Keywords 1 Immersive technologies, Sensors, Multimodal, Psychomotor skills 1. Introduction in a correct manner, which include safety and effectiveness. Timely and consistent feedback from the teacher is essential for the learner to The global pandemic event of Covid-19 has avoid developing improper techniques during affected various learning and teaching activities training, thus ensuring the desired goal can be acutely. This necessitates the notion of online achieved in a shorter time [2]. However, doing learning or e-learning in which web so in a remote manner makes the learning conferencing tools (e.g., Zoom, Teams) are process ineffective and inefficient due to the widely utilised by teachers and students for lack of modalities such as haptic feedback or classroom activities. However, this is rarely the 3D full-body perception, hence impeding the case for psychomotor skills development as learner’s progress. Due to this, psychomotor they require hands-on practice. Psychomotor skill learners and teachers have been skills need to be physically executed, in most substantially affected. cases, repetitively to the extent that the muscle Nowadays, educational technology and memory is trained, which will automate the artificial intelligence (AI) researchers are muscle movements [1]. Furthermore, the progressively embedding sensor technologies presence of teachers is needed in order to for the collection of multimodal data, and explain, demonstrate, and assess certain machine learning approaches for tracking procedures. To achieve this, the human learning learners' behaviour and progress in authentic model has to be in a structured form where learning contexts. The combination of these instructions are well-defined, and feedback can technologies introduces new technological be given to ensure that the tasks are performed Proceedings of the Doctoral Consortium of Sixteenth European Conference on Technology Enhanced Learning, September 20–21, 2021, Bolzano, Italy (online). EMAIL: ks@colognegamelab.de (A. 1); deniz.iren@ou.nl (A. 2); rk@colognegamelab.com (A. 3) ORCID: 0000-0001-6766-4416 (A. 1); 0000-0002-0727-3445 (A. 2); 0000-0002-9268-3229 (A. 3) ©️ 2021 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings (CEUR-WS.org) affordances that can be leveraged in the immersive technologies and sensor psychomotor education, especially in a remote technologies in a multi-sensor setup for manner, to further improve the learning collecting multimodal data and giving outcome. immediate feedback in an immersive training Multimodality is a theoretical assumption environment in which learners can use to that can be applied to provide more structure in improve their psychomotor skills sensors for exploring learning. The general idea independently. of multimodality in learning comes from the The paper is structured as follows. In theory of embodied communication. Based on Section 2, we present related studies that utilise this theory, humans use their whole bodies to sensors for the collection of multimodal data communicate with each other, applying various and immersive training environments for channels to exchange messages such as immersion, and to what extent they are used in gestures, facial expressions, prosody, etc. [3]. the psychomotor domain. Next, we explain our Subsequently, the trend of multimodality has research questions in Section 3. Subsequently, been employed in human-computer interaction. in Section 4, we visualise and describe the Sensor-based multimodal interfaces allow the research model and methods of this study. monitoring of different modalities and have Finally, in Section 5 we discuss the expected been applied in various domains to improve outcomes of our study in theoretical and learning [[4], [5], [6]]. practical implications, followed by the While the multimodal approach helps to conclusion. improve the psychomotor learning outcome, designing virtual training environments adds 2. Related work immersion to the learning activity. As such, immersive learning technologies such as virtual 2.1. Sensors in psychomotor reality (VR), augmented reality (AR) and game elements enable the creation of virtual training learning environments or simulations that typically consist of nearly, if not entirely, realistic Sensor technologies are increasingly physical similarity to an actual learning context. becoming more portable and increasingly used Herrington et al. [7] stress that the learning in psychomotor training, enabling efficient environment and designated tasks create the methods for the acquisition of performance conditions for the “True” immersion. Hence, it data, which allows effective monitoring and can be argued that the instructions and feedback intervention. That being said, such devices have provided by the learning environment should be been explored to provide support in the learning pragmatic for the learners to learn to perform domain. For example, Schneider et al. [8] the tasks in a correct manner; for example, analysed 82 prototypes found in literature personalised feedback (human-teacher-like) to studies based on Bloom’s taxonomy of learning create immersive learning experiences. domains (psychomotor, cognitive, and Intrinsically, virtual training environments affective). Their research suggests researchers allow the learner to actively interact with the in- and educators to consider utilising sensor-based game objects which may create more platforms as reliable learning tools for reducing engagement and increase motivation for the the workload of teachers and, therefore, learner when performing tasks. contribute to the solution of many current In this research, we aim to design and educational challenges. implement an immersive training environment Motion sensors such as accelerometers and for psychomotor skills using immersive gyroscopes are predominantly used to acquire technologies which will be integrated with motion data to recognize human activities, sensor technologies and AI, in order to deliver especially in the psychomotor domain. These instructions and feedback to learners in a sensors are commonly combined and used in a meaningful manner. Furthermore, we intend to synchronized manner to achieve a higher investigate the effectiveness of the system and accuracy of detecting not only simple but whether it can be applied to train skills in complex activities as well [9]. This enables the different psychomotor domains. The collection of multimodal data and provides a development of this system provides an early more accurate representation of the learning and significant step towards combining process [10]. Furthermore, multimodal data can be collected using various sensors such as development in recent years, such technologies wearable sensors, depth camera sensors, are being used in various psychomotor Internet of Things devices, etc. domains, including sports, physical training, For instance, Schneider et al. [4] designed a rehabilitation therapy, and much more. These system to support the development of public technologies transport individuals into an speaking skills using the Kinect v2 depth interactive training or learning environment, camera sensor to track the skeletal joints of the either virtually or physically, to replicate the learner's body and the HoloLens headset to authentic learning context of a specific skill. provide feedback in real-time when mistakes For example, Song et al. [12] designed and are detected while presenting. Limbu et al. [11] implemented an immersive VR environment developed a system to teach basic calligraphy for teaching tennis using high-definition skills, which uses the pen sensor in Microsoft stereoscopic display, robust and accurate Surface tablet and EMG sensors in a Myo hybrid sensor tracking, shader-based skin armband to provide feedback to learners during deformation, intelligent animation control, and practice. It also allows the calligraphy teacher haptic feedback mechanism. The authors to create an expert model, which the learners reported that, through these technologies, a can later use to practice and receive guidance real-time immersive tennis playing experience and feedback based on the expert model. is achieved. Potentially, the system can be To better understand learners’ performance, scaled to adapt various application cases such educational researchers are progressively using as other sports game simulations and even machine learning approaches to classify military training simulations. activities based on the multimodal data Ali et al. [13] experimented with multiple collected. For instance, in the medical domain, VR fitness applications (e.g., VR Fitness, Di Mitri et al. [5] investigated how multimodal VirZOOM, BOXVR) for physical training such data and Neural Networks can be used for as walking, running, and jogging. In addition, learning Cardiopulmonary Resuscitation skills they implemented a mobile application that by utilising a multi-sensor system comprising uses built-in sensors such as an accelerometer of a Kinect v2 and a Myo armband. In the sports and gyroscope for motion detection. As a result, domain, Mat Sanusi et al. [6] applied the same they achieved up to 82.46% of accuracy and framework as the previous author by using thus, described the effectiveness of VR built-in accelerometer and gyroscope sensors in technology in physical training, which is a smartphone and also a Kinect v2 to detect helpful for the development of psychomotor forehand table tennis strokes during training. skills. Both study results show a high classification In our research, we aim to incorporate rate of the activities when combining the immersive technologies into the mix for the sensors, emphasising the importance of a creation of immersive training environments to multimodal approach in classifying complex enhance the immersive experience of the activities. learner in the learning setting. Our grand vision In this research, we aim to use a multi- is to have a theoretical framework with a sensory system (e.g., wearable technologies, structured human learning model (feedback and depth cameras) with the help of machine instructions) within these immersive training learning to help learners improve their environments that can be applied to not only psychomotor skills. We intend to have a one but also multiple psychomotor domains. theoretical framework that can be used for training skills in one psychomotor domain and 3. Research questions subsequently applied in multiple domains. Based on the problem identified and the 2.2. Immersive training related work analysed, we aim to investigate the environments following research questions (RQs): Immersive learning technologies such as VR 1. What level of technological support and AR are progressively becoming a (technology) is available in the significant medium for psychomotor training. literature and appropriate for delivering Due to the substantial improvement and effective instructions and feedback (pedagogy) to the learners in 4. Methodology psychomotor training? 4.1. Research methods Fundamentally, it is crucial to identify the most promising pedagogical approaches in It is essential for this research to follow a psychomotor skills learning that can be applied methodological approach for designing, in multiple psychomotor domains. developing, testing, and evaluating such a Furthermore, with technologies that have been system. Hence, we conduct our research based widely used to improve the learning outcome in on the Design-based Research (DBR) recent years, we survey the state-of-the-art of approach, a common iterative methodological technology that may potentially be helpful for approach for prototypical solutions. In the our research. Therefore, a systematic review context of our research, we combined two DBR will be carried out for these two processes and models from Amiel & Reeves [14], and De thus, answer our RQ1. The outcome of Vielliers & Harpur [15], which are used in the answering this question would be the domain of educational technologies. theoretical framework of the system. Figure 1 shows the phases of the DBR approach for this research, and the following 2. How can we create an immersive and subsections explain each of the phases. information-rich (remote/self-learning) training environment for psychomotor 1. Problem analysis: In the first phase, a skills that deliver effective instructions systematic literature will be reviewed to and meaningful feedback to the determine the importance of the problem and learner? identify the current theory on the immersive multimodal environments in the psychomotor Subsequently, we design and implement a domain. Furthermore, the selection of virtual training environment based on the application cases will be made in this phase. theoretical framework retrieved from RQ1. The With these approaches, we are analysing the instruction and feedback systems should be problem and defining research goals. The given in a realistic manner to create immersive outcome of this step is a detailed research learning experiences. Therefore, it is vital to proposal containing goals and evaluation research how can we maximise the system’s criteria. effectiveness in providing feedback and instructions. This includes the framing of 2. Design solution: A theoretical framework interaction and the appropriate modalities for is proposed based on the results from the instructions and feedback. Consequently, we systematic review, identifying the most can investigate the effectiveness of the system: promising pedagogical model in psychomotor can the system help learners improve their training and the technologies that can be skills during training? contributed to such a model. Our conceptual model (see Figure 2) states how we transfer the 3. To what extent can we generalise our theoretical framework into our system design, training framework to multiple suggesting to address the problem from phase psychomotor domains? 1. Finally, we explore if the system can, both 3. Develop solution: The next phase is the theoretically and practically, be adapted and implementation of the immersive training applicable in multiple psychomotor domains. environment that serves the research purpose. More exercise routines and common mistakes The development of the system is based on the of the selected applications cases will be theoretical framework proposed in phase 2. The identified to suit the system's needs. Hence, it is outcome is an innovative and functional crucial to know what are other possible immersive training environment system with application cases that the new system can be the integration of immersive technologies, used to train related psychomotor tasks and can sensor technologies, and AI that aims to address the system effectively help learners learn the challenges of remote psychomotor training different psychomotor skills? and help us achieve our research goals. Figure 1: The synthesised model for DBR in the context of this research [[14], [15]]. In learning sciences, a conceptual model is 4. Evaluate in practice: Subsequently, in the commonly used to improve explanations and next phase, focus group experiments involving provide visual representations of abstracts [16]. the teachers/experts will be carried out for Following this theory, we sketched a model for qualitative analysis to gather important visualising the overall learning process using details that can be added to the system. Further, the immersive training environment from the a user test will be conducted to reveal essential human learner perspective (see Figure 2). aspects of how the system can be improved. Based on the model, multimodal data will Additionally, questionnaires and surveys for be collected by tracking the skeletal points and the quantitative analysis are helpful to provide capturing the body motion of the human a general idea of how users perceive the learner's body. Instructional tasks are ideally interaction between the system. The refinement given before the learner performs the specific of the system should then be followed involving tasks. Feedback is typically given in real-time the teachers/experts to ensure that the system is when mistakes are detected during training and ready to be tested with the learners in the real- as visual summative, after training. Instructions world setting. Then, the data is collected and are also given during training to help learners analysed to answer the research questions and progress to the next steps or even in the form of to construct design principles. detailed feedback. These two aspects of the human learning model - instructions and 5. Reflection, dual outcomes: feedback - can be given in multiple modalities. In the context of our research, the most Practical: This phase enhances the common modalities that can be applied are implementation of the solution. As reflection visual, audio, and haptic. These modalities form occurs, new designs can be further developed various types of interaction that can be and implemented, which leads to an ongoing potentially used in the immersive training sub-cycle of the design-reflection process. environment to give instructions and feedback such as virtual avatars, videos, etc. Finally, Theoretical: It is imperative to keep detailed these aspects help validate the effectiveness of records during the design research process the immersive training environment. concerning how the design outcomes (e.g., Since the research is in an early phase, the principles) have worked or have not worked, conceptual model is still on the abstract level. how the innovation has been improved, and However, this constitutes the groundwork of what are changes have been made. Through this this research and will be extended into a bigger documentation, it can be helpful for other model with more aspects in the later phases. researchers and designers who are interested in those findings and examine them in relation to their context and needs. 5. Discussion and conclusion The expected outcomes of this research are 4.2. Solution approach divided into two implications: theoretical and practical. From the theoretical perspective, Figure 2: The conceptual model of this research. systematic literature review findings on improve the learning outcome in the requirements to create immersive training psychomotor domain, especially in remote- environments for psychomotor skills will be learning scenarios. delivered. Based on these findings, a conceptual framework of the immersive 6. Acknowledgements training environment consisting of guidelines and methodologies on delivering instructions, This research is funded by the Federal providing feedback, and tracking learner's Ministry of Education and Research (BMBF) in performance will be constructed. We envision the program of Digital Higher Education for the this framework to constitute the groundwork project Multimodal Immersive Learning with for the design. Moreover, it will extend immersive training environments for Artificial Intelligence for Psychomotor Skills psychomotor skills training in multiple (MILKI-PSY) with the grant number: 16DHB4013. domains. This framework will potentially be useful for researchers as a basis for their theoretical and practical research. 7. References From the practical perspective, a system for delivering effective instructions, providing [1] R. W. Barnes, Surgical handicraft: meaningful feedback, and tracking learner's Teaching andlearning surgical skills, The performance will be developed. Similarly, as American Journal ofSurgery 153 (1987) the theoretical implication, such a system needs 422–427. Papers of the NorthPacific to be adapted in various psychomotor domains Surgical Association. for different skills training. The empirical [2] A. Ericsson, M. Prietula, E. Cokely, The studies will be carried out with learners to making of an expert harvard business measure the effectiveness of the system and the review (july–august 2007), Expert outcome should deliver promising results. Harvard Business Review,July–August Consequently, learners and teachers can benefit (2007). from the system to help them with the training. [3] I. Wachsmuth, M. Lenzen, G. Knoblich, This research investigates the effectiveness Embodied communication in humans and of an immersive training environment in the machines, Oxford University Press, 2008. development of psychomotor skills training. [4] J. Schneider, D. Börner, P. van Rosmalen, The proposed theoretical framework integrates M. Specht, Can you help me with my immersive technologies and sensor pitch? studying a tool for real-time technologies for the immersion and multimodal automated feedback, IEEE Transactionson data, respectively, providing a preliminary yet Learning Technologies 9 (2016) 318–327. significant step towards combining such technologies in a multi-sensor setup to further [5] D. Di Mitri, J. Schneider, K. Trebing, S. agenda, Journal of educational technology Sopka, M. Specht, H. Drachsler, Real-time & society 11 (2008) 29–40. multimodal feedback with the cpr tutor, [15] M. De Villiers, P. Harpur, Design-based in: I. I. Bittencourt, M. Cukurova, K. research the educational technology Muldner, R. Luckin, E. Millán (Eds.), variant of design research: illustrated by Artificial Intelligence in Education, the design of an m-learning environment, Springer International Publishing, Cham, in: proceedings of the South African 2020, pp. 141–152. institute for computer scientists and [6] K. A. Mat Sanusi, D. D. Mitri, B. Limbu, information technologists conference, R. Klemke, Table tennis tutor: Forehand 2013, pp. 252–261. strokes classification based on multimodal [16] G. D. Chittleborough, D. F. Treagust, Why data and neural networks, Sensors 21 models are advantageous to learning (2021) 3121. science, Educación química 20 (2009) 12– [7] J. Herrington, T. C. Reeves, R. Oliver, 17. Immersive learning technologies: Realism and online authentic learning, Journal of computing in Higher Education 19 (2007) 80–99. [8] J. Schneider, D. Börner, P. Van Rosmalen, M. Specht, Augmenting the senses: A review on sensor-based learning support, Sensors 15 (2015) 4097–4133. [9] A. Dias Pereira dos Santos, K. Yacef, R. Martinez-Maldonado, Let’s dance: How to build a user model for dance students using wearable technology, in: Proceedings of the 25th Conference on User Mod-eling, Adaptation and Personalization, UMAP ’17,Association for Computing Machinery, New York, NY, USA, 2017, p. 183–191. [10] P. Blikstein, M. Worsley, Multimodal learning analytics and education data mining: using computational technologies to measure complex learning tasks, Journal of Learning Analytics 3 (2016) 220–238. [11] B. H. Limbu, H. Jarodzka, R. Klemke, M. Specht, Can you ink while you blink? assessing mental effort in a sensor-based calligraphy trainer, Sensors19 (2019). [12] P. Song, S. Xu, W. T. Fong, C. L. Chin, G. G. Chua, Z. Huang, An immersive vr system for sports education, IEICE TRANSACTIONS on Information and Systems 95 (2012) 1324–1331. [13] S. F. Ali, S. Noor, S. A. Azmat, A. U. Noor, H. Siddiqui, Virtual reality as a physical training assistant, in: 2017 International Conference on Information and Communication Technologies (ICICT), IEEE, 2017,pp. 191–196. [14] T. Amiel, T. C. Reeves, Design-based research and educational technology: Rethinking technology and the research