Immersive training environments for psychomotor skills development: A student driven prototype development approach Khaleel Asyraaf Mat Sanusi[0000−0001−6766−4416] , Daniel [0000−0003−4792−0472] Majonica , Lukas Künz[0000−0001−8623−311X] , and Roland Klemke[0000−0002−9268−3229] Cologne Game Lab, TH Köln, Cologne, Germany ks@colognegamelab.de Abstract. Learning psychomotor skills requires constant deliberate prac- tice, typically hands-on training, which necessitates constant feedback from the mentor. This is, however, uncommon and typically ineffective in a remote setting. Modern immersive learning technologies enable the learners to completely get immersed in various learning situations in a way that feels like experiencing an authentic learning environment and thus, can be applicable in the psychomotor domain. In this paper, we present two students’ prototype examples, namely ”Yu and Mi” and ”Flowmotion”; immersive learning environments which are designed to help learners improve their psychomotor skills in the domains of human- robotic interaction and sports. Keywords: psychomotor skills · immersive technologies · augmented reality · mixed reality 1 Introduction The development of psychomotor skills has a wide range of domains. Generally, psychomotor skills need to be physically executed and require constant feed- back from an instructor [1]. Thus, its development is most prevalent in offline, hands-on work, however, it is not common in a remote setting. Nonetheless, the emergence of immersive learning technologies such as virtual reality (VR) and augmented reality (AR) has made possible the adoption of smart sensing sys- tems solution and creation of immersive (or virtual) training environments to increase the quality of services in the field of psychomotor training. Immersive training environments are learning situations constructed using various techniques and software tools, including game-based learning, simula- tions, virtual 3D worlds, and gamification. These immersive environments are distinguished from the traditional learning methods by their ability to simulate realistic scenarios and environments that allow learners to practice the intended skills remotely. As such, the training environment and designated tasks should create the conditions for ”True” immersion [4]. Therefore, it can be argued that Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). 2 Mat Sanusi K. A. et al. the instructions and feedback provided by the training environment should be pragmatic for the learners to perform the tasks in a correct manner. Further- more, such environments enable the collection of multimodal data, which can be used with artificial intelligence (AI) to further improve the immersion and learning outcomes [2, 3]. That being said, the research project ”Multimodal Immersive Learning with Artificial Intelligence for Psychomotor Skills” (MILKI-PSY) was introduced. MILKI-PSY, a consortium led by the Cologne Game Lab at TH Köln, is de- veloping an interactive training environment by integrating immersive technolo- gies, sensors, and AI to support the training of psychomotor skills in multiple domains. In this paper, we focus on the immersive aspect by introducing two students’ prototype examples of immersive training environments in the learn- ing domains of human-robotic interaction and sports, which later can serve as a groundwork for further development of the project. 2 Related Work Previous studies have utilised immersive training environments for the devel- opment of psychomotor skills in various domains. The following papers were obtained when reviewing the existing literature. Immersive training environments allow the learners to interact with and ma- nipulate both physical and virtual items and environments, using next-generation sensing and imaging technologies. For example, Schneider et al. [6] designed a system to support the development of complex non-verbal communication skills for public speaking. The system uses the Kinect v2 depth camera sensor that is placed in front of the learner to track the skeletal joints of the learner’s body, along with the Microsoft HoloLens headset, to provide feedback to the learner while presenting based on common public speaking mistakes such as facial ex- pressions, body posture, voice volume, gestures, and pauses. We intend to use this approach for visualising the instructions to the learners so that the tasks can be done in a sequential manner and feedback when mistakes or dangers are detected. The immersive training environment should also allow the learner to reflect on their performance in real-time and, by doing so, creates a more realistic learning experience for the learner [4]. As such, a full-body tracking technique is commonly used to visualise the learner’s body movement, which can be achieved by using depth camera sensors. For instance, Kyan et al. [5] implemented a VR ballet dance training system that tracks the learner’s skeletal joints using the Microsoft Kinect depth camera sensor. This system applies a similar concept to the “magic mirror” approach in which the virtual character projected on the wall screen moves accordingly to the learners based on the tasks given and provides immediate feedback, enabling them to reflect on their performance in real-time. Using this approach, particularly in the sports case, we can track the learner’s movements when they perform the tasks given and provide meaningful feedback in the forms of visual and audio when mistakes occur during training. Immersive training environments for psychomotor skills development 3 In terms of scalability, Song et al. [7] designed and implemented an immer- sive VR environment for teaching tennis using a high-definition stereoscopic display, robust and accurate hybrid sensor tracking, shader-based skin deforma- tion, intelligent animation control, and haptic feedback mechanism. The authors reported that, through these technologies, a real-time immersive tennis playing experience is achieved. Potentially, the system can be scaled to adapt various ap- plication cases such as other sports game simulations and even military training simulations. 3 Immersive Training Environments: Prototype Examples Based on this research project, we review two student prototypes within the learning domains of human-robotic interaction and sports. These prototypes were designed and implemented by sixth-semester students for the Impact Games course at Cologne Game Lab. 3.1 Yu and Mi Our first student-made prototype example focuses on the human-robotic inter- action domain, which includes a structured series of movements and learning to cooperate efficiently with the robot to solve the task. In this domain, challenges include the full understanding of the interaction with proper handling of the robot and the non-repetitiveness of the movement tasks. Interacting with the robot in a proper manner is crucial not only for the efficiency of the task but also for the safety of the learner. For this, the learner must fully understand the task and how to interact with the robot. This can be achieved through a visual tutorial like a video or practical tutorial in an AR environment. The first student group designed ”Yu and Mi” (see Figure 1a), an AR game about human-robot interaction for mobile devices. The gameplay stems from four key elements and their interactions: 1) the YuMi IRB 14000 robot, 2) the conveyor, 3) the safe, and 4) the screws. The first three of these elements have to be placed in the real world (i.e., the level) by the player, while the screws are stored inside the safe. Once all the necessary parts are placed correctly, the final test procedure requires the player to open said safe to get access to the screws, which are again needed to finish the assembly process. The safe is opened by completing a UI-minigame, in which the player is presented with a grid of 3x4 dots and a shape they have to recreate on the grid. The player is free to start at any dot and can only draw in straight lines to connect it with other adjacent dots, while 1) the drawn shape must continue from where the previous line ended, 2) the same line cannot be used twice, and 3) the final shape must match the provided shape perfectly. As the scene becomes more and more complex, most of the accompanying instructions and feedback are provided by a robot companion through text overlays within the real world; however, additional UI overlays are used to convey minor and easily understandable information. 4 Mat Sanusi K. A. et al. The target group of Yu and Mi are workers in the manufacturing or industrial sector who have to collaborate with a YuMi robot on a regular basis. Therefore, it is mainly intended for the workers who have less experience collaborating with the YuMi. For the measurement of impact, an experiment will be conducted in order to measure the impact of this application. Participants will be randomly sorted into individual groups; an experimental group, a control group with an explanatory video, and a control group without external support. Experts measure the time each participant took and the amount of mistakes they made during the process. After 2-4 days, the following experiment will be conducted to analyze the long- term learning progress of the application. (a) Yu and Mi (b) Flowmotion Fig. 1: Screenshots of the two prototypes 3.2 Flowmotion The second prototype example focuses on the sports domain, which involves more dynamic and complex movements than the human-robotic interaction. As these pose challenges in the aforementioned domain, most of these movements are repetitive and can be trained deliberately. Therefore, a full-body tracking approach can be potentially used for effectively training these skills. As a proof of concept, the second student group designed ”Flowmotion” (see Figure 1b), a prototype that teaches the fundamentals of yoga poses. The system uses the webcam camera to track the full-body of the learner while performing Yoga movements and channels these movements into a virtual avatar. Similar to the ”magic mirror” approach, it allows the avatar to move accordingly to the learner, enabling the learner to reflect on his/her performance in real-time. Like some exercise games such as Wii Fit, Just Dance 3, Dance Central 3, etc., the students’ motivations were based on these examples. The essential aspects of designing Flowmotion were the identification of basic Yoga movements, common beginner mistakes, and deliverance of instructions and feedback. As for the former two, with the help from the German Sport Immersive training environments for psychomotor skills development 5 University Cologne, the students were able to collect the information they need for designing their prototype. Additionally, the students need to ensure that the selection of modalities for both instructions and feedback is suitable in a real- world setting to prevent the learners from experiencing cognitive overload. As such, visual and audio modalities are chosen for this system. Fundamentally, the system implements a virtual coach that provides instructions for the learner to imitate the movements. Subsequently, the virtual coach will prompt the learner with visual and audio feedback when these movements are correctly or incorrectly done. To further add immersion to the game, the students designed the game environment to be more soothing and appropriate for the learner to perform the Yoga tasks effectively. Thus, yoga studios were created. Flowmotion is designed to help adults with no or less exercising experience and instructors with their workout routines. The students envision that the instructor would record their workout routines and share them online, enabling the learners to perform the exercises remotely. For the measurement of impact, an experiment will be conducted with two groups - the experimental group and the control group. To further validate the effectiveness, a workout video will be included for comparison. The results will show whether there are significant improvements between both groups, and the prototype can help the participants learn skills that can be transferable to other exercises. 4 Future Work For Yu and Mi, changing the digital robot presentation from the AR space to the real world might be a path this prototype can be developed into. A physical robot must be present with these changes, which can be extended through AR instead of being entirely virtual. This will bring new challenges to the application, but it can help convey an immersive learning environment more effectively to the learner. In the case of Flowmotion, further work could also include extensive exercise routines. Besides Yoga poses, the prototype can be further explored in differ- ent sport applications such as running or boxing, which involve more dynamic movements and complex techniques. This will pose new challenges due to the distinction of these activities but provides a significant step into improving psy- chomotor training with immersive learning environments. For the future work of the MILKI-PSY project, we propose using these stu- dent prototypes and create a second iteration of each prototype in cooperation with the students. Concerning the technical aspects of this development, we are planning to replace the lower-tier sensors (i.e., mainly webcams) which were pre- viously used in the student prototypes with hardware that is specialised for these kinds of tasks (e.g., Microsoft Hololens 2, Azure Kinect). This way, we hope to achieve better and more precise outputs due to the more reliable data. These newly implemented prototypes can later be used to create an applica- tion that will be publicised with the MILKI-PSY project. 6 Mat Sanusi K. A. et al. 5 Conclusion In this paper, we introduced two student-made prototypes called ”Yu and Mi” and ”Flowmotion”. These prototypes feature completely different domains, yet both focus on developing psychomotor skills with immersive technologies. In order to measure the impact of each prototype, we propose at least two test groups, one experimental group, and a minimum of one control group. In the final session, we can compare the results of these groups, which will show any possible significant improvements made by using the prototype. In the future, we want to take these student-made prototypes as a starting point for creating and implementing the second iteration of said prototypes. Acknowledgments A special thanks to the student groups ”Yu and Mi” and ”Flowmotion” from the BA6 Impact Games course of the Cologne Game Lab for their contribution to this paper. References 1. Barnes, R. W. (1987). Surgical handicraft: teaching and learning surgical skills. The American journal of surgery, 153(5), 422-427. 2. Divekar, R. R., Drozdal, J., Chabot, S., Zhou, Y., Su, H., Chen, Y., Zhu, H., Hendler, J. A., & Braasch, J. (2021). Foreign language acquisition via artificial intelligence and extended reality: design and evaluation. Computer Assisted Language Learning, 1-29. 3. Echeverria, J., & Santos, O. C. (2021). KUMITRON: A Multimodal Psychomotor Intelligent Learning System to Provide Personalized Support when Training Karate Combats. 4. Herrington, J., Reeves, T. C., & Oliver, R. (2007). Immersive learning technologies: Realism and online authentic learning. Journal of computing in Higher Education, 19(1), 80-99. 5. Kyan, M., Sun, G., Li, H., Zhong, L., Muneesawang, P., Dong, N., Elder, B., & Guan, L. (2015). An approach to ballet dance training through ms kinect and visualization in a cave virtual reality environment. ACM Transactions on Intelligent Systems and Technology (TIST), 6(2), 1-37. https://doi.org/https://doi.org/10.1145/2735951 6. Schneider, J., Börner, D., Van Rosmalen, P., & Specht, M. (2016). Can you help me with my pitch? Studying a tool for real-time auto- mated feedback. IEEE Transactions on Learning Technologies, 9(4), 318-327. https://doi.org/10.1109/TLT.2016.2627043 7. Song, P., Xu, S., Fong, W. T., Chin, C. L., Chua, G. G., & Huang, Z. (2012). An im- mersive VR system for sports education. IEICE TRANSACTIONS on Information and Systems, 95(5), 1324-1331.