Real-time visual feedback on sports performance in an immersive training environment: Presentation of a study concept Mai Geisen, Tobias Baumgartner, Nina Riedl and Stefanie Klatt Institute of Exercise Training and Sport Informatics, German Sport University Cologne, Cologne, Germany m.geisen@dshs-koeln.de Abstract. Motor learning is particularly favored by the provision of feedback on the learner’s actions. The essential role of feedback is specifically evident in sports, where important components refer to learning and optimizing individual motion techniques and sequences. With the help of motion feedback, both athletes and novices can optimize and learn as well as internalize the correct motion execution in order to improve their sports performance in the long term. Due to innovative, immersive training environments, it is possible to provide humans with visual feedback via screens during motion execution for real-time corrections in motor learning. Accordingly, this paper presents a study design for the use of real-time visual feedback in an immersive environment that aims to enable subjects to optimize their performance of a motor task. This concept is elaborated and implemented particularly under the aspect of improving psychomotor learning within the framework of the MILKI-PSY project. Keywords: immediate feedback · psychomotor learning · motion adaptation 1 Introduction Learning and optimizing individual motion techniques and sequences is particularly important in sports for enabling a long-term increase in athletic performance [4]. To optimize motor skills, it is necessary to provide feedback on the individual’s movement execution. This refers to feedback both in terms of spatial characteristics, i.e., single body part positions (e.g., joint angles), as well as temporal aspects, i.e., temporal coordination of motion sequences [11]. Possible errors in motion execution can thus be identified and improved, ultimately enabling the learning of correct motion execution [14]. In addition, feedback can also be used positively to promote the physical health of humans, specifically referring to athletes, by informing them about incorrect postures, with the aim of preventing possible injuries [3]. It has been shown that feedback provided by a human expert or a technical device effectively promotes motor learning. Previous studies have focused particularly on visual feedback [15]. Specifically, the use of videos has been proven to help athletes increase their performance [9]. With the help of newer technologies that enable training in immersive environments, athletes can even be provided with visual feedback during motion execution for real-time corrections in motor learning [1]. Consequently, real-time visual feedback can improve motion perception and accelerate learning [7]. The study concept presented here aims to investigate the effectiveness of real-time visual feedback, implemented with the help of an immersive training environment, for optimizing a sport-specific motor task, i.e., the squat. 2 Related Work Previous research has been predominantly focusing on the provision of feedback on a completed performance [8]. However, for a learner, this involves noticing possible mistakes after having performed the respective motion. Thereby a direct implementation of the information is made Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). 2 Geisen M. et al. more difficult. Specifically, having the opportunity to receive timely and accurate feedback has been found to be essential for motion optimization of athletes [10]. Within the medical- and health sector, much research has already been done in regard to real-time practice of psychomotor skills, such as surgery suture training [12] and rehabilitation of upper limb motions [2], using immersive environments. In the field of sports, a few studies including the use of immersive real-time feedback have already demonstrated the benefits for sports training and performance, as well. A system for applying multimodal feedback in an immersive environment has allowed the perception of differences in motions during a golf swing between a learner and an expert in three dimensions, in real-time. Based on the possibility of trying to imitate the optimized motion, the system proves to be an optimized learning tool [6]. Furthermore, the application of verbal and visual stimuli during the execution of squats and Tai Chi pushes was investigated by means of an immersive sports training environment. Real-time feedback was generated in the form of color highlights on the learner's avatar, which was concluded by Hülsmann et al. [5] to be a useful feedback method for sports training. 3 Research Approach The aim of this study concept is to compare different real-time feedback methods for motion learning and optimization of the squat, specifically investigating the use of visual stimuli in a novel immersive training environment. The concept of real-time visual feedback is to be tested in the form of a comparison between the optimized motion and the learner’s motion execution. Possibilities are created to obtain direct feedback on the self-performed motion during training without having to rely on the perspective of another person. Within an immersive training environment, this is specifically tested by two variations of the visual stimuli to be shown in order to determine how each of these types of visual feedback can be beneficial for learners. In addition, a further feedback method as well as training without feedback provision for the learner, both being more conventional training methods, are included in the study concept. The division into four training methods is particularly intended to allow the gradual acquisition and investigation of real-time visual stimuli, especially within an immersive training environment. Accordingly, it can be investigated to what extent different approaches of real-time feedback with varying specifications of immersive training environments are accepted by learners and can be used successfully. Later it is the aim to establish new training concepts in the areas of strength and strength endurance as well as in different sports when adapted to the respective motion goals. Within the framework of the new study concept, the following questions are to be answered: 1. Does real-time visual feedback via an immersive training environment help to adapt the learner’s motion to the optimized motion execution in a sports motor task (squat)? ❖ Specifically investigating two innovative visualization methods, i.e., superimposing video recordings of motions (optimized motion and learner motion) vs. projecting the skeleton of the optimized motion on top of the learner’s skeleton as well as highlighting (color coding) deviations from the optimized motion 2. Which feedback method is most supportive in terms of motion learning and optimization? ❖ Comparison of the four different training methods, i.e., gradual increase of innovative and immersive visual stimuli in real-time with each additional feedback method Real-time visual feedback on sports performance in an immersive training environment 3 4 Methodology The variables to be investigated within the study on the one hand refer to the spatial, motion-related characteristics, i.e., the joint angle positions, especially in regard to the hip, knee, and ankle joints, and on the other hand relate to the temporal aspects of the repetitive motion of the squat. Motion adaptation via the provision of visual stimuli in real-time will be analyzed in 18-23 year old subjects. The subjects will be randomly divided into four groups (hereafter referred to as Group A, B, C, and D). All groups will conduct one training session including 50 trials of executing the squat. Group A and B will receive the respective variation of the immersive real-time visual feedback, i.e., the interventions shown in Figure 1. Fig. 1. Demonstration of the two variations of real-time visual feedback. Real-time visual feedback in the form of providing the skeleton of the optimized motion on top of the learner’s skeleton and highlighting deviations is shown in the left picture, while real-time visual feedback in the form of superimposing video recordings of motions is presented in the right picture. To ensure that a precise motion execution of the squat will be learned and optimized in the training of these two groups, the desired motion, i.e., the spatially and temporally optimized motion execution, is captured in advance. Two Microsoft Azure Kinect DKs will be used to capture different perspectives of the squat. With regard to the spatial and temporal motion characteristics of an optimized squat, reference is made to previous research [13]. The execution of the optimized motion to be captured will be done by an expert. Group C and D serve as control groups (conventional training methods), also performing different forms of squat training, but without having the opportunity of being provided with the newly developed real-time visual feedback in an immersive training environment. Group D will be conducting a training program that includes the provision of a video in which an expert, i.e., the same expert whose motion execution will be captured in regard to the training programs for Group A and B, is shown executing the optimized squat. The same will also be done for Group C with the 4 Geisen M. et al. addition of filming the subject during the training and projecting this video next to the video of the expert, thereby allowing the subject to compare the two motion executions during the training process. In conclusion, Group D will thereby train within the most conventional, familiar training environment, while Group A experiences the novel, immersive training environment that includes previously unknown visual stimuli. In order to compare the current motion execution with the performance of the squat after the training, i.e., for measuring the learning success, all groups will conduct a pretest and a posttest. This is done with the help of manual annotation of the joint angle positions and temporal execution of the motion, which is possible on the basis of video recordings. In this way, the motion performance of the subjects at both time points (pre- and posttest) can be compared with each other and a possible motion adaptation in the respective groups can be determined. In addition, a comparison can be drawn between the two variations of the visual feedback with regard to motor learning (Group A and B). 5 Future Work With regard to the results of the current study, the concept of real-time feedback in immersive training environments is to be verified and, if necessary, will be revised. Specifically, the concept will be further elaborated considering the respective needs of the learners as well as regarding technical components. Based on this, further studies on motor tasks in various sports, presumably in running and dancing, are planned to be conducted within the framework of the MILKI-PSY project. Future studies shall thereby not only aim to transfer the immersive training program to other sports, but will also involve the development and investigation of other sensory feedback methods (e.g., tactile and auditive), as multimodal immersive learning is a key aspect of MILKI-PSY. 6 Conclusion In summary, the planned study will be conducted in order to expand research on the topic of immersive learning of psychomotor skills, specifically in the field of sports. In particular, the use of sensory real-time feedback by means of immersive training environments is aimed to simplify learning and optimizing motion execution in the future. In sports, this could help athletes to improve their athletic performance in the long term and enrich the important work of coaches, especially with regard to feedback provision. References 1. Davaris, M., Wijewickrema, S., Zhou, Y., Piromchai, P., Bailey, J., Kennedy, G., O’Leary, S.: The importance of automated real-time performance feedback in virtual reality temporal bone surgery training. In: S. Isotani, E. Millán, A. Ogan, P. Hastings, B. McLaren, R. Luckin. (eds). Artificial Intelligence in Education. Lecture Notes in Computer Science, 11625. Springer (2019) https://doi.org/10.1007/978-3-030-23204-7_9 2. Dias, P., Silva, R., Amorim, P., Lains, J., Roque, E., Pereira, I. S. F., Pereira, F., Santos, B. S., Potel M.: Using virtual reality to increase motivation in poststroke rehabilitation. IEEE Computer Graphics and Applications 39(1), 64-70 (2019). https://doi.org/10.1109/MCG.2018.2875630 3. Harris, S. A., Chivers, P., McIntyre, F. L., Piggott, B., Bulsara, M., Farringdon, F.: Exploring the association between recent concussion, subconcussive impacts and depressive symptoms in male Australian football players. BMJ Open Sport & Exercise Medicine, 6(1), (2020). https://doi.org/ 10.1136/bmjsem-2019-000655 4. Hirtz, P., & Hummel, A.: Motorisches Lernen im Sportunterricht. Handbuch Bewegungswissenschaft – Bewegungslehre. Hofmann Verlag. (2003). Real-time visual feedback on sports performance in an immersive training environment 5 5. Hülsmann, F., Göpfert, J. P., Hammer, B., Kopp, S., Botsch, M.: Classification of motor errors to provide real-time feedback for sports coaching in virtual reality — A case study in squats and Tai Chi pushes. Computers & Graphics, 76, 47-59 (2018). https://doi.org/10.1016/j.cag.2018.08.003 6. Ikeda, A., Hwang, D. H., Koike, H.: AR based self-sports learning system using decayed dynamic time warping algorithm. International Conference on Artificial Reality and Telexistence Eurographics Symposium on Virtual Environments. (2018). 7. Katz, L., Parker, J., Treyman, H., Kopp, G., Levy, R., Chang, E.: Virtual reality in sport and wellness: Promise and reality. International Journal of Computer Science in Sport, 4, 4-16 (2006). 8. Kelley, H., Miltenberger, R. G.: Using video feedback to improve horseback‐riding skills. Journal of Applied Behavior Analysis, 49(1), 138-147 (2015). https://doi.org/10.1002/jaba.272 9. Keogh, J. W., Hume, P. A.: Evidence for biomechanics and motor learning research improving golf performance. Sports Biomechanics, 11, 288-309 (2012). https://doi.org/10.1080/14763141.2012.67 1354 10. Kirby, R.: Development of a real‐time performance measurement and feedback system for alpine skiers. Sports Technology, 2(1-2), 43-52 (2009). https://doi.org/10.1080/19346182.2009.9648498 11. Rauter, G., Sigrist, R., Riener, R., Wolf, P.: Learning of temporal and spatial movement aspects: A comparison of four types of haptic control and concurrent visual feedback. IEEE Transactions on Haptics 8(4), (2015). https://doi.org/10.1109/TOH.2015.2431686 12. Sadeghi-Esfahlani, S., Izsof, V., Minter, S., Kordzadeh, A., Shirvani, H., Esfahlani, K. S.: Development of an interactive virtual reality for medical skills training supervised by artificial neural network. In: Bi, Y., Bhatia, R., Kapoor, S. (eds). Intelligent Systems and Applications. Advances in Intelligent Systems and Computing, 1038. Springer. (2019). https://doi.org/10.1007/978-3-030-29513-4_34 13. Schoenfeld, B. J.: Squatting kinematics and kinetics and their application to exercise performance. Journal of Strength and Conditioning Research, 24(12), 3497-3506 (2010). https://doi.org/10.1519/JSC.0b013e3181bac2d7 14. Seitz, R.J.: Motorisches Lernen: Untersuchungen mit der funktionellen Bildgebung. Deutsche Zeitschrift für Sportmedizin. 52, 343-349 (2001). 15. Sigrist, R., Rauter, G., Riener, R., Wolf, P.: Augmented visual, auditory, haptic, and multimodal feedback in motor learning: A review. Psychonomic Bulletin & Review, 20, 21-53 (2013).