Psychomotor abilities are typically taught in a physical learning environment since they require focused practice and techniques to be learned. However, the lack of feedback modalities in remote psychomotor training makes learning processes inefective and ineficient and can impede the learner's progress. In this paper, we propose an immersive learning system to facilitate direct multimodal feedback for psychomotor skills training. Moreover, survey data to measure the perceived efectiveness of each used feedback modality was collected and analyzed. We ofer a theoretical feedback model and its practical implementation. Initial study results show promising efectiveness of the employed instruction and feedback modalities. This solution could facilitate the multimodal training of psychomotor skills in a time-eficient manner. More research needs to be conducted in order to further test the system.
The presence of teachers in a sports setting is crucial to explain, demonstrate, and assess learning
of psychomotor skills, so that movement errors can be identified and corrected for improving
performances, while also positively influencing the promotion of human physical health [
In this paper, we introduce the Immersive Multimodal Psychomotor Environments for Competence Training (IMPECT)-Sports tool, an ILS prototype which utilizes sensors and immersive technologies for improving psychomotor training, primarily in the sports domain. We discuss the current state of prototypical development and the preliminary results from a qualitative study with the involvement of TEL researchers.
This article is structured as follows. Section 2 describes the proposed tool. Section 3 explains the methodology and shows the results of the collected data. Section 4 summarizes the findings, limitations, future work, and finally, concludes the paper.
The IMPECT-Sports tool is a desktop-based application which utilizes the following input sensors: (1) Perception Neuron 3 (PN3)1, an inertial measurement unit (IMU) sensor-based, full-body motion capture system and (2) Microsoft Azure Kinect (MAK)2, a depth camera sensor for skeleton tracking. For visualizing the training environment with its feedback components, a large screen projected against a wall is used.
System architecture and feedback model Figure 1 shows the system architecture of the IMPECT-Sports tool. The two live avatars, namely Neuron- and Kinect Avatar, receive information from PN3 (motion) and MAK (skeletal) respectively and mirror the moves of the learner. The instruction objects consist of video tutorials of the expert performing three diferent exercises (squats, lunges, and arm lateral tilt) with text instructions, and the Expert Avatar performing the animation of the same exercises. The feedback objects consist of both visual (text and icon-based) and auditory (sound and speech) modalities. In the context of our paper, the expert drives the simulation by observing the movements of the learner and pressing a corresponding key when a mistake is detected.
Figure 2 shows the proposed feedback model for providing multimodal feedback in psychomotor learning scenarios. Depending on the severity of the mistake and the task success measurement, diferent levels of feedback are selected and displayed to the learner in the form of auditory and visual cues. We categorized three levels of mistakes of each exercise based on commonality and criticality. The highest level of mistake was defined to lead to an injury in a shorter time, triggering more alarming feedback. For instance, the Level 3 mistake would trigger 1https://neuronmocap.com/perception-neuron-3-motion-capture-system 2https://azure.microsoft.com/en-us/services/kinect-dk Perceptron Neuron 3
Mocap System Azure Kinect 3D
Depth Sensor IMPECT-Sports (C#)
Kinect Avatar Neuron Avatar
Screen
Audio Speakers Human Learning Elements
Video Tutorials
Expert Avatar
Feedback Elements a higher alarming sound and warning icon/text to prompt the learner that such a mistake is crucial and needs to be corrected immediately.
(a) Efectiveness of instruction modalities (b) Efectiveness of feedback modalities
The IMPECT-Sports tool was evaluated in two workshops: “Multimodality and AI in Education” at JTELSS 20223 and “IMTSECT” at CTE-STEM 20224. In total, there were 33 TEL researchers attending the workshops. After a brief introduction, 21 participants of the workshop tested the IMPECT-Sports tool.
Each performed three diferent exercises with 8 to 10 repetitions observed by a sportsscientific expert who critically rated the users motion executions and provided immediate relevant feedback (either praising or corrective). After the session, the participants were asked to fill out a questionnaire consisting of questions with the following aspects; (a) Instruction modalities - Efectiveness, (b) Feedback modalities - Efectiveness, and (c) Suggestions on improving the tool.
Figures 3a and 3b show the efectiveness of both instruction and feedback modalities. Based on the results, we observed that the icon-based visual is more preferred than the text-based visual in the instruction modality (Figure 3a). For the feedback modality (Figure 3b), we noticed that the icon-based was preferred for visual and the speech-based was favoured for auditory.
In this paper, we discussed the prototypical development of IMPECT-Sports, an ILS intended to facilitate the psychomotor skills acquisition process, and the preliminary results from a 3https://ea-tel.eu/jtelss22 4https://cte-stem2022.tudelft.nl/home qualitative study with the TEL researchers. The results from the questionnaire served as feedback to the researchers and developers of the IMPECT-Sports tool for considering possible user-specific adaptations and further development. Additionally, it might motivate other TEL researchers to obtain their opinions on how to further use the tool and improve it accordingly.
In summary, the paper ofers a theoretical feedback model and its practical implementation. From the theoretical perspective, the multimodal feedback model can be used to provide feedback based on task success and mistake severity, enhancing the learner experience. The practical IMPECT-Sports tool could be used as a training toolbox to provide feedback and instructional components for training multiple psychomotor domains. The complexity of skills could potentially be learned as long as the selected sensor device is ideal.
Several shortcomings were encountered during the study. Primarily, some of the questions were not answered suficiently and properly. The use of the live survey tool could potentially be the reason for such a limitation, as the amount of time to show each question was restricted. Additionally, the process of attaching the PN3 sensors to the participant’s body and performing the calibration was time-consuming, delaying the whole study and reducing the chance for non-participants to test the IMPECT-Sports tool.
For future work, further evaluation will be conducted to ensure that the tool can be iteratively enhanced as an innovative and long-term solution for psychomotor skills training. This includes additional modalities such as haptic which can be used as a feedback component and more game elements that can be utilized within the immersive training environments provided by the tool, further enhancing the psychomotor learning experiences.
This work was funded by the German Federal Ministry of Education and Research (BMBF) within the project “MILKI-PSY”5 under the project ID: 16DHB4015. Artificial Intelligence in Education, volume 11625 of Lecture Notes in Computer Science, Springer International Publishing, Cham, 2019, pp. 96–109. [4] G. Caccianiga, A. Mariani, C. G. de Paratesi, A. Menciassi, E. de Momi, Multi-sensory guidance and feedback for simulation-based training in robot assisted surgery: A preliminary comparison of visual, haptic, and visuo-haptic, IEEE Robotics and Automation Letters 6 (2021) 3801–3808. doi:10.1109/LRA.2021.3063967. [5] K. A. Mat Sanusi, D. Majonica, L. Künz, R. Klemke, Immersive training environments for psychomotor skills development: A student driven prototype development approach, in: R. Klemke, K. Sanusi, D. Majonica, A. Richert, V. Varney, T. Keller, J. Schneider, ..., N. Riedl (Eds.), First International Workshop on Multimodal Immersive Learning Systems (MILeS 2021), 2021, pp. 53–58. URL: http://ceur-ws.org/Vol-2979/paper7.pdf. [6] J. Herrington, T. C. Reeves, R. Oliver, Immersive learning technologies: Realism and online authentic learning, Journal of Computing in Higher Education 19 (2007) 80–99. doi:10.1007/BF03033421. [7] D. Di Mitri, J. Schneider, K. Trebing, S. Sopka, M. Specht, H. Drachsler, Real-time multimodal feedback with the cpr tutor, in: I. I. Bittencourt, M. Cukurova, K. Muldner, R. Luckin, E. Millán (Eds.), Artificial Intelligence in Education, volume 12163 of Lecture Notes in Computer Science, Springer International Publishing, Cham, 2020, pp. 141–152. [8] K. A. Mat Sanusi, D. Di Mitri, B. Limbu, R. Klemke, Table tennis tutor: Forehand strokes classification based on multimodal data and neural networks, Sensors (Basel, Switzerland) 21 (2021). doi:10.3390/s21093121. [9] K. A. Ericsson, M. J. Prietula, E. T. Cokely, The making of an expert, Harvard Business Review 85 (2007) 114–21, 193.