The coordination of psychomotor skills requires deliberate practice and techniques, all of which are typically taught in a physical setting, where instructions and timely feedback are given by the teachers. However, doing so remotely is commonly inefficient and ineffective, therefore, hindering the learner's progress. Sensors and immersive technologies enable the collection of multimodal data and the creation of immersion, respectively. These technologies have been widely used to further improve the learning outcome, especially in the psychomotor domain. In this paper, we present our research on designing an immersive training environment for remote psychomotor skill training and investigating how such an environment can be used for training skills in different psychomotor domains.
The global pandemic event of Covid-19 has
affected various learning and teaching activities
acutely. This necessitates the notion of online
learning or e-learning in which web
conferencing tools (e.g., Zoom, Teams) are
widely utilised by teachers and students for
classroom activities. However, this is rarely the
case for psychomotor skills development as
they require hands-on practice. Psychomotor
skills need to be physically executed, in most
cases, repetitively to the extent that the muscle
memory is trained, which will automate the
muscle movements [
Nowadays, educational technology and artificial intelligence (AI) researchers are progressively embedding sensor technologies for the collection of multimodal data, and machine learning approaches for tracking learners' behaviour and progress in authentic learning contexts. The combination of these technologies introduces new technological affordances that can be leveraged in the psychomotor education, especially in a remote manner, to further improve the learning outcome.
Multimodality is a theoretical assumption
that can be applied to provide more structure in
sensors for exploring learning. The general idea
of multimodality in learning comes from the
theory of embodied communication. Based on
this theory, humans use their whole bodies to
communicate with each other, applying various
channels to exchange messages such as
gestures, facial expressions, prosody, etc. [
While the multimodal approach helps to
improve the psychomotor learning outcome,
designing virtual training environments adds
immersion to the learning activity. As such,
immersive learning technologies such as virtual
reality (VR), augmented reality (AR) and game
elements enable the creation of virtual training
environments or simulations that typically
consist of nearly, if not entirely, realistic
physical similarity to an actual learning context.
Herrington et al. [
In this research, we aim to design and implement an immersive training environment for psychomotor skills using immersive technologies which will be integrated with sensor technologies and AI, in order to deliver instructions and feedback to learners in a meaningful manner. Furthermore, we intend to investigate the effectiveness of the system and whether it can be applied to train skills in different psychomotor domains. The development of this system provides an early and significant step towards combining immersive technologies and sensor technologies in a multi-sensor setup for collecting multimodal data and giving immediate feedback in an immersive training environment in which learners can use to improve their psychomotor skills independently.
The paper is structured as follows. In Section 2, we present related studies that utilise sensors for the collection of multimodal data and immersive training environments for immersion, and to what extent they are used in the psychomotor domain. Next, we explain our research questions in Section 3. Subsequently, in Section 4, we visualise and describe the research model and methods of this study. Finally, in Section 5 we discuss the expected outcomes of our study in theoretical and practical implications, followed by the conclusion.
Sensor technologies are increasingly
becoming more portable and increasingly used
in psychomotor training, enabling efficient
methods for the acquisition of performance
data, which allows effective monitoring and
intervention. That being said, such devices have
been explored to provide support in the learning
domain. For example, Schneider et al. [
Motion sensors such as accelerometers and
gyroscopes are predominantly used to acquire
motion data to recognize human activities,
especially in the psychomotor domain. These
sensors are commonly combined and used in a
synchronized manner to achieve a higher
accuracy of detecting not only simple but
complex activities as well [
For instance, Schneider et al. [
To better understand learners’ performance,
educational researchers are progressively using
machine learning approaches to classify
activities based on the multimodal data
collected. For instance, in the medical domain,
Di Mitri et al. [
In this research, we aim to use a multisensory system (e.g., wearable technologies, depth cameras) with the help of machine learning to help learners improve their psychomotor skills. We intend to have a theoretical framework that can be used for training skills in one psychomotor domain and subsequently applied in multiple domains.
Immersive learning technologies such as VR and AR are progressively becoming a significant medium for psychomotor training. Due to the substantial improvement and development in recent years, such technologies are being used in various psychomotor domains, including sports, physical training, rehabilitation therapy, and much more. These technologies transport individuals into an interactive training or learning environment, either virtually or physically, to replicate the authentic learning context of a specific skill.
For example, Song et al. [
Ali et al. [
In our research, we aim to incorporate immersive technologies into the mix for the creation of immersive training environments to enhance the immersive experience of the learner in the learning setting. Our grand vision is to have a theoretical framework with a structured human learning model (feedback and instructions) within these immersive training environments that can be applied to not only one but also multiple psychomotor domains.
Based on the problem identified and the related work analysed, we aim to investigate the following research questions (RQs): 1. What level of technological support (technology) is available in the literature and appropriate for delivering effective instructions and feedback (pedagogy) to the psychomotor training? learners in
Fundamentally, it is crucial to identify the most promising pedagogical approaches in psychomotor skills learning that can be applied in multiple psychomotor domains. Furthermore, with technologies that have been widely used to improve the learning outcome in recent years, we survey the state-of-the-art of technology that may potentially be helpful for our research. Therefore, a systematic review will be carried out for these two processes and thus, answer our RQ1. The outcome of answering this question would be the theoretical framework of the system.
2. How can we create an immersive and information-rich (remote/self-learning) training environment for psychomotor skills that deliver effective instructions and meaningful feedback to the learner?
Subsequently, we design and implement a virtual training environment based on the theoretical framework retrieved from RQ1. The instruction and feedback systems should be given in a realistic manner to create immersive learning experiences. Therefore, it is vital to research how can we maximise the system’s effectiveness in providing feedback and instructions. This includes the framing of interaction and the appropriate modalities for instructions and feedback. Consequently, we can investigate the effectiveness of the system: can the system help learners improve their skills during training? 3. To what extent can we generalise our training framework to multiple psychomotor domains?
Finally, we explore if the system can, both theoretically and practically, be adapted and applicable in multiple psychomotor domains. More exercise routines and common mistakes of the selected applications cases will be identified to suit the system's needs. Hence, it is crucial to know what are other possible application cases that the new system can be used to train related psychomotor tasks and can the system effectively help learners learn different psychomotor skills?
It is essential for this research to follow a
methodological approach for designing,
developing, testing, and evaluating such a
system. Hence, we conduct our research based
on the Design-based Research (DBR)
approach, a common iterative methodological
approach for prototypical solutions. In the
context of our research, we combined two DBR
models from Amiel & Reeves [
Figure 1 shows the phases of the DBR approach for this research, and the following subsections explain each of the phases.
1. Problem analysis: In the first phase, a systematic literature will be reviewed to determine the importance of the problem and identify the current theory on the immersive multimodal environments in the psychomotor domain. Furthermore, the selection of application cases will be made in this phase. With these approaches, we are analysing the problem and defining research goals. The outcome of this step is a detailed research proposal containing goals and evaluation criteria.
2. Design solution: A theoretical framework is proposed based on the results from the systematic review, identifying the most promising pedagogical model in psychomotor training and the technologies that can be contributed to such a model. Our conceptual model (see Figure 2) states how we transfer the theoretical framework into our system design, suggesting to address the problem from phase 1.
3. Develop solution: The next phase is the implementation of the immersive training environment that serves the research purpose. The development of the system is based on the theoretical framework proposed in phase 2. The outcome is an innovative and functional immersive training environment system with the integration of immersive technologies, sensor technologies, and AI that aims to address the challenges of remote psychomotor training and help us achieve our research goals. 4. Evaluate in practice: Subsequently, in the next phase, focus group experiments involving the teachers/experts will be carried out for qualitative analysis to gather important details that can be added to the system. Further, a user test will be conducted to reveal essential aspects of how the system can be improved. Additionally, questionnaires and surveys for the quantitative analysis are helpful to provide a general idea of how users perceive the interaction between the system. The refinement of the system should then be followed involving the teachers/experts to ensure that the system is ready to be tested with the learners in the realworld setting. Then, the data is collected and analysed to answer the research questions and to construct design principles.
5. Reflection, dual outcomes:
Practical: This phase enhances the implementation of the solution. As reflection occurs, new designs can be further developed and implemented, which leads to an ongoing sub-cycle of the design-reflection process.
Theoretical: It is imperative to keep detailed records during the design research process concerning how the design outcomes (e.g., principles) have worked or have not worked, how the innovation has been improved, and what are changes have been made. Through this documentation, it can be helpful for other researchers and designers who are interested in those findings and examine them in relation to their context and needs. 4.2.
In learning sciences, a conceptual model is
commonly used to improve explanations and
provide visual representations of abstracts [
Based on the model, multimodal data will be collected by tracking the skeletal points and capturing the body motion of the human learner's body. Instructional tasks are ideally given before the learner performs the specific tasks. Feedback is typically given in real-time when mistakes are detected during training and as visual summative, after training. Instructions are also given during training to help learners progress to the next steps or even in the form of detailed feedback. These two aspects of the human learning model - instructions and feedback - can be given in multiple modalities. In the context of our research, the most common modalities that can be applied are visual, audio, and haptic. These modalities form various types of interaction that can be potentially used in the immersive training environment to give instructions and feedback such as virtual avatars, videos, etc. Finally, these aspects help validate the effectiveness of the immersive training environment.
Since the research is in an early phase, the conceptual model is still on the abstract level. However, this constitutes the groundwork of this research and will be extended into a bigger model with more aspects in the later phases.
The expected outcomes of this research are divided into two implications: theoretical and practical. From the theoretical perspective, systematic literature review findings on requirements to create immersive training environments for psychomotor skills will be delivered. Based on these findings, a conceptual framework of the immersive training environment consisting of guidelines and methodologies on delivering instructions, providing feedback, and tracking learner's performance will be constructed. We envision this framework to constitute the groundwork for the design. Moreover, it will extend immersive training environments for psychomotor skills training in multiple domains. This framework will potentially be useful for researchers as a basis for their theoretical and practical research.
From the practical perspective, a system for delivering effective instructions, providing meaningful feedback, and tracking learner's performance will be developed. Similarly, as the theoretical implication, such a system needs to be adapted in various psychomotor domains for different skills training. The empirical studies will be carried out with learners to measure the effectiveness of the system and the outcome should deliver promising results. Consequently, learners and teachers can benefit from the system to help them with the training.
This research investigates the effectiveness of an immersive training environment in the development of psychomotor skills training. The proposed theoretical framework integrates immersive technologies and sensor technologies for the immersion and multimodal data, respectively, providing a preliminary yet significant step towards combining such technologies in a multi-sensor setup to further improve the learning outcome in the psychomotor domain, especially in remotelearning scenarios.
This research is funded by the Federal Ministry of Education and Research (BMBF) in the program of Digital Higher Education for the project Multimodal Immersive Learning with Artificial Intelligence for Psychomotor Skills (MILKI-PSY) with the grant number: 16DHB4013.