=Paper=
{{Paper
|id=Vol-3076/paper11
|storemode=property
|title=Conceptualising immersive multimodal environments for psychomotor skills training
|pdfUrl=https://ceur-ws.org/Vol-3076/ECTEL2021_DC_paper11.pdf
|volume=Vol-3076
|authors=Khaleel Asyraaf Mat Sanusi,Deniz Iren,Roland Klemke
|dblpUrl=https://dblp.org/rec/conf/ectel/SanusiIK21
}}
==Conceptualising immersive multimodal environments for psychomotor skills training==
https://ceur-ws.org/Vol-3076/ECTEL2021_DC_paper11.pdf
Conceptualising immersive multimodal environments for
psychomotor skills training.
Khaleel Asyraaf Mat Sanusia, Deniz Irenb and Roland Klemkea,b
a
Cologne Game Lab, TH Köln, Cologne, Germany
b
Open Universiteit, Heerlen, The Netherlands
Abstract
The coordination of psychomotor skills requires deliberate practice and techniques, all of which
are typically taught in a physical setting, where instructions and timely feedback are given by
the teachers. However, doing so remotely is commonly inefficient and ineffective, therefore,
hindering the learner's progress. Sensors and immersive technologies enable the collection of
multimodal data and the creation of immersion, respectively. These technologies have been
widely used to further improve the learning outcome, especially in the psychomotor domain. In
this paper, we present our research on designing an immersive training environment for remote
psychomotor skill training and investigating how such an environment can be used for training
skills in different psychomotor domains.
Keywords 1
Immersive technologies, Sensors, Multimodal, Psychomotor skills
1. Introduction in a correct manner, which include safety and
effectiveness. Timely and consistent feedback
from the teacher is essential for the learner to
The global pandemic event of Covid-19 has
avoid developing improper techniques during
affected various learning and teaching activities
training, thus ensuring the desired goal can be
acutely. This necessitates the notion of online
achieved in a shorter time [2]. However, doing
learning or e-learning in which web
so in a remote manner makes the learning
conferencing tools (e.g., Zoom, Teams) are
process ineffective and inefficient due to the
widely utilised by teachers and students for
lack of modalities such as haptic feedback or
classroom activities. However, this is rarely the
3D full-body perception, hence impeding the
case for psychomotor skills development as
learner’s progress. Due to this, psychomotor
they require hands-on practice. Psychomotor
skill learners and teachers have been
skills need to be physically executed, in most
substantially affected.
cases, repetitively to the extent that the muscle
Nowadays, educational technology and
memory is trained, which will automate the
artificial intelligence (AI) researchers are
muscle movements [1]. Furthermore, the
progressively embedding sensor technologies
presence of teachers is needed in order to
for the collection of multimodal data, and
explain, demonstrate, and assess certain
machine learning approaches for tracking
procedures. To achieve this, the human learning
learners' behaviour and progress in authentic
model has to be in a structured form where
learning contexts. The combination of these
instructions are well-defined, and feedback can
technologies introduces new technological
be given to ensure that the tasks are performed
Proceedings of the Doctoral Consortium of Sixteenth European
Conference on Technology Enhanced Learning, September 20–21,
2021, Bolzano, Italy (online).
EMAIL: ks@colognegamelab.de (A. 1); deniz.iren@ou.nl (A. 2);
rk@colognegamelab.com (A. 3)
ORCID: 0000-0001-6766-4416 (A. 1); 0000-0002-0727-3445 (A.
2); 0000-0002-9268-3229 (A. 3)
©️ 2021 Copyright for this paper by its authors. Use permitted under Creative
Commons License Attribution 4.0 International (CC BY 4.0).
CEUR Workshop Proceedings (CEUR-WS.org)
affordances that can be leveraged in the immersive technologies and sensor
psychomotor education, especially in a remote technologies in a multi-sensor setup for
manner, to further improve the learning collecting multimodal data and giving
outcome. immediate feedback in an immersive training
Multimodality is a theoretical assumption environment in which learners can use to
that can be applied to provide more structure in improve their psychomotor skills
sensors for exploring learning. The general idea independently.
of multimodality in learning comes from the The paper is structured as follows. In
theory of embodied communication. Based on Section 2, we present related studies that utilise
this theory, humans use their whole bodies to sensors for the collection of multimodal data
communicate with each other, applying various and immersive training environments for
channels to exchange messages such as immersion, and to what extent they are used in
gestures, facial expressions, prosody, etc. [3]. the psychomotor domain. Next, we explain our
Subsequently, the trend of multimodality has research questions in Section 3. Subsequently,
been employed in human-computer interaction. in Section 4, we visualise and describe the
Sensor-based multimodal interfaces allow the research model and methods of this study.
monitoring of different modalities and have Finally, in Section 5 we discuss the expected
been applied in various domains to improve outcomes of our study in theoretical and
learning [[4], [5], [6]]. practical implications, followed by the
While the multimodal approach helps to conclusion.
improve the psychomotor learning outcome,
designing virtual training environments adds 2. Related work
immersion to the learning activity. As such,
immersive learning technologies such as virtual 2.1. Sensors in psychomotor
reality (VR), augmented reality (AR) and game
elements enable the creation of virtual training
learning
environments or simulations that typically
consist of nearly, if not entirely, realistic Sensor technologies are increasingly
physical similarity to an actual learning context. becoming more portable and increasingly used
Herrington et al. [7] stress that the learning in psychomotor training, enabling efficient
environment and designated tasks create the methods for the acquisition of performance
conditions for the “True” immersion. Hence, it data, which allows effective monitoring and
can be argued that the instructions and feedback intervention. That being said, such devices have
provided by the learning environment should be been explored to provide support in the learning
pragmatic for the learners to learn to perform domain. For example, Schneider et al. [8]
the tasks in a correct manner; for example, analysed 82 prototypes found in literature
personalised feedback (human-teacher-like) to studies based on Bloom’s taxonomy of learning
create immersive learning experiences. domains (psychomotor, cognitive, and
Intrinsically, virtual training environments affective). Their research suggests researchers
allow the learner to actively interact with the in- and educators to consider utilising sensor-based
game objects which may create more platforms as reliable learning tools for reducing
engagement and increase motivation for the the workload of teachers and, therefore,
learner when performing tasks. contribute to the solution of many current
In this research, we aim to design and educational challenges.
implement an immersive training environment Motion sensors such as accelerometers and
for psychomotor skills using immersive gyroscopes are predominantly used to acquire
technologies which will be integrated with motion data to recognize human activities,
sensor technologies and AI, in order to deliver especially in the psychomotor domain. These
instructions and feedback to learners in a sensors are commonly combined and used in a
meaningful manner. Furthermore, we intend to synchronized manner to achieve a higher
investigate the effectiveness of the system and accuracy of detecting not only simple but
whether it can be applied to train skills in complex activities as well [9]. This enables the
different psychomotor domains. The collection of multimodal data and provides a
development of this system provides an early more accurate representation of the learning
and significant step towards combining process [10]. Furthermore, multimodal data can
be collected using various sensors such as development in recent years, such technologies
wearable sensors, depth camera sensors, are being used in various psychomotor
Internet of Things devices, etc. domains, including sports, physical training,
For instance, Schneider et al. [4] designed a rehabilitation therapy, and much more. These
system to support the development of public technologies transport individuals into an
speaking skills using the Kinect v2 depth interactive training or learning environment,
camera sensor to track the skeletal joints of the either virtually or physically, to replicate the
learner's body and the HoloLens headset to authentic learning context of a specific skill.
provide feedback in real-time when mistakes For example, Song et al. [12] designed and
are detected while presenting. Limbu et al. [11] implemented an immersive VR environment
developed a system to teach basic calligraphy for teaching tennis using high-definition
skills, which uses the pen sensor in Microsoft stereoscopic display, robust and accurate
Surface tablet and EMG sensors in a Myo hybrid sensor tracking, shader-based skin
armband to provide feedback to learners during deformation, intelligent animation control, and
practice. It also allows the calligraphy teacher haptic feedback mechanism. The authors
to create an expert model, which the learners reported that, through these technologies, a
can later use to practice and receive guidance real-time immersive tennis playing experience
and feedback based on the expert model. is achieved. Potentially, the system can be
To better understand learners’ performance, scaled to adapt various application cases such
educational researchers are progressively using as other sports game simulations and even
machine learning approaches to classify military training simulations.
activities based on the multimodal data Ali et al. [13] experimented with multiple
collected. For instance, in the medical domain, VR fitness applications (e.g., VR Fitness,
Di Mitri et al. [5] investigated how multimodal VirZOOM, BOXVR) for physical training such
data and Neural Networks can be used for as walking, running, and jogging. In addition,
learning Cardiopulmonary Resuscitation skills they implemented a mobile application that
by utilising a multi-sensor system comprising uses built-in sensors such as an accelerometer
of a Kinect v2 and a Myo armband. In the sports and gyroscope for motion detection. As a result,
domain, Mat Sanusi et al. [6] applied the same they achieved up to 82.46% of accuracy and
framework as the previous author by using thus, described the effectiveness of VR
built-in accelerometer and gyroscope sensors in technology in physical training, which is
a smartphone and also a Kinect v2 to detect helpful for the development of psychomotor
forehand table tennis strokes during training. skills.
Both study results show a high classification In our research, we aim to incorporate
rate of the activities when combining the immersive technologies into the mix for the
sensors, emphasising the importance of a creation of immersive training environments to
multimodal approach in classifying complex enhance the immersive experience of the
activities. learner in the learning setting. Our grand vision
In this research, we aim to use a multi- is to have a theoretical framework with a
sensory system (e.g., wearable technologies, structured human learning model (feedback and
depth cameras) with the help of machine instructions) within these immersive training
learning to help learners improve their environments that can be applied to not only
psychomotor skills. We intend to have a one but also multiple psychomotor domains.
theoretical framework that can be used for
training skills in one psychomotor domain and 3. Research questions
subsequently applied in multiple domains.
Based on the problem identified and the
2.2. Immersive training related work analysed, we aim to investigate the
environments following research questions (RQs):
Immersive learning technologies such as VR 1. What level of technological support
and AR are progressively becoming a (technology) is available in the
significant medium for psychomotor training. literature and appropriate for delivering
Due to the substantial improvement and effective instructions and feedback
(pedagogy) to the learners in 4. Methodology
psychomotor training?
4.1. Research methods
Fundamentally, it is crucial to identify the
most promising pedagogical approaches in It is essential for this research to follow a
psychomotor skills learning that can be applied methodological approach for designing,
in multiple psychomotor domains. developing, testing, and evaluating such a
Furthermore, with technologies that have been system. Hence, we conduct our research based
widely used to improve the learning outcome in on the Design-based Research (DBR)
recent years, we survey the state-of-the-art of approach, a common iterative methodological
technology that may potentially be helpful for approach for prototypical solutions. In the
our research. Therefore, a systematic review context of our research, we combined two DBR
will be carried out for these two processes and models from Amiel & Reeves [14], and De
thus, answer our RQ1. The outcome of Vielliers & Harpur [15], which are used in the
answering this question would be the domain of educational technologies.
theoretical framework of the system. Figure 1 shows the phases of the DBR
approach for this research, and the following
2. How can we create an immersive and subsections explain each of the phases.
information-rich (remote/self-learning)
training environment for psychomotor 1. Problem analysis: In the first phase, a
skills that deliver effective instructions systematic literature will be reviewed to
and meaningful feedback to the determine the importance of the problem and
learner? identify the current theory on the immersive
multimodal environments in the psychomotor
Subsequently, we design and implement a domain. Furthermore, the selection of
virtual training environment based on the application cases will be made in this phase.
theoretical framework retrieved from RQ1. The With these approaches, we are analysing the
instruction and feedback systems should be problem and defining research goals. The
given in a realistic manner to create immersive outcome of this step is a detailed research
learning experiences. Therefore, it is vital to proposal containing goals and evaluation
research how can we maximise the system’s criteria.
effectiveness in providing feedback and
instructions. This includes the framing of 2. Design solution: A theoretical framework
interaction and the appropriate modalities for is proposed based on the results from the
instructions and feedback. Consequently, we systematic review, identifying the most
can investigate the effectiveness of the system: promising pedagogical model in psychomotor
can the system help learners improve their training and the technologies that can be
skills during training? contributed to such a model. Our conceptual
model (see Figure 2) states how we transfer the
3. To what extent can we generalise our theoretical framework into our system design,
training framework to multiple suggesting to address the problem from phase
psychomotor domains? 1.
Finally, we explore if the system can, both
3. Develop solution: The next phase is the
theoretically and practically, be adapted and
implementation of the immersive training
applicable in multiple psychomotor domains.
environment that serves the research purpose.
More exercise routines and common mistakes
The development of the system is based on the
of the selected applications cases will be
theoretical framework proposed in phase 2. The
identified to suit the system's needs. Hence, it is
outcome is an innovative and functional
crucial to know what are other possible
immersive training environment system with
application cases that the new system can be
the integration of immersive technologies,
used to train related psychomotor tasks and can
sensor technologies, and AI that aims to address
the system effectively help learners learn
the challenges of remote psychomotor training
different psychomotor skills?
and help us achieve our research goals.
Figure 1: The synthesised model for DBR in the context of this research [[14], [15]].
In learning sciences, a conceptual model is
4. Evaluate in practice: Subsequently, in the commonly used to improve explanations and
next phase, focus group experiments involving provide visual representations of abstracts [16].
the teachers/experts will be carried out for Following this theory, we sketched a model for
qualitative analysis to gather important visualising the overall learning process using
details that can be added to the system. Further, the immersive training environment from the
a user test will be conducted to reveal essential human learner perspective (see Figure 2).
aspects of how the system can be improved. Based on the model, multimodal data will
Additionally, questionnaires and surveys for be collected by tracking the skeletal points and
the quantitative analysis are helpful to provide capturing the body motion of the human
a general idea of how users perceive the learner's body. Instructional tasks are ideally
interaction between the system. The refinement given before the learner performs the specific
of the system should then be followed involving tasks. Feedback is typically given in real-time
the teachers/experts to ensure that the system is when mistakes are detected during training and
ready to be tested with the learners in the real- as visual summative, after training. Instructions
world setting. Then, the data is collected and are also given during training to help learners
analysed to answer the research questions and progress to the next steps or even in the form of
to construct design principles. detailed feedback. These two aspects of the
human learning model - instructions and
5. Reflection, dual outcomes: feedback - can be given in multiple modalities.
In the context of our research, the most
Practical: This phase enhances the common modalities that can be applied are
implementation of the solution. As reflection visual, audio, and haptic. These modalities form
occurs, new designs can be further developed various types of interaction that can be
and implemented, which leads to an ongoing potentially used in the immersive training
sub-cycle of the design-reflection process. environment to give instructions and feedback
such as virtual avatars, videos, etc. Finally,
Theoretical: It is imperative to keep detailed these aspects help validate the effectiveness of
records during the design research process the immersive training environment.
concerning how the design outcomes (e.g., Since the research is in an early phase, the
principles) have worked or have not worked, conceptual model is still on the abstract level.
how the innovation has been improved, and However, this constitutes the groundwork of
what are changes have been made. Through this this research and will be extended into a bigger
documentation, it can be helpful for other model with more aspects in the later phases.
researchers and designers who are interested in
those findings and examine them in relation to
their context and needs.
5. Discussion and conclusion
The expected outcomes of this research are
4.2. Solution approach divided into two implications: theoretical and
practical. From the theoretical perspective,
Figure 2: The conceptual model of this research.
systematic literature review findings on improve the learning outcome in the
requirements to create immersive training psychomotor domain, especially in remote-
environments for psychomotor skills will be learning scenarios.
delivered. Based on these findings, a
conceptual framework of the immersive 6. Acknowledgements
training environment consisting of guidelines
and methodologies on delivering instructions,
This research is funded by the Federal
providing feedback, and tracking learner's
Ministry of Education and Research (BMBF) in
performance will be constructed. We envision
the program of Digital Higher Education for the
this framework to constitute the groundwork
project Multimodal Immersive Learning with
for the design. Moreover, it will extend
immersive training environments for Artificial Intelligence for Psychomotor Skills
psychomotor skills training in multiple (MILKI-PSY) with the grant number:
16DHB4013.
domains. This framework will potentially be
useful for researchers as a basis for their
theoretical and practical research. 7. References
From the practical perspective, a system for
delivering effective instructions, providing [1] R. W. Barnes, Surgical handicraft:
meaningful feedback, and tracking learner's Teaching andlearning surgical skills, The
performance will be developed. Similarly, as American Journal ofSurgery 153 (1987)
the theoretical implication, such a system needs 422–427. Papers of the NorthPacific
to be adapted in various psychomotor domains Surgical Association.
for different skills training. The empirical [2] A. Ericsson, M. Prietula, E. Cokely, The
studies will be carried out with learners to making of an expert harvard business
measure the effectiveness of the system and the review (july–august 2007), Expert
outcome should deliver promising results. Harvard Business Review,July–August
Consequently, learners and teachers can benefit (2007).
from the system to help them with the training. [3] I. Wachsmuth, M. Lenzen, G. Knoblich,
This research investigates the effectiveness Embodied communication in humans and
of an immersive training environment in the machines, Oxford University Press, 2008.
development of psychomotor skills training. [4] J. Schneider, D. Börner, P. van Rosmalen,
The proposed theoretical framework integrates M. Specht, Can you help me with my
immersive technologies and sensor pitch? studying a tool for real-time
technologies for the immersion and multimodal automated feedback, IEEE Transactionson
data, respectively, providing a preliminary yet Learning Technologies 9 (2016) 318–327.
significant step towards combining such
technologies in a multi-sensor setup to further
[5] D. Di Mitri, J. Schneider, K. Trebing, S. agenda, Journal of educational technology
Sopka, M. Specht, H. Drachsler, Real-time & society 11 (2008) 29–40.
multimodal feedback with the cpr tutor, [15] M. De Villiers, P. Harpur, Design-based
in: I. I. Bittencourt, M. Cukurova, K. research the educational technology
Muldner, R. Luckin, E. Millán (Eds.), variant of design research: illustrated by
Artificial Intelligence in Education, the design of an m-learning environment,
Springer International Publishing, Cham, in: proceedings of the South African
2020, pp. 141–152. institute for computer scientists and
[6] K. A. Mat Sanusi, D. D. Mitri, B. Limbu, information technologists conference,
R. Klemke, Table tennis tutor: Forehand 2013, pp. 252–261.
strokes classification based on multimodal [16] G. D. Chittleborough, D. F. Treagust, Why
data and neural networks, Sensors 21 models are advantageous to learning
(2021) 3121. science, Educación química 20 (2009) 12–
[7] J. Herrington, T. C. Reeves, R. Oliver, 17.
Immersive learning technologies: Realism
and online authentic learning, Journal of
computing in Higher Education 19 (2007)
80–99.
[8] J. Schneider, D. Börner, P. Van Rosmalen,
M. Specht, Augmenting the senses: A
review on sensor-based learning support,
Sensors 15 (2015) 4097–4133.
[9] A. Dias Pereira dos Santos, K. Yacef, R.
Martinez-Maldonado, Let’s dance: How to
build a user model for dance students using
wearable technology, in: Proceedings of
the 25th Conference on User Mod-eling,
Adaptation and Personalization, UMAP
’17,Association for Computing
Machinery, New York, NY, USA, 2017, p.
183–191.
[10] P. Blikstein, M. Worsley, Multimodal
learning analytics and education data
mining: using computational technologies
to measure complex learning tasks,
Journal of Learning Analytics 3 (2016)
220–238.
[11] B. H. Limbu, H. Jarodzka, R. Klemke, M.
Specht, Can you ink while you blink?
assessing mental effort in a sensor-based
calligraphy trainer, Sensors19 (2019).
[12] P. Song, S. Xu, W. T. Fong, C. L. Chin, G.
G. Chua, Z. Huang, An immersive vr
system for sports education, IEICE
TRANSACTIONS on Information and
Systems 95 (2012) 1324–1331.
[13] S. F. Ali, S. Noor, S. A. Azmat, A. U.
Noor, H. Siddiqui, Virtual reality as a
physical training assistant, in: 2017
International Conference on Information
and Communication Technologies
(ICICT), IEEE, 2017,pp. 191–196.
[14] T. Amiel, T. C. Reeves, Design-based
research and educational technology:
Rethinking technology and the research