=Paper=
{{Paper
|id=Vol-2979/paper7
|storemode=property
|title=Immersive training environments for psychomotor skills development: A student driven prototype development approach
|pdfUrl=https://ceur-ws.org/Vol-2979/paper7.pdf
|volume=Vol-2979
|authors=Khaleel Asyraaf Mat Sanusi,Daniel Majonica,Lukas Künz,Roland Klemke
|dblpUrl=https://dblp.org/rec/conf/ectel/SanusiMKK21
}}
==Immersive training environments for psychomotor skills development: A student driven prototype development approach==
https://ceur-ws.org/Vol-2979/paper7.pdf
Immersive training environments for
psychomotor skills development: A student
driven prototype development approach
Khaleel Asyraaf Mat Sanusi[0000−0001−6766−4416] , Daniel
[0000−0003−4792−0472]
Majonica , Lukas Künz[0000−0001−8623−311X] , and Roland
Klemke[0000−0002−9268−3229]
Cologne Game Lab, TH Köln, Cologne, Germany
ks@colognegamelab.de
Abstract. Learning psychomotor skills requires constant deliberate prac-
tice, typically hands-on training, which necessitates constant feedback
from the mentor. This is, however, uncommon and typically ineffective
in a remote setting. Modern immersive learning technologies enable the
learners to completely get immersed in various learning situations in a
way that feels like experiencing an authentic learning environment and
thus, can be applicable in the psychomotor domain. In this paper, we
present two students’ prototype examples, namely ”Yu and Mi” and
”Flowmotion”; immersive learning environments which are designed to
help learners improve their psychomotor skills in the domains of human-
robotic interaction and sports.
Keywords: psychomotor skills · immersive technologies · augmented
reality · mixed reality
1 Introduction
The development of psychomotor skills has a wide range of domains. Generally,
psychomotor skills need to be physically executed and require constant feed-
back from an instructor [1]. Thus, its development is most prevalent in offline,
hands-on work, however, it is not common in a remote setting. Nonetheless, the
emergence of immersive learning technologies such as virtual reality (VR) and
augmented reality (AR) has made possible the adoption of smart sensing sys-
tems solution and creation of immersive (or virtual) training environments to
increase the quality of services in the field of psychomotor training.
Immersive training environments are learning situations constructed using
various techniques and software tools, including game-based learning, simula-
tions, virtual 3D worlds, and gamification. These immersive environments are
distinguished from the traditional learning methods by their ability to simulate
realistic scenarios and environments that allow learners to practice the intended
skills remotely. As such, the training environment and designated tasks should
create the conditions for ”True” immersion [4]. Therefore, it can be argued that
Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
2 Mat Sanusi K. A. et al.
the instructions and feedback provided by the training environment should be
pragmatic for the learners to perform the tasks in a correct manner. Further-
more, such environments enable the collection of multimodal data, which can
be used with artificial intelligence (AI) to further improve the immersion and
learning outcomes [2, 3].
That being said, the research project ”Multimodal Immersive Learning with
Artificial Intelligence for Psychomotor Skills” (MILKI-PSY) was introduced.
MILKI-PSY, a consortium led by the Cologne Game Lab at TH Köln, is de-
veloping an interactive training environment by integrating immersive technolo-
gies, sensors, and AI to support the training of psychomotor skills in multiple
domains. In this paper, we focus on the immersive aspect by introducing two
students’ prototype examples of immersive training environments in the learn-
ing domains of human-robotic interaction and sports, which later can serve as a
groundwork for further development of the project.
2 Related Work
Previous studies have utilised immersive training environments for the devel-
opment of psychomotor skills in various domains. The following papers were
obtained when reviewing the existing literature.
Immersive training environments allow the learners to interact with and ma-
nipulate both physical and virtual items and environments, using next-generation
sensing and imaging technologies. For example, Schneider et al. [6] designed a
system to support the development of complex non-verbal communication skills
for public speaking. The system uses the Kinect v2 depth camera sensor that is
placed in front of the learner to track the skeletal joints of the learner’s body,
along with the Microsoft HoloLens headset, to provide feedback to the learner
while presenting based on common public speaking mistakes such as facial ex-
pressions, body posture, voice volume, gestures, and pauses. We intend to use
this approach for visualising the instructions to the learners so that the tasks
can be done in a sequential manner and feedback when mistakes or dangers are
detected.
The immersive training environment should also allow the learner to reflect
on their performance in real-time and, by doing so, creates a more realistic
learning experience for the learner [4]. As such, a full-body tracking technique is
commonly used to visualise the learner’s body movement, which can be achieved
by using depth camera sensors. For instance, Kyan et al. [5] implemented a VR
ballet dance training system that tracks the learner’s skeletal joints using the
Microsoft Kinect depth camera sensor. This system applies a similar concept to
the “magic mirror” approach in which the virtual character projected on the wall
screen moves accordingly to the learners based on the tasks given and provides
immediate feedback, enabling them to reflect on their performance in real-time.
Using this approach, particularly in the sports case, we can track the learner’s
movements when they perform the tasks given and provide meaningful feedback
in the forms of visual and audio when mistakes occur during training.
Immersive training environments for psychomotor skills development 3
In terms of scalability, Song et al. [7] designed and implemented an immer-
sive VR environment for teaching tennis using a high-definition stereoscopic
display, robust and accurate hybrid sensor tracking, shader-based skin deforma-
tion, intelligent animation control, and haptic feedback mechanism. The authors
reported that, through these technologies, a real-time immersive tennis playing
experience is achieved. Potentially, the system can be scaled to adapt various ap-
plication cases such as other sports game simulations and even military training
simulations.
3 Immersive Training Environments: Prototype
Examples
Based on this research project, we review two student prototypes within the
learning domains of human-robotic interaction and sports. These prototypes
were designed and implemented by sixth-semester students for the Impact Games
course at Cologne Game Lab.
3.1 Yu and Mi
Our first student-made prototype example focuses on the human-robotic inter-
action domain, which includes a structured series of movements and learning to
cooperate efficiently with the robot to solve the task. In this domain, challenges
include the full understanding of the interaction with proper handling of the
robot and the non-repetitiveness of the movement tasks. Interacting with the
robot in a proper manner is crucial not only for the efficiency of the task but
also for the safety of the learner. For this, the learner must fully understand the
task and how to interact with the robot. This can be achieved through a visual
tutorial like a video or practical tutorial in an AR environment.
The first student group designed ”Yu and Mi” (see Figure 1a), an AR game
about human-robot interaction for mobile devices. The gameplay stems from
four key elements and their interactions: 1) the YuMi IRB 14000 robot, 2) the
conveyor, 3) the safe, and 4) the screws. The first three of these elements have
to be placed in the real world (i.e., the level) by the player, while the screws are
stored inside the safe. Once all the necessary parts are placed correctly, the final
test procedure requires the player to open said safe to get access to the screws,
which are again needed to finish the assembly process. The safe is opened by
completing a UI-minigame, in which the player is presented with a grid of 3x4
dots and a shape they have to recreate on the grid. The player is free to start
at any dot and can only draw in straight lines to connect it with other adjacent
dots, while 1) the drawn shape must continue from where the previous line ended,
2) the same line cannot be used twice, and 3) the final shape must match the
provided shape perfectly. As the scene becomes more and more complex, most of
the accompanying instructions and feedback are provided by a robot companion
through text overlays within the real world; however, additional UI overlays are
used to convey minor and easily understandable information.
4 Mat Sanusi K. A. et al.
The target group of Yu and Mi are workers in the manufacturing or industrial
sector who have to collaborate with a YuMi robot on a regular basis. Therefore,
it is mainly intended for the workers who have less experience collaborating with
the YuMi.
For the measurement of impact, an experiment will be conducted in order to
measure the impact of this application. Participants will be randomly sorted into
individual groups; an experimental group, a control group with an explanatory
video, and a control group without external support. Experts measure the time
each participant took and the amount of mistakes they made during the process.
After 2-4 days, the following experiment will be conducted to analyze the long-
term learning progress of the application.
(a) Yu and Mi (b) Flowmotion
Fig. 1: Screenshots of the two prototypes
3.2 Flowmotion
The second prototype example focuses on the sports domain, which involves
more dynamic and complex movements than the human-robotic interaction. As
these pose challenges in the aforementioned domain, most of these movements
are repetitive and can be trained deliberately. Therefore, a full-body tracking
approach can be potentially used for effectively training these skills.
As a proof of concept, the second student group designed ”Flowmotion” (see
Figure 1b), a prototype that teaches the fundamentals of yoga poses. The system
uses the webcam camera to track the full-body of the learner while performing
Yoga movements and channels these movements into a virtual avatar. Similar
to the ”magic mirror” approach, it allows the avatar to move accordingly to the
learner, enabling the learner to reflect on his/her performance in real-time. Like
some exercise games such as Wii Fit, Just Dance 3, Dance Central 3, etc., the
students’ motivations were based on these examples.
The essential aspects of designing Flowmotion were the identification of basic
Yoga movements, common beginner mistakes, and deliverance of instructions
and feedback. As for the former two, with the help from the German Sport
Immersive training environments for psychomotor skills development 5
University Cologne, the students were able to collect the information they need
for designing their prototype. Additionally, the students need to ensure that the
selection of modalities for both instructions and feedback is suitable in a real-
world setting to prevent the learners from experiencing cognitive overload. As
such, visual and audio modalities are chosen for this system. Fundamentally, the
system implements a virtual coach that provides instructions for the learner to
imitate the movements. Subsequently, the virtual coach will prompt the learner
with visual and audio feedback when these movements are correctly or incorrectly
done. To further add immersion to the game, the students designed the game
environment to be more soothing and appropriate for the learner to perform the
Yoga tasks effectively. Thus, yoga studios were created.
Flowmotion is designed to help adults with no or less exercising experience
and instructors with their workout routines. The students envision that the
instructor would record their workout routines and share them online, enabling
the learners to perform the exercises remotely.
For the measurement of impact, an experiment will be conducted with two
groups - the experimental group and the control group. To further validate the
effectiveness, a workout video will be included for comparison. The results will
show whether there are significant improvements between both groups, and the
prototype can help the participants learn skills that can be transferable to other
exercises.
4 Future Work
For Yu and Mi, changing the digital robot presentation from the AR space to the
real world might be a path this prototype can be developed into. A physical robot
must be present with these changes, which can be extended through AR instead
of being entirely virtual. This will bring new challenges to the application, but
it can help convey an immersive learning environment more effectively to the
learner.
In the case of Flowmotion, further work could also include extensive exercise
routines. Besides Yoga poses, the prototype can be further explored in differ-
ent sport applications such as running or boxing, which involve more dynamic
movements and complex techniques. This will pose new challenges due to the
distinction of these activities but provides a significant step into improving psy-
chomotor training with immersive learning environments.
For the future work of the MILKI-PSY project, we propose using these stu-
dent prototypes and create a second iteration of each prototype in cooperation
with the students. Concerning the technical aspects of this development, we are
planning to replace the lower-tier sensors (i.e., mainly webcams) which were pre-
viously used in the student prototypes with hardware that is specialised for these
kinds of tasks (e.g., Microsoft Hololens 2, Azure Kinect). This way, we hope to
achieve better and more precise outputs due to the more reliable data.
These newly implemented prototypes can later be used to create an applica-
tion that will be publicised with the MILKI-PSY project.
6 Mat Sanusi K. A. et al.
5 Conclusion
In this paper, we introduced two student-made prototypes called ”Yu and Mi”
and ”Flowmotion”. These prototypes feature completely different domains, yet
both focus on developing psychomotor skills with immersive technologies. In
order to measure the impact of each prototype, we propose at least two test
groups, one experimental group, and a minimum of one control group. In the
final session, we can compare the results of these groups, which will show any
possible significant improvements made by using the prototype. In the future,
we want to take these student-made prototypes as a starting point for creating
and implementing the second iteration of said prototypes.
Acknowledgments A special thanks to the student groups ”Yu and Mi” and
”Flowmotion” from the BA6 Impact Games course of the Cologne Game Lab
for their contribution to this paper.
References
1. Barnes, R. W. (1987). Surgical handicraft: teaching and learning surgical skills. The
American journal of surgery, 153(5), 422-427.
2. Divekar, R. R., Drozdal, J., Chabot, S., Zhou, Y., Su, H., Chen, Y., Zhu, H., Hendler,
J. A., & Braasch, J. (2021). Foreign language acquisition via artificial intelligence
and extended reality: design and evaluation. Computer Assisted Language Learning,
1-29.
3. Echeverria, J., & Santos, O. C. (2021). KUMITRON: A Multimodal Psychomotor
Intelligent Learning System to Provide Personalized Support when Training Karate
Combats.
4. Herrington, J., Reeves, T. C., & Oliver, R. (2007). Immersive learning technologies:
Realism and online authentic learning. Journal of computing in Higher Education,
19(1), 80-99.
5. Kyan, M., Sun, G., Li, H., Zhong, L., Muneesawang, P., Dong, N., Elder, B., & Guan,
L. (2015). An approach to ballet dance training through ms kinect and visualization
in a cave virtual reality environment. ACM Transactions on Intelligent Systems and
Technology (TIST), 6(2), 1-37. https://doi.org/https://doi.org/10.1145/2735951
6. Schneider, J., Börner, D., Van Rosmalen, P., & Specht, M. (2016).
Can you help me with my pitch? Studying a tool for real-time auto-
mated feedback. IEEE Transactions on Learning Technologies, 9(4), 318-327.
https://doi.org/10.1109/TLT.2016.2627043
7. Song, P., Xu, S., Fong, W. T., Chin, C. L., Chua, G. G., & Huang, Z. (2012). An im-
mersive VR system for sports education. IEICE TRANSACTIONS on Information
and Systems, 95(5), 1324-1331.