MILeS 2021 Workshop on Multimodal Immersive Learning Systems Proceedings of the 1st International Workshop on Multimodal Immersive Learning Systems At the Sixteenth European Conference on Technology Enhanced Learning: Technology- Enhanced Learning for a Free, Safe, and Sustainable World. Online (Bozen-Bolzano, Italy) September 20 – 24, 2021 Roland Klemke Fernando P. Cardenas- Khaleel Asyraaf Mat Sanusi Hernandez Daniel Majonica Miloš Kravčík Anja Richert Benjamin Paaßen Valérie Varney Ralf Klamma Tobias Keller Michal Slupczynski Jan Schneider Stefanie Klatt Daniele Di Mitri Mai Geisen George-Petru Ciordas-Hertel Tobias Baumgartner Gianluca Romano Nina Riedl 1 Please refer to these proceedings as: Roland Klemke, Khaleel Asyraaf Mat Sanusi, Daniel Majonica, Anja Richert, Valérie Varney, Tobias Keller, Jan Schneider, Daniele Di Mitri, George-Petru Ciordas-Hertel, Fernando P. Cardenas-Hernandez, Gian- luca Romano, Miloš Kravčı́k, Benjamin Paaßen, Ralf Klamma, Michal Slupczynski, Stefanie Klatt, Mai Geisen, Tobias Baumgartner, & Nina Riedl: Proceedings of the 1st International Workshop on Multimodal Multimodal Immersive Learning Systems. At the Sixteenth European Conference on Technology Enhanced Learning: Technology-Enhanced Learning for a Free, Safe, and Sustainable World. Online, Bozen-Bolzano, Italy, September 20-24, 2021, CEUR-WS.org/Vol-2979, ISSN 1613-0073. © 2021 for the individual papers by the papers’ authors. Copying is permitted for private and academic purposes. Re-publication of material from this volume requires permission by the copyright owners. Address of first editor: Khaleel Asyraaf Mat Sanusi Cologne Game Lab - Cologne University of Applied Sciences (TH Köln) Schanzenstr. 28, 51063 Köln, Germany ks@colognegamelab.de 2 Preface Advances in the related fields of wearable sensors, virtual/augmented reality, and artificial intelligence make it possible to connect these technologies into integrated learning solutions. The intersection of these fields of emerging tech- nologies is an area of many opportunities for innovative learning systems, but likewise a field of fuzzy expectations. With this workshop, we contribute to the systematic organisation of the field and to the advancement of solutions. The MILeS (Multimodal Immersive Learning Systems) 2021 workshop was organised in the context of the German BMBF-funded research project MILKI- PSY (Multimodal Immersive Learning with Artificial Intelligence for Psychomo- tor Skills), which aims to develop AI-supported, data-intensive, multimodal, immersive learning environments for the independent learning of psychomotor skills. This leads to a cross-domain approach that makes it possible to record the activities of experts in a multimodal manner and to use these recordings as blueprints for learners. With the help of AI-based analysis, learning progress is to be supported by automated error detection and automatically generated, in- dividual feedback. This creates holistic, innovative environments for cultivating psychomotor skills, in which personalized AI support enables individual learning processes based on complex data analyzes. With the interdisciplinary workshop, we brought together experts and prac- titioners from technology-enhanced learning and educational application of tech- nologies to collect ideas, requirements, best practices, and example cases in the intersection of Artificial Intelligence, Multimodal Systems, Immersive Systems, and their application into actual education. With this context in mind, in this first edition of the International Workshop on Multimodal Immersive Learning Systems (MILeS 2021), we have compiled eight research studies that go from early stages of development to present empirical studies where novel experimen- tal designs, theoretical contributions, and practical demonstrations. MILeS 2021 took place on September 21st, 2021, and was run virtually in conjunction with the Sixteenth European Conference on Technology Enhanced Learning (EC-TEL 2021). Following are the core topics for this workshop: – Mobile, wearable, and pervasive technologies – Sensors, sensor networks, and Internet of Things – Augmented reality, virtual reality, and mixed reality – Artificial intelligence The website of the workshop can be found here https://milki-psy.de/miles- workshop/. Contributions A peer-reviewed process was carried out to select the workshop papers. At least three members of the Program Committee with expertise in the area reviewed each paper. As a result, the following eight submissions were accepted, which discuss ideas and progress on several interesting topics: 3 – Paaßen et al. sketched a machine learning approach as an early significant step into providing feedback to the learners in the domains of running and human-robotic interaction. The authors evaluated how movements can be compared to highlight the variations between the student and expert move- ments. – Cardenas et al. proposed the existing Multimodal Learning Analytics Pipeline to be applied in the domains of running and human-robotic interaction through the process of data collection, storage, annotation, preparation, and exploitation. Suitable sensors that can be potentially applied in the two ap- plication cases are also suggested. – Slupczynski et al. proposed a cloud-based architecture as a basis for an AI-based multimodal training and learning environment. The article dis- cusses some related work on service-based architectures for machine learning, blockchain, and learning analytics. – Quin et al. introduced a prototype that utilizes holographic technologies to foster natural interactions between teachers and students in an online class- room setting. The paper described the system architecture of the prototype, two different 3D modeling techniques, and students’ experience. – Keller et al. presented an approach on how to develop a design framework for an augmented reality-based training system for the acquisition of psychomo- tor skills, more specifically how to teach workers to properly collaborate with a robot (i.e., human-robot interaction) and how to keep them motivated along the way. This is done by exploring the results of three interdisciplinary workshops. – Dikken et al. proposed a framework for a handwriting learning system that compares a learner trace to an expert trace via dynamic time warping and then performs an error classification based on the remaining deviation com- pared to the expert. – Mat Sanusi et al. presented two student prototypes in different application areas related to multimodal learning of psychomotor skills in immersive en- vironments. The prototypes are presented in detail and possible future op- timisation processes within the framework of MILKY-PSY are discussed in order to ultimately determine the feasibility and advantages of psychomotor learning with AI-based on both use cases. – Geisen et al. proposed a study design to enable subjects to optimize their psychomotor training performance with real-time feedback in an immersive training environment. In the context of this paper, a squat exercise was selected. The authors aim to compare different feedback methods that can be given in real-time and identify the most suitable feedback for motion learning and optimization. Conclusions MILeS 2021 workshop aimed at gathering new insights around the use of Ar- tificial Intelligence, Multimodal Systems, and Immersive Systems for education and learning leveraging multimodal data sources. With the broad spectrum of 4 submissions received, the first edition of the MILeS workshop showed an inter- esting perspective on the field of multimodal immersive learning and highlights the relevance of this emerging multidisciplinary field of research, that connects expertise in computer science, artificial intelligence, human-computer interac- tion, sensor-based systems, educational science, technology-enhanced learning, and game design. We expect that the first edition of the MILeS workshop sets the foundation for a continued and growing series of workshops and publications in the field. Acknowledgments The MILeS 2021 chairs would like to thank the authors for their submissions and the EC-TEL workshop chairs for their advice and guidance during the MILeS workshop. The MILeS chairs also served as Program Committee that reviewed high-quality reviews for the received submissions. The following project has supported the organisation of the MILeS 2021 workshop: Multimodal Immersive Learning with Artificial Intelligence for Psy- chomotor Skills (MILKI-PSY) funded under the grant number: 16DHB4013 by the Federal Ministry of Education and Research (BMBF). Cologne, October 4th, 2021 Roland Klemke, Khaleel Asyraaf Mat Sanusi 5 Program Committee Members Roland Klemke Cologne University of Applied Sciences (TH Köln) and Open University of the Netherlands (OUNL) Khaleel Asyraaf Mat Sanusi Cologne University of Applied Sciences (TH Köln) Daniel Majonica Cologne University of Applied Sciences (TH Köln) Anja Richert Cologne University of Applied Sciences (TH Köln) Valérie Varney Cologne University of Applied Sciences (TH Köln) Tobias Keller Cologne University of Applied Sciences (TH Köln) Jan Schneider Leibniz Institute for Human Development and Educational Information (DIPF) Daniele Di Mitri Leibniz Institute for Human Development and Educational Information (DIPF) George-Petru Ciordas-Hertel Leibniz Institute for Human Development and Educational Information (DIPF) Fernando P. Cardenas-Hernandez Leibniz Institute for Human Development and Educational Information (DIPF) Gianluca Romano Leibniz Institute for Human Development and Educational Information (DIPF) Miloš Kravčı́k German Research Center for Artificial Intelligence (DFKI) Benjamin Paaßen 6 German Research Center for Artificial Intelligence (DFKI) Ralf Klamma RWTH Aachen Michal Slupczynski RWTH Aachen Stefanie Klatt German Sport University Cologne (DSHS) Mai Geisen German Sport University Cologne (DSHS) Tobias Baumgartner German Sport University Cologne (DSHS) Nina Riedl German Sport University Cologne (DSHS) 7 Table of Contents Teaching psychomotor skills using machine learning for error detec- tion Benjamin Paaßen, Miloš Kravčı́k . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 Multimodal Immersive Learning with Artificial Intelligence for Robot and Running application cases Fernando P. Cardenas-Hernandez, Gianluca Romano, Hendrik Drachsler . 15 MILKI-PSY Cloud: Facilitating multimodal learning analytics by ex- plainable AI and blockchain Michal Slupczynski, Ralf Klamma . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 HoloLearn: Using holograms to support naturalistic interaction in vir- tual classrooms Tristan Quin, Bibeg Limbu, Michel Beerens, Marcus Specht . . . . . . . . . . . . . . . 29 WIP: Development of a design framework for the provision of multi- modal content in an AR-based training system for the acquisition of psychomotor skills Tobias Keller, Valérie Varney, Anja Richert . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 Expert Distribution Similarity Model: Feedback methodology for non- imitation based handwriting practice Olivier Dikken, Bibeg Limbu, Marcus Specht . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 Immersive training environments for psychomotor skills development: A student driven prototype development approach Khaleel Asyraaf Mat Sanusi, Daniel Majonica, Lukas Künz, Roland Klemke 53 Real-time visual feedback on sports performance in an immersive training environment: Presentation of a study concept Mai Geisen, Tobias Baumgartner, Nina Riedl, Stefanie Klatt . . . . . . . . . . . . . 59