©2023 for the individual papers by the papers’ authors. Copying is permitted for private and academic purposes. Re-publication of material from this volume requires permission from the copyright owners.
Khaleel Asyraaf Mat Sanusi Cologne Game Lab - Cologne University of Applied Sciences (TH Köln) Schanzenstr. 28, 51063 Cologne, Germany ks@colognegamelab.de The third edition of “The Multimodal Immersive Learning Systems”, marks a significant milestone, thanks to the promising contributions from some of the leading authors in the domain of multimodal immersive learning and sensors. It would be amiss to not reflect on the previous editions and look back at the tumultuous landscape that has been overcome. The second edition of MILeS 2022 was held in Toulouse, France, with 9 papers across three core themes. In comparison, the third edition of MILeS 2023 which was held in Aveiro, Portugal, accepted 10 high-quality papers, across four themes. This edition of MILeS also conferred the first “Sir Roger Bannister” award for excellence in the dissemination of science to Wicaksono Febriantoro from University College London.
The four main themes from the accepted papers encompass infrastructures for guidance and feedback, the application use cases, providing support in collaborative tasks, and new possible directions for this research field. Concerning the infrastructures for guidance and feedback, Slupczynski et al. explain how personalized feedback in psychomotor training can be produced with sophisticated Machine Learning-based algorithms, requiring the utilization of cloud-based computational power for efficiency and scalability. They later explain how by incorporating a cloud-based feedback system, learners may receive individualized feedback on their psychomotor performance in real-time or as summative analysis. This allows learners to develop their skills more efficiently. The paper also discusses the essential components of the feedback system, including data collection, analysis, and dissemination, as well as the obstacles and issues related to its implementation. In the same theme, Romano et al. discuss three main limitations of Intelligent Tutoring Systems (ITS) for psychomotor skill training. First, the feedback provided is insufficient and there is lack of evidence in terms of long-term studies to support the effectiveness of the presented systems. Second, often the underlying model is evaluated instead of the whole ITS. Third, and consequently, task, student, and teacher models are not explicitly defined. Thus, the paper presents a hierarchical method of organizing exercises of domain-specific and generic skills which can be used to recommend personalized workouts to improve psychomotor skill development.
In this edition of MILeS, three use case papers discuss systems that provide feedback to learners. The work of Kotte et al. proposes a pioneering strategy to provide real-time feedback on posture during fitness exercises using computer vision methods, allowing for instant self-correction and motivation, even without professional guidance. The proposed system utilizes a versatile learning framework to analyze live expert demonstrations or recorded video content. The system delivers immediate feedback to rectify posture by collecting comprehensive tracking data. To demonstrate the usefulness of the system, the authors benchmarked it to professional fitness videos and evaluated it with five inexperienced participants. The results show a positive reaction from the participants, suggesting improvements to the user interface. Similarly, the contribution of Geisen et al. presents a feedback system where both experts and novices can optimize and learn dance choreography and internalize it to improve dance performance. The authors present a concept study design for the use of real-time visual feedback in dance classes, specifically for facilitating learning of dance choreography. The work of Mat Sanusi et al. evaluates IMPECT, a training toolkit designed for teaching psychomotor skills with immersive learning environments, on two distinct scenarios: human-robot interaction and dancing. The presented evaluation gathers survey data to assess the system's usability and incorporates participant suggestions, which are subsequently analyzed and discussed. The results of the evaluation demonstrate the training toolkit's potential applicability across diverse psychomotor domains.
The new thematic addition to this year’s MILeS is the collaborative multimodal immersive learning systems. Febriantoro and Cukurova attempt to integrate physiological data with verbal and non-verbal indicators of a generalized competence model of Collaborative Problem Solving (CPS) in small groups, to potentially further improve the detection of cognitive and affective aspects of CPS. The authors discuss the beneficial implication of doing so on the evaluation of collaboration which challenges researchers to date. Similarly, Zhou and Cukurova propose a Multimodal Learning Analytics (MMLA) framework for evaluating collaborative learning at both individual and group levels. In their framework, the authors discuss the need for holistic investigations at multiple dimensions to analyze and support collaboration, which, they argue, requires meaningful interactions among learners at cognitive, social, emotional, and regulatory levels. This is also reflected in Hyperchalk, an open-source online collaborative whiteboard tool developed by Gombert et al. which also collects trace data to study the collaborative process. The authors reflect on the multi-dimensionality of a collaborative process and discuss the integration of voice and text chat features into their application and data collection. Last but certainly not least, two paper contributions provide insights into the future directions of multimodal immersive learning systems. The contribution of Schneider et al. extends the previous work to help distinguish expert and novice performance by simply observing the sensor data without having to understand nor apply models to the sensor signal. Their method consists of plotting the sensor data and identifying uniformities in both novice and expert data. The authors solidify that, with the help of sensors, expert performances are smoother, contain fewer irregularities, and have consistently uniform patterns than novice performances. They test the extended methodology on the same data set from their previous five cases, namely running, bachata dance, salsa dance, tennis swings, and football penalty kicks, pointing out this assertion. An interesting contribution from Cardenas-Hernandez et al. explores the motivational aspect of the learner, which essentially plays a part in the improvement of psychomotor skills in the case of running. The authors investigate and describe a number of mental factors that running experts consider crucial and suggest common approaches to assess them. Moreover, they review some psychological theories and frameworks that can guide research in this field. The success and the potential of MILeS is evident by the quality of papers, ideas, participants, and the passionate discussions that ensued during the workshop. We may, individually, stand at different summits but face the same objective that the third international workshop on Multimodal Immersive Learning Systems echoed, i.e., sharing state-of-the-art ideas on how to push forward the field of multimodal and immersive systems that support the learning process.
Khaleel, Bibeg, Jan, & Milos
Cologne University of Applied Sciences (TH Köln)
University of Duisburg-Essen (UDE)
Leibniz Institute for Human Development and Educational Information (DIPF)
German Research Center for Artificial Intelligence (DFKI)
Cologne University of Applied Sciences (TH Köln) & Open University of the Netherlands (OUNL) Program Committee Members
Cologne University of Applied Sciences (TH Köln)
Cologne University of Applied Sciences (TH Köln)
Leibniz Institute for Human Development and Educational Information (DIPF)
Leibniz Institute for Human Development and Educational Information (DIPF)
Leibniz Institute for Human Development and Educational Information (DIPF)
German Research Center for Artificial Intelligence (DFKI)
German Sport University Cologne (DSHS)
German Sport University Cologne (DSHS)
German Sport University Cologne (DSHS)