Personalized feedback in psychomotor training often involves the use of sophisticated Machine Learning (ML)-based algorithms, requiring the utilization of cloud-based computational power for eficiency and scalability. By incorporating a cloud-based feedback system, learners may receive individualized feedback on their psychomotor performance in real-time or as summative analysis, allowing them to develop their abilities more eficiently. The integration of a cloud-based feedback mechanism into immersive learning environments is explored to improve the acquisition of psychomotor abilities. The essential components of the feedback system are discussed in this article, including data collection, analysis, and dissemination, as well as the obstacles and issues related to its implementation.
The development and mastery of psychomotor abilities is essential in a variety of fields,
including athletic training, healthcare, vocational education and in human-robot interaction.
Immersive Learning Environments (ILEs) have emerged as efective platforms for facilitating
psychomotor learning by providing learners with safe environments to practice and refine their
skills. However, to optimize the learning experience, it is essential to implement a blend of
real-time [
Immersive technologies, such as Virtual Reality (VR), Augmented Reality (AR), and Mixed
Reality (MR), integrate real and simulated environments to generate innovative artificial
experiences [
Psychomotor skills involve the interplay between motor skills and cognitive abilities for
performing physical activities such as catching a ball or playing an instrument [18]. In psychomotor
learning, teachers play a vital role by providing explanations, demonstrations, and evaluations
of skills. Their presence enables the identification and correction of movement errors, leading
to improved performance. Additionally, they have a positive impact on promoting human
physical health [19]. However, the traditional methods of teaching these abilities based on
repetition and teacher feedback [20] are often being complemented with immersive learning
environments [21, 22] to provide scalable and engaging feedback. Two communication
channels (visual and auditory) of the five human senses (visual [23], auditory [24], gustatory [25],
olfactory [26], and haptic [27]) have received the majority of the focus in multisensory feedback
systems, potentially hindering the eficiency of learning [ 28]. Receiving timely feedback allows
learners to gather information about their execution of movements and apply it efectively for
immediate adjustments [
A hands-on experience improves learners’ practical abilities, but dificulties such as the scalability of such courses, teaching big groups, providing feedback, and evaluating learning gains, provide substantial challenges [30]. Leveraging cloud computing capabilities can help overcome local processing limitations [31], providing access to extensive computational resources that are frequently necessary for the high-demand Artificial Intelligence ( AI) algorithms utilized in generating feedback during psychomotor learning. The utilization of cloud-based virtual learning environments has demonstrated promise in reducing the growing disparity in academic ability between rural and urban learners, enabling rural students to be more competitive academically [32]. By leveraging a cloud-based learning environment, instructors can extend the reflective activities of learners, overcoming constraints of face-to-face conversations and fostering reflective skill development [ 33]. This can be achieved by facilitating infrastructure for scalable feedback [34, 35], hosting of collaborative ILE sessions [36], provisioning of cloudbased media elements [37], and enabling of long-term persistence of educational materials [38]. Cloud-based feedback mechanisms can ofer diverse learning opportunities in ILEs [ 39] by generating personalized and interactive instructions [40], provisioning of formative and summative feedback [41], or enabling peer-based feedback [42]. To enable scalable transmission, storage, and analysis, distributed communication systems like Apache Kafka [43] are used to efectively manage the heterogeneous data of multimodal sensor applications. Diferent modalities are processed using data fusion strategies such as early fusion, late fusion, and cross-modality fusion [44]. Time series databases, like InfluxDB, are designed to handle and analyze large volumes of time series data, such as sensor readings [45].
In Immersive Learning Environments, the collection of sensor-based data plays a crucial role in accurately recording and assessing learner behavior to enhance their learning experience with accurate feedback. The proposed cloud-based feedback generation system, illustrated in Fig. 1, incorporates a comprehensive framework that eficiently collects sensor data from learners (1) through the utilization of a Unity-based frontend template. The gathered sensor data spans multiple modalities and can be used for movement tracking and other psychomotor performance metrics. This collected data is then transmitted to the backend (2) via a distributed messaging channel. The backend of the Cloud-supported feedback generation platform manages learner sessions and conducts learning analytics. Raw sensor data collected from learners is transmitted to the backend, where it is stored in a time series database (3) and analyzed to identify areas for improvement. Based on the information stored in the database, the Machine Learning (ML)-based Feedback Generation component (4) can analyze learner data and generate personalized feedback tailored to the unique context of the learners. Insights derived from the collected sensor data allow the system to identify patterns, trends, and potential areas of focus for personalized feedback. Currently, the feedback generating decision interface is enabled using a mocked Wizard-of-Oz method, allowing us to gather insights from the collected sensor data and observe patterns, trends, and potential areas of focus for giving individualized feedback. The analyzed information is subsequently sent to the frontend (5), enabling the delivery of targeted feedback instructions to learners (6) through one of the reusable IDMs provided by the frontend template (see Section 2.2).
Session 1 Session 2
...
Session n
Backend Admin Console
MILKI PSY Cloud Time-series
DB
3 InfluxDB 5 4 ML-based Feedback
Generation Distributed Messaging
Kafka 2
SSeennssoorsrs 1 IMPECT Frontend Template
Feedback instructions Learner Immersive Learning Environments (ILEs)
Frontend 1 Frontend 2
...
Frontend n 6 AR VR Sim
Our cloud-based platform provides a scalable and high-performance infrastructure to ofer feedback in ILEs, independent of the user count or complexity of the learning activities. The design and architecture of our infrastructure have been designed to accommodate various requirements of diverse use cases of ILEs, encompassing AR, VR, and simulated environments.
This work explores the potential for a cloud-based feedback system in ILEs to increase the development of psychomotor skills, which could enhance skill development and learning outcomes. The cloud-based feedback generation system has been implemented successfully; however, an in-depth evaluation remains necessary to determine its performance and eficacy. Our approach attempts to address the drawbacks of conventional teaching techniques by utilizing cloud computing to give feedback that is personalized for a large number of students. This research contributes to the field of Immersive Learning and inspires further exploration of cloud-based solutions for more engaging and efective ILEs.
Future work For future work, we plan to address a few crucial areas to further improve our cloud-based feedback mechanism. One of the most significant additions is the integration of an ML-based Feedback Generation component, which will automatically evaluate the learner data stored in the database and produce individualized feedback particularly adapted to the individual context of the learners. Following this, we will evaluate scalability in large-scale deployments to ensure the system’s eficacy and eficiency while serving a sizable user base.
To assure the validity and integrity of the obtained data for proper feedback creation, we will also investigate the accuracy and reliability of the sensors. Another key part is reviewing long-term assessment and skill advancement, with the goal of providing learners with continual feedback and insights into their skill improvement over time. Reducing feedback latency is crucial in ILEs to enhance learning results. Minimizing delay in feedback delivery allows for improved students’ feeling of presence and immersion in the learning process. Future research will focus on optimizing feedback generation and transmission speed to provide instantaneous, tailored feedback in psychomotor training within ILEs. Minimizing feedback latency for learner actions is crucial in ILEs, which further prompts research on end-to-end latency-aware feedback generation algorithms. This could include investigating edge computing approaches and establishing an optimal threshold for partitioning client-side and server-side responsibilities to minimize feedback latency. Furthermore, we intend to focus on enhancing the security and privacy features of our system to ensure comprehensive data protection in ILEs, including federated learning, advanced encryption, and robust data handling procedures.
The research leading to these results has received funding from the German Federal Ministry of Education and Research (BMBF) through the project “Multimodales Immersives Lernen mit künstlicher Intelligenz für Psychomotorische Fähigkeiten” (“MILKI-PSY”1) (grant no. 16DHB4015).
Reality medical training system for anatomy education, in: 2014 Science and Information Conference, IEEE, London, 2014, pp. 752–758. doi:10.1109/SAI.2014.6918271. [7] M. Akçayır, G. Akçayır, Advantages and challenges associated with augmented reality for education: A systematic review of the literature, Educational Research Review 20 (2017) 1–11. doi:10.1016/j.edurev.2016.11.002. [8] D. Fonseca, N. Martí, E. Redondo, I. Navarro, A. Sánchez, Relationship between student profile, tool use, participation, and academic performance with the use of Augmented Reality technology for visualized architecture models, Computers in Human Behavior 31 (2014) 434–445. doi:10.1016/j.chb.2013.03.006. [9] H.-M. Huang, U. Rauch, S.-S. Liaw, Investigating learners’ attitudes toward virtual reality learning environments: Based on a constructivist approach, Computers & Education 55 (2010) 1171–1182. doi:10.1016/j.compedu.2010.05.014. [10] A. Dengel, What Is Immersive Learning?, in: 2022 8th International Conference of the Immersive Learning Research Network (iLRN), IEEE, Vienna, Austria, 2022, pp. 1–5. doi:10.23919/iLRN55037.2022.9815941. [11] C. Jennett, A. L. Cox, P. Cairns, S. Dhoparee, A. Epps, T. Tijs, A. Walton, Measuring and defining the experience of immersion in games, International Journal of Human-Computer Studies 66 (2008) 641–661. doi:10.1016/j.ijhcs.2008.04.004. [12] B. Dalgarno, M. J. W. Lee, What are the learning afordances of 3-D virtual environments?: Learning afordances of 3-D virtual environments, British Journal of Educational Technology 41 (2010) 10–32. doi:10.1111/j.1467-8535.2009.01038.x. [13] N. Pellas, A. Dengel, A. Christopoulos, A Scoping Review of Immersive Virtual Reality in STEM Education, IEEE Transactions on Learning Technologies 13 (2020) 748–761. doi:10.1109/TLT.2020.3019405. [14] B. Peixoto, R. Pinto, M. Melo, L. Cabral, M. Bessa, Immersive Virtual Reality for Foreign Language Education: A PRISMA Systematic Review, IEEE Access 9 (2021) 48952–48962. doi:10.1109/ACCESS.2021.3068858. [15] L. Freina, M. Ott, A Literature Review On Immersive Virtual Reality In Education: State Of The Art And Perspectives., in: eLSE 2015, Bucharest, RO, 2015, pp. 133–141. doi:10.12753/ 2066-026X-15-020. [16] D. Hamilton, J. McKechnie, E. Edgerton, C. Wilson, Immersive virtual reality as a pedagogical tool in education: A systematic literature review of quantitative learning outcomes and experimental design, Journal of Computers in Education 8 (2021) 1–32. doi:10.1007/s40692-020-00169-2. [17] G. Makransky, G. B. Petersen, The Cognitive Afective Model of Immersive Learning (CAMIL): A Theoretical Research-Based Model of Learning in Immersive Virtual Reality, Educational Psychology Review 33 (2021) 937–958. doi:10.1007/s10648-020-09586-2. [18] E. J. Simpson, The Classification of Educational Objectives, Psychomotor Domain, ERIC Clearinghouse; Illinois Univ., Urbana, Washington, D.C., 1966. URL: https://eric.ed.gov/?id= ED010368. [19] S. A. Harris, P. T. Chivers, F. L. McIntyre, B. Piggott, M. Bulsara, F. H. Farringdon, Exploring the association between recent concussion, subconcussive impacts and depressive symptoms in male Australian Football players, BMJ Open Sport & Exercise Medicine 6 (2020) e000655. doi:10.1136/bmjsem-2019-000655. [20] A. J. Harrow, A Taxonomy of the Psychomotor Domain: A Guide for Developing Behavioral
Objectives, David McKay, New York, 1977. [21] K. A. Mat Sanusi, D. Majonica, L. Künz, R. Klemke, Immersive training environments for psychomotor skills development: A student driven prototype development approach, in: Proceedings of the First International Workshop on Multimodal Immersive Learning Systems (MILeS 2021), volume 2979, CEUR-WS, Online (Bozen-Bolzano, Italy), 2021, pp. 53–58. URL: https://ceur-ws.org/Vol-2979/paper7.pdf. [22] Y. M. Tang, K. Y. Chau, A. P. K. Kwok, T. Zhu, X. Ma, A systematic review of immersive technology applications for medical practice and education - Trends, application areas, recipients, teaching contents, evaluation methods, and performance, Educational Research Review 35 (2022) 100429. doi:10.1016/j.edurev.2021.100429. [23] C. Cruz-Neira, D. J. Sandin, T. A. DeFanti, R. V. Kenyon, J. C. Hart, The CAVE: Audio visual experience automatic virtual environment, Communications of the ACM 35 (1992) 64–72. doi:10.1145/129888.129892. [24] C. Summers, M. Jesse, Creating immersive and aesthetic auditory spaces in virtual reality, in: 2017 IEEE 3rd VR Workshop on Sonic Interactions for Virtual Environments (SIVE), IEEE, Los Angeles, CA, 2017, pp. 1–6. doi:10.1109/SIVE.2017.7938144. [25] A. S. Duggal, R. Singh, A. Gehlot, M. Rashid, S. S. Alshamrani, A. S. AlGhamdi, Digital Taste in Mulsemedia Augmented Reality: Perspective on Developments and Challenges, Electronics 11 (2022) 1315. doi:10.3390/electronics11091315. [26] N. Murray, O. A. Ademoye, G. Ghinea, G.-M. Muntean, A Tutorial for Olfaction-Based Multisensorial Media Application Design and Evaluation, ACM Computing Surveys 50 (2018) 1–30. doi:10.1145/3108243. [27] M. Mihelj, J. Podobnik, Haptics for Virtual Reality and Teleoperation, volume 64 of Intelligent Systems, Control and Automation: Science and Engineering, Springer Netherlands, Dordrecht, 2012. doi:10.1007/978-94-007-5718-9. [28] P. T. Kovacs, N. Murray, G. Rozinaj, Y. Sulema, R. Rybarova, Application of immersive technologies for education: State of the art, in: 2015 International Conference on Interactive Mobile Communication Technologies and Learning (IMCL), IEEE, Thessaloniki, Greece, 2015, pp. 283–288. doi:10.1109/IMCTL.2015.7359604. [29] B. Limbu, A. Vovk, H. Jarodzka, R. Klemke, F. Wild, M. Specht, WEKIT.One: A Sensor-Based Augmented Reality System for Experience Capture and Re-enactment, in: M. Schefel, J. Broisin, V. Pammer-Schindler, A. Ioannou, J. Schneider (Eds.), Transforming Learning with Meaningful Technologies, volume 11722, Springer International Publishing, Cham, 2019, pp. 158–171. doi:10.1007/978-3-030-29736-7_12. [30] J. Vykopal, P. Celeda, P. Seda, V. Svabensky, D. Tovarnak, Scalable Learning Environments for Teaching Cybersecurity Hands-on, in: 2021 IEEE Frontiers in Education Conference (FIE), IEEE, Lincoln, NE, USA, 2021, pp. 1–9. doi:10.1109/FIE49875.2021.9637180. [31] S. A. Osia, A. S. Shamsabadi, A. Taheri, H. R. Rabiee, H. Haddadi, Private and Scalable Personal Data Analytics Using Hybrid Edge-to-Cloud Deep Learning, Computer 51 (2018) 42–49. doi:10.1109/MC.2018.2381113. [32] S. Safdar, M. Ren, M. A. Z. Chudhery, J. Huo, H.-U. Rehman, R. Rafique, Using cloud-based virtual learning environments to mitigate increasing disparity in urban-rural academic competence, Technological Forecasting and Social Change 176 (2022) 121468. doi:10.1016/ j.techfore.2021.121468. [33] Y.-T. Lin, M.-L. Wen, M. Jou, D.-W. Wu, A cloud-based learning environment for developing student reflection abilities, Computers in Human Behavior 32 (2014) 244–252. doi: 10.1016/ j.chb.2013.12.014. [34] G. Riahi, E-learning Systems Based on Cloud Computing: A Review, Procedia Computer
Science 62 (2015) 352–359. doi:10.1016/j.procs.2015.08.415. [35] I. Khan, A. Pardo, Data2U: Scalable real time student feedback in active learning environments, in: Proceedings of the Sixth International Conference on Learning Analytics & Knowledge - LAK ’16, ACM Press, Edinburgh, United Kingdom, 2016, pp. 249–253. doi:10.1145/2883851.2883911. [36] T. T. De Back, A. M. Tinga, P. Nguyen, M. M. Louwerse, Benefits of immersive collaborative learning in CAVE-based virtual reality, International Journal of Educational Technology in Higher Education 17 (2020) 51. doi:10.1186/s41239-020-00228-9. [37] M. Herczeg, A. Ohlei, T. Reins, T. Schumacher, Ambient Learning Spaces: Constructing Timelines Through Distributed Collaborative Learning, in: 14th Annual International Conference of Education, Research and Innovation, Online Conference, 2021, pp. 3972– 3981. doi:10.21125/iceri.2021.0942. [38] Ł. Czekierda, K. Zieliński, S. Zieliński, Automated Orchestration of Online Educational Collaboration in Cloud-based Environments, ACM Transactions on Multimedia Computing, Communications, and Applications 17 (2021) 1–26. doi:10.1145/3412381. [39] R. Klamma, M. Slupczynski, MILKI-PSY Cloud: Multimodal Sensor Stream Processing Pipeline for LA in Psychomotor Education, in: Proceedings of the Second International Workshop on Multimodal Immersive Learning Systems (MILeS 2022), volume 3247, CEURWS, Toulouse, France, 2022, pp. 8–14. URL: https://ceur-ws.org/Vol-3247/paper1.pdf. [40] S. Williams, R. Enatsky, H. Gillcash, J. J. Murphy, D. Gracanin, Immersive Technology in the Public School Classroom: When a Class Meets, in: 2021 7th International Conference of the Immersive Learning Research Network (iLRN), IEEE, Eureka, CA, USA, 2021, pp. 1–8. doi:10.23919/iLRN52045.2021.9459371. [41] P. Vittorini, S. Menini, S. Tonelli, An AI-Based System for Formative and Summative Assessment in Data Science Courses, International Journal of Artificial Intelligence in Education 31 (2021) 159–185. doi:10.1007/s40593-020-00230-2. [42] A. Blankenship, L. Hawthorne, M. McCall, M. Simoneaux, Promoting self-assessment of psychomotor skills in ASN students through peer mentoring and deliberate practice, Teaching and Learning in Nursing (2023) S1557308723000471. doi:10.1016/ j.teln.2023.03.006. [43] B. R. Hiraman, C. Viresh M., K. Abhijeet C., A Study of Apache Kafka in Big Data Stream Processing, in: 2018 International Conference on Information , Communication, Engineering and Technology (ICICET), IEEE, Pune, 2018, pp. 1–3. doi:10.1109/ ICICET.2018.8533771. [44] W. C. Sleeman, R. Kapoor, P. Ghosh, Multimodal Classification: Current Landscape, Taxonomy and Future Directions, ACM Computing Surveys 55 (2023) 1–31. doi:10.1145/ 3543848. [45] F. A. Tsvetanov, Storing data from sensors networks, IOP Conference Series: Materials Science and Engineering 1032 (2021) 012012. doi:10.1088/1757-899X/1032/1/012012.