© 2023 for the individual papers by the papers’ authors. Copying permitted for private and academic purposes. Re-publication of material from this volume requires permission by the copyright owners. Over the last several years, Multimodal Learning Analytics (MMLA) has brought together diverse fields that combine educational, computational, psychological, and related research into how people learn and how this complex process can be supported with multichannel and multimodal technology.
The MMLA community strives to untangle complex learning dynamics by analysing multimodal data from multiple devices, such as sensors, cameras, and Internet of Things tools, as well as the expert and self-reported data. Similarly, the MMLA community uses multimodal interfaces to explore new ways of learning that allow learners to interact with learning material and stimulate them to use the psychomotor and afective domains of learning besides the purely cognitive domain. The MMLA community promotes research that tries to make sense of complex educational data that involve multiple interaction modalities, people, and learning spaces. Understanding and optimising learning traces from the real world requires renewed connections between technology, learning, and design; and building upon the ongoing and previous work from Learning Analytics & Knowledge and related data-driven communities.
The work of the MMLA community is far from complete. MMLA needs to develop theories about analysing human behaviours during diverse learning processes across spaces and to create valuable tools that could augment the capabilities of learners and instructors. These tools and practices must be designed and implemented ethically to provide value and equity for all learners.
The COVID-19 pandemic and the rapid global shift to online learning have heavily challenged
the research and practice for this research field and educational practices. The SOLAR’s Special
Interest Group on Multimodal Learning Analytics Across Spaces (CROSSMMLA SIG) has tried to
foster the exchange of knowledge and peer learning activities by organising the CROSSMMLA
workshop series at LAK conferences, editing the MMLA Handbook to be published soon with
Springer [
The CROSSMMLA workshop explored how multimodal learning analytics can efectively
capture students’ learning experiences across diverse learning spaces (online, in presence, in
the field) and diverse learning domains (psychomotor, cognitive, and emotional). The core
challenge of MMLA is to capture these interactions meaningfully so that they can be translated
as part of formative assessment in real-time and as post-reflective reviews [
In this edition of the CROSSMMLA workshop, special attention was put on “leveraging multimodal data for generating meaningful feedback”. We have invited all the prospective authors to reflect on the possible ways their MMLA solutions and approaches can generate meaningful feedback for teachers and learners. The workshop served as a forum to exchange ideas on how we, as a community, can use our knowledge and experiences from CROSSMMLA to design new tools to analyse evidence from multimodal and multichannel data. How can we extract meaning from these increasingly fluid and complex data generated from various transformative learning situations, and how can best feedback on these analyses’ results positively support the learning processes.
The dimensions and contexts of MMLA are complex and layered, providing researchers with
multiple challenges [
In the first paper, Kawashima explores the potential of using machine learning methods for multimodal learning analytics (MMLA) and feedback generation. By integrating behavioural measurements and content analysis of learning materials, the aim of MMLA should be to estimate learners’ states and generate adaptive feedback. The paper discusses the recent trend of representation learning and its potential for integrating multimodal behavioural and contextual data. Kawashima asserts that the collaboration between learning analytics and machine learning can lead to a new framework of feedback loops in MMLA.
The second paper from Li, Majumdar, Yang and Ogata proposes a learner model-based feedback model that leverages students’ daily life activity data to provide multi-dimensional feedback on a contextual activity, self-direction management, and skill assessment. The feedback model is implemented in two learning dashboards for English learning and physical activity contexts, and the potential efects of the feedback model on student engagement and skill improvement are demonstrated through two case studies in K-12 settings. The results indicate that K-12 students can continuously engage in learning and physical activities and regularly take feedback with the learning dashboard support. Future research directions include investigating the transferability of SDS in other contexts and improving the feedback model from a learner’s lifestyle perspective.
The third paper from Kwon et al. proposes an advanced ontological knowledge structure called Knowledge Objects (KOs) to enhance the process of real-time knowledge sharing. KOs consist of metadata linked to multiple data streams in examining a specific task. They allow multiple data streams to be combined and analyzed in real-time. The authors present the development of KOs as a solution to adopting modern technology and techniques for real-time knowledge sharing. The proposed KOs were tested on a model that identifies moments in which a refrigerator is opened, and a sink faucet is turned on, achieving 91.7% recall, 58.3% precision, and 80% average precision.
The fourth paper by Wang, Ruis and Shafer proposes a method called Qualitative Parameter Triangulation (QPT) to address the challenges of data fusion and parameterization in multimodal learning analytics. QPT generates optimized parameter values for event-based, process-oriented, and connection-structured multimodal learning models. The key concept of engaging qualitative researchers in the loop ensures interpretive alignment, providing potential for closing feedback loops with other stakeholders in the multimodal study. Future work includes testing the eficacy of QPT using empirical data and exploring its applicability to other ifelds beyond learning analytics.
The fith paper from Chejara et al. discusses the challenges and potential solutions of conducting (MMLA) research in physical settings. MMLA has enabled researchers to understand learning through a new perspective by using sensors to collect data such as audio, video, eye-gaze, and physiological indicators. The paper presents some open challenges, such as noisy data from the classroom, and proposes potential solutions. The authors also mention that their solutions are the result of general engineering and not MMLA-specific. Overall, this paper provides insights into the challenges faced by MMLA researchers and how they have addressed them.
The sixth and last paper from Schneider et al. presented a novel approach to assess human performance and expertise levels based on sensor data. The study collected accelerometer data of a domain expert and a novice performing tasks in semi-constrained settings. The results showed that expert performances are smoother, contain fewer irregularities, and have consistently uniform patterns than novice performances. This approach can be used in various ifelds, such as sports or dance, to distinguish between experts and novices quickly.
Hiroaki Kawashima . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Modeling Feedback for Self-Direction Skills in K-12 Educational Settings with Learning and Physical Activity Data
Huiyong Li, Rwitajit Majumdar, Yuanyuan Yang, and Hiroaki Ogata . . . . . . . . . . . . . . . . . . 12 Qualitative Parameter Triangulation: A Formulated Approach to Parameterize Multimodal Models
Yeyu Wang, Andrew R. Ruis and David Williamson Shafer . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 Multimodal Learning Analytics research in the wild: challenges and their potential solutions