This paper introduces Generative Multimodal Analysis (GMA), a novel method designed for utilizing Artificial Intelligence (GenAI) in the analysis of multimodal data derived from learning processes. The method is encapsulated in a systematic framework that integrates and optimizes GenAI technology with multimodal large language models (MLLMs) for application in multimodal learning analytics. The recent emergence and advancement of GenAI, particularly MLLMs, has opened new avenues for the automated interpretation and meaningful analysis of varied data sources. Current research in the field has sightseen diverse applications of GenAI in transforming learning and teaching practice. However, there is a noticeable gap in systematic methodologies for applying GenAI to scrutinize learning process data. This paper aims to bridge this gap by proposing the GMA method in the sphere of multimodal learning analytics with learning process data. In addition to the proposed methodological framework, this study also proposes an operational prototype for the practical implementation of GMA. This prototype serves as a tool for examining multimodal data in learning processes. To demonstrate the applicability and effectiveness of our proposed method, we conducted and presented a case study. Our approach offers essential guidance for learning scientists and educational technology application developers, reflecting the contemporary trends and needs in educational technologies. By providing a structured, innovative approach for employing GenAI in learning process data analysis, this study contributes significantly to the advancement of learning analytics methods.
The rapid evolution of educational technologies and learning sciences, particularly the groundbreaking
strides in artificial intelligence (AI) technology, has heralded an era where data-driven insights have
become valuable in enhancing learning processes. This intersection of AI with educational
methodologies is reshaping how information is analyzed, interpreted, and applied to improve teaching
strategies and learning outcomes. For instance, Luckin and Cukurova [
The advent of GenAI and its integration into educational contexts, particularly through multimodal
large language models (MLLMs), represents a significant leap in the domain of educational technology.
GenAI, a specialized subset of artificial intelligence that focuses on generating new content, has seen
rapid evolution over the past decade. A key development in this evolution has been the creation of
sophisticated multimodal large language models (MLLMs), such as OpenAI's GPT (Generative
Pretrained Transformer) series, which exemplify the advancements in natural language processing and
understanding. The release of ChatGPT, based on the GPT architecture, in November 2022 represented
a significant milestone [
Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). capabilities, offered a more interactive and intuitive way for users to engage with AI. Furthermore, GenAI opens new possibilities in content creation and data analysis. Accordingly, learning analytics, encompassing the collection, measurement, analysis, and reporting of data about learners and their contexts, could greatly benefit from the integration of GenAI. This integration enables more nuanced extraction of insights from diverse learning process data, thereby enhancing our understanding of student learning behaviors and outcomes.
This paper addresses the identified gap in comprehensive methodologies for GenAI in multimodal
learning analytics and proposes Generative Multimodal Analysis (GMA) as a structured approach to
employ GenAI effectively in this field. Such a methodology is essential due to the growing complexity
of learning environments and the variety of data they produce. As argued by Järvelä and Bannert [
Generative Multimodal Analysis (GMA) represents a comprehensive methodological framework designed to transform the approach of researchers and analysts in educational settings. This framework leverages the capabilities of GenAI to expedite and enrich the process of extracting and integrating verbal and nonverbal elements from learning process data. Furthermore, GMA extends its integration to include material objects and the environment, capturing how they are utilized by learners engaged in active interaction within the learning context. By harnessing the power of GenAI, GMA facilitates a more profound and holistic understanding of the multifaceted nature of learning environments, where verbal communication, nonverbal cues, physical objects, and the surrounding environment all play integral roles in the learning process.
Figure 1 provides a visual depiction of the Generative Multimodal Analysis (GMA) Framework,
illustrating its dynamic capabilities in leveraging generative AI for educational research. Within this
framework, generative AI is adept at producing various types of outputs, including 1) predefined events,
2) improvisational events, 3) detailed descriptions of the learning process, and 4) comprehensive
descriptions of the learning context. Crucially, as per the human-AI collaboration approach in research
suggested by Järvelä, Nguyen and Hadwin [
Furthermore, the information generated through the GMA Framework can be dissected and examined through various analytical lenses. These include a) process-oriented analysis, which focuses on the dynamics and phases of the learning process; b) quantitative modeling, offering a statistical perspective and uncovering patterns and correlations; and c) qualitative inquiry, which delves into the deeper, nuanced aspects of the learning environment and experiences. This multifaceted analytical approach allows for a comprehensive and multi-dimensional understanding of the learning process, harnessing the strengths of both AI and human analysis.
As for illustration, Figure 2 provides a demonstration of how the GMA is applied to automatically extract specific pre-defined events. In this case, the focus is on identifying and analysing the non-verbal posture states of learners engaged in a collaborative learning setting. The figure showcases the capability of GMA to discern and categorize various non-verbal cues, particularly the postures of learners, which play a significant role in understanding engagement, interaction dynamics, and overall effectiveness of collaborative learning processes. This example highlights the advanced analytical power of GMA in recognizing and interpreting subtle yet crucial aspects of learner behaviour in collaborative learning.
Figure 3 showcases the user interface of our newly developed Generative Multimodal Analysis (GMA) Toolkit for describing video data. This interface is specifically designed to facilitate direct interaction with video data, particularly focusing on observational data from learning processes. The example presented within the figure provides a detailed demonstration of how the toolkit can be utilized to analyze observational video data. Specifically, it illustrates an analysis generated by the toolkit from a video capturing a collaborative learning session. This visual representation highlights the toolkit’s capabilities in processing and interpreting complex, real-time learning environments, thereby offering valuable insights into the dynamics of collaborative learning.
future studies will significantly contribute to enhancing the utility and scalability of GMA, thereby advancing its role in educational technology, learning analytics, and learning sciences.
This research has been funded by the Research Council of Finland (aka. Academy of Finland) grants 350249, and the University of Oulu profiling project Profi7 Hybrid Intelligence - 352788.