onlinefrom Utrecht,TheNetherlands
IntelligenceinEducation atthe22ndInternationalConferenceonArtificial IntelligenceinEducation Please refer to these proceedings as:
Daniele Di Mitri, Roberto Martinez-Maldonado, Olga C. Santos, Jan Schneider, Khaleel Asyraaf Mat Sanusi, Mutlu Cukurova, Daniel Spikol, Inge Molenaar, Michail Giannakos, Roland Klemke & Roger Azevedo (eds.): Proceedings of the 1st International Workshop on Multimodal Arti cial Intelligence in Education. At the 22nd International Conference on Arti cial Intelligence in Education. Online, Utrecht, The Netherlands June 14, 2021, CEUR-WS.org/Vol-2902, ISSN 1613-0073. © 2021 for the individual papers by the papers' authors. Copying permitted for private and academic purposes. Re-publication of material from this volume requires permission by the copyright owners.
Address of rst editor: Daniele Di Mitri DIPF | Leibniz Institute for Research and Information in Education Rostocker Str. 6, 60323 Frankfurt am Main, Germany. dimitri@dipf.de
Learning is a complex multidimensional process that involves multiple sources of
data. In current Arti cial Intelligence (AI) in Education research, however, many
modalities of data are neglected to a large extent. Traditionally, the AI-based
systems designed for education, typically Intelligent Tutoring Systems (ITS),
rely on click-stream data generated from computer interaction with a mouse
and keyboard or screen interaction in the case of mobile devices. The variety of
interactions, multi-sensor devices and multimodal data can provide a more
detailed digital representation of the learner. There is an increasing interest in the
data-driven educational research communities in multimodal and multi-sensor
interaction methods as an opportunity. The physiological sensors can provide a
wealth of information that can be used further to contextualise the learning
performance or strategies [
Existing studies using Multimodal AI for education approaches can be
divided into two groups. The rst group aims at extending computer-based
learning activities with physical sensors for tracking learners' behaviour, including,
for example, learner's eye-gaze, facial expressions or physiological responses such
as heart rate or neural activation (e.g. [
In AI, there is a philosophical discussion on whether AI systems are set to augment human abilities or replace them entirely. This discussion carries over to Multimodal AI in Education, across the degree of intervention that the AI assumes in the learning processes and to what extent it is set to replace human support and feedback.
On the \Multimodal augmentation" side, the AI algorithms are used for
nding patterns and regularities in the data, which are then communicated to human
actors in the learning process. This approach is advocated by the \CrossMMLA"
community { learning across physical and digital spaces using multimodal
learning analytics, a particular interest group within the Society of Learning Analytics
Research [
The \intelligent tutoring" aims instead to the automatic provision of automatic feedback using intelligent and autonomous agents has been a longstanding discussion and topic of investigation within the Arti cial Intelligence in Education (AIED) community. Although only recently, the community has started to look into more diverse data sources alternative to learner-computer interaction.
With this context in mind, in this rst edition of the International Workshop on multimodal Arti cial Intelligence in Education (MAIEd 2021), we have compiled ten research studies that go from early stages of developments to present empirical studies where novel experimental designs, theoretical contributions and practical demonstrations. MAIEd 2021 took place on June 14th, 2021 and was run virtually in conjunction with the 22nd International Conference on Arti cial Intelligence in Education (AIED 2021).
In particular, at the MAIEd workshop, we have discussed which scienti c, state-of-the-art ideas and approaches are being pursued and which impacts we expect on educational technologies and education, targeting the intersection of these two elds of AI and multimodal interaction. We tried to advance state of art in theories, technologies, methods, and knowledge towards the development of multimodal intelligent tutors with the following workshop topics: { Multimodal Intelligent Tutoring Systems or User Interfaces { Multimodality in Augmented, Virtual, and Mixed Reality { Multimodal Learner Modeling and A ective Computing { Adaptive Feedback, Guidance, and Process in Multimodal Learning { Arti cial Intelligence for Learning Analytics { Big Data-driven Visual Analytics for Learning { Error detection and classi cation for multimodal data { Cognitive Load in Multimodal Interaction with Intelligent Tutoring Systems { Explainability, Trust, and Safety in Multimodal Intelligent Tutoring Systems { Multimodal data for Self-Regulated learning
The website of the workshop can be found here https://maied.edutec.science/. A video recording of the event is avaialble upon request.
A peer-reviewed process was carried out to select the workshop papers. At least three members of the Program Committee with expertise in the area reviewed each paper. As a result, ten submissions were accepted (out of 13 submissions), which discuss ideas and progress on several interesting topics, such as multimodal analytics for a ective computing, multimodal intelligent tutoring, self-regulated learning, multimodal intelligence augmentation.
Jiang et al. investigate if an ITS gives supportive, empathetic, or motivational feedback messages to the learner to alter the learner's emotional state and detect the change.
Huang, Fridolin and Whitelock carry a comparative analysis of holographic intelligent agents by analysing nine `Holographic AIs' characters. They derive various design dimensions and principles for future research.
Zhou, Wannapon and Cukurova explore self-regulated learner clusters' engagement behaviours at individual, group and cohort activities. Their study analyses the relationship between students' SRL competence and their learning engagement behaviours observed in multimodal data. The results revealed that students with di erent SRL competence clusters might exhibit di erent behaviours in individual, group, and cohort level activities.
Lim and Leinonen propose an experimental design for fostering creativity with AI in multimodal and sociocultural learning environments. In the proposed Creative Peer System, humans and machines learn from each other in a multimodal learning environment and develop original artefacts.
Ronda et al. focus on simulated teamwork practice and how multimodal data processing can be performed to identify if the arousal levels matched the teachers' expectations regarding the students' emotional situation in the di erent phases in which the teachers' teamwork practice can be divided according to the instructional design.
Yang, Cukurova and Porayska-Pomsta focus on dyadic joint visual attention interaction in face-to-face collaborative problem-solving at K-12 Maths Education. Their results indicate that the multimodal approach can bring more insights into students' problem-solving. In addition, they propose a method for capturing gaze convergence by considering eye xations, eye blinks, and the overlapping time between two eye gazes.
Echeverria and Santos introduce KUMITRON, a multimodal psychomotor intelligent learning system that can provide personalised support when training karate combats.
Howell-Munson et al. move preliminary steps towards detecting proactive and reactive control states during learning with fNIRS brain signals. They distinguish between proactive and reactive control using fNIRS brain imaging in a controlled continuous performance task. They also propose integrating the fNIRS datastream with the ITS to create a multimodal system to detect the user's cognitive state and adapt the environment to promote better learning strategies.
Gupta et al. focus on multimodal and multi-task stealth assessment for re ection-enriched game-based learning. They present a stealth assessment framework that takes as input multimodal data streams (e.g., game trace logs, pre-test data, natural language responses to in-game re ection prompts) to predict posttest scores and written re ection depth scores jointly.
Lee-Cultura, Sharma and Giannakos propose a multimodal AI agent to support students' motion-based educational gameplay. The AI agent identi es and delivers appropriate feedback mechanisms to support a student's play learning experience. A Dashboard visualises the measurements to keep teachers informed of a student's progress.
MAIEd 2021 workshop aimed at gathering new insights around the use of Arti cial Intelligence (AI) systems and autonomous agents for education and learning leveraging multimodal data sources. It built upon the CrossMMLA workshop series at the Learning Analytics & Knowledge Conference and called for new empirical studies, even if in their early stages of development. As a result, ten papers have been accepted for presentation at the workshop that consists of novel experimental designs, theoretical contributions, and practical demonstrations which can prove the use of multimodal and multi-sensor devices \beyond mouse and keyboard" in learning contexts with the purpose of automatic feedback generation, adaptation, and personalisation in learning. Through the organisation of this rst edition of the workshop, we sought to engage the scienti c community in opening up the scope of AI in Education towards novel and diverse data sources.
The MAIEd 2021 chairs would like to thank the authors for their submissions and the AIED workshop chairs for their advice and guidance during the MAIEd workshop. The MAIEd chairs also served as Program Committee that reviewed high quality reviews for the received submissions.
The following project has partially supported the organisation of the MAIEd 2021 workshop: \INTelligent INTra-subject development approach to improve actions in AFFect-aware adaptive educational systems" INT2AFF funded under Grant PGC2018-102279-B-I00 (MCIU/AEI/FEDER, UE) by the Spanish Ministry of Science, Innovation and Universities, the Spanish Agency of Research and the European Regional Development Fund (ERDF).
Han Jiang, Zewelanji Serpell, and Jacob Whitehill . . . . . . . . . . . . . . . . . . . . . . 7
Xinyu Huang, Fridolin Wild, and Denise Whitelock . . . . . . . . . . . . . . . . . . . . 17
Qi Zhou, Wannapon Suraworachet, and Mutlu Cukurova . . . . . . . . . . . . . . . 27
ativity with Arti cial Intelligence in Multimodal and Sociocultural
Jeongki Lim and Teemu Leinonen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
Chiao-Wei Yang, Mutlu Cukurova, and Kaska Porayska-Pomsta . . . . . . . 60
Jon Echeverria and Olga C. Santos . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
Alicia Howell-Munson, Deniz Sonmez Unal, Erin Walker, Catherine Arrington, and Erin Solovey . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
Anisha Gupta, Dan Carpenter, Wookhee Min, Jonathan Rowe, Roger Azevedo, and James Lester . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
Serena Lee-Cultura, Kshitij Sharma, and Michail Giannakos . . . . . . . . . . 102