YarnSense: Automated Data Storytelling for Multimodal Learning Analytics Gloria Milena Fernández-Nieto1,∗ , Vanessa Echeverria1,3 , Roberto Martinez-Maldonado1 and Simon Buckingham Shum2 1 Monash University 2 University of Technology Sydney 3 Escuela Superior Politecnica del Litoral, Guayaquil, Ecuador Abstract Professional development and training often require students to reflect on their performance, especially recalling the mistakes they have made in safe training environments, but these can occur in rapidly evolving and busy environments where key actions are often missed. Promisingly, rapid improvements in wearable sensing technologies are opening up new opportunities to capture large amounts of multimodal behaviour data that can serve as evidence to support student reflection about their performance. How- ever, while some preliminary research has highlighted the potential of analysing such data to identify interesting patterns, less work has focused on the problem of automatically communicating meaningful and contextualised data and insights to end-users. Based on the notion of data storytelling as a means of extracting actionable insights from data, this paper presents YarnSense, an architecture to automatically generate data stories with the intention of supporting student reflection and learning. YarnSense maps low-level sensor data to the pedagogical intentions of teachers, bringing human instructors into the data analysis loop. We illustrate this approach with a reference implementation of the system and an in-the-wild study in the context of immersive simulation in healthcare. Keywords Data Storytelling, multimodal data, data visualisation, sensor data 1. Introduction Learning by doing is essential in sectors like emergency response [1], safety training [2], and healthcare [3], where professionals gain knowledge through practical experiences, including bodily interactions and emotional responses [4]. However, capturing critical events or errors during fast-paced training scenarios is challenging. Integrating digital technologies and sensing devices in physical learning spaces offers a solution to improve teaching and learning[5]. These technologies, including infrared sensors, video and audio recorders, and wearables, capture multimodal behaviour data in real time , Joint Proceedings of LAK 2024 Workshops, co-located with the 14th International Conference on Learning Analytics and Knowledge (LAK 2024), Kyoto, Japan, March 18-22, 2024. Envelope-Open gloriamilena.fernandeznieto@monash.edu (G. M. Fernández-Nieto); vanessa.echeverria@monash.edu (V. Echeverria); roberto.martinezmaldonado@monash.edu (R. Martinez-Maldonado); simon.buckinghamshum@uts.edu.au (S. Buckingham Shum) Orcid 0000-0002-8163-2303 (G. M. Fernández-Nieto); 0000-0002-2022-9588 (V. Echeverria); 0000-0002-8375-1816 (R. Martinez-Maldonado); 0000-0002-6334-7429 (S. Buckingham Shum) © 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings http://ceur-ws.org ISSN 1613-0073 CEUR Workshop Proceedings (CEUR-WS.org) CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings supporting the understanding of processes such as teamwork [6–8] and communication [9–12], as well as the impact of emotions on learning [13, 14]. They also assist in analysing teacher- student interactions [15–18]. Multimodal data can be difficult to interpret if the intention is to open these data up to educational users, ultimately closing the feedback loop [19]. Thus, researchers have started to use InfoVis and visual design principles to unpack and communicate insights coming from multimodal data to non-expert users, such as teachers and students. Previous research has explored the use of data storytelling (DS) in learning analytics dash- boards (LAD) for conveying insights to educational users [20–22]. These studies have shown promising outcomes, demonstrating that DS elements effectively aid in interpreting complex data. However, in these studies, the integration of pedagogical intentions with DS elements has been manually conducted. Researchers typically engage in an inquiry process with edu- cational stakeholders to identify these pedagogical intentions, which are then mapped to DS elements in the LADs. While advancements have been notable, the field still lacks integrated, automated solutions that are tailored to both students and teachers, incorporating educators’ instructional strategies and offering custom data interfaces suited to their specific teaching skills and requirements [20, 23–31]. To overcome these challenges, we introduce YarnSense, a system architecture that employs data storytelling, an approach combining data, visuals, and narrative [32, 33], to simplify and communicate insights from multimodal behaviour data in dynamic and collocated settings. YarnSense includes a context modeller for educators to guide analysis, an automated sensor data capture, a multimodal modeller to translate sensor data into meaningful constructs, and a data storytelling generator for learner-facing interfaces. We demonstrate its application through a reference implementation in a clinical nursing healthcare setting with 254 students and six teachers, showing how YarnSense helps define and interpret the pedagogical intentions. 2. Background and Related Work 2.1. Automated multimodal sensor-data visual interfaces to support learning The integration of digital technologies into learning spaces has led to the use of various sensing devices such as infrared sensors and physiological wearables to capture multimodal behaviour data in educational settings [5, 34–36]. These data help to study key learning processes such as effective teamwork [6–8] and communication [9–12]. However, creating effective user interfaces for non-data experts remains a challenge. Current implementations, such as EduSense [16] and Sensai [37], have been used to provide feedback in educational contexts, but often lack the ability to simplify complex multimodal data for end users, such as students and teachers. Recent efforts have aimed to address these challenges by developing tools that can elucidate complex team dynamics using collecting audio and user interactions in an online setting (BLINC [38]) and narrative visualisations for MOOCs [39]. Despite these advances, the need to automatically generate user-friendly interfaces that can translate sensor data into meaningful insights for educational purposes remains unmet. 2.2. Data storytelling foundations and approaches in education Data Storytelling (DS) has emerged as an effective technique for communicating complex data insights through a combination of data, visuals, and narrative [32, 33, 40–42]. DS transforms data into intuitive visualisations and narratives, making it easier for non-experts to grasp complex information. They identified key data storytelling principles of effective data storytelling: (1) focus on purposeful communication, (2) drive audience attention through meaningful visual elements, (3) select appropriate visuals for different purposes, (4) adhere to the basic principles of information visualisation design, such as removing unnecessary elements and using captions, space, shape, and colour wisely, and (5) incorporate narrative structures, as in narrative visualisation or visual narratives [20, 43]. In education, the application of the principles of DS has shown promise in enhancing multimodal data visualisation. For instance, Martinez-Maldonado et al. [21] used a layered storytelling approach to categorise multimodal data into meaningful information structures. However, most existing DS applications in education, including those by Martinez-Maldonado et al. [21], Echeverria et al. [44], Fernández-Nieto et al. [45], are not fully automated and have been tested primarily in high-fidelity prototypes or controlled settings. Although previous research has investigated how to automatically generate visual outcomes using multimodal data, there remains a gap in the generation of meaningful automated interfaces guided by teacher’s pedagogical intentions and data storytelling principles to facilitate the communication of complex multimodal data. This paper presents an architecture and its implementation to automatically generate multimodal data storytelling interfaces to support students’ reflection in a nursing simulation setting. 2.3. Architectures for Multimodal Learning Analytics -MMLA Recent efforts in automating Multimodal Learning Analytics (MMLA) interfaces have been reviewed by Shankar et al. [46], focusing on nine different architectures through the Data Value Chain (DVC) framework. This framework includes data discovery, integration, and exploitation. In data discovery, all architectures leveraged multiple data sources, such as physiological data and posture data. Most included data preparation steps such as pretransformation and organising data relevant to the learning context. For data integration, over half of the architectures incorporated mechanisms to merge data from specific modalities, with databases being a popular choice. The literature review highlights that, in terms of data exploitation, almost all architectures carried out analysis activities, including statistical analysis or machine learning, and most produced visualisations like dashboards. The review also noted that three architectures provided decision-making support, specifically targeting teachers and students. However, two main challenges were identified: the lack of learning alignment or connections to the learning context in MMLA architectures, and the complexity of MMLA data and its visualisation posing challenges for stakeholders’ data literacy. A more recent architecture by Noël et al. [47] focuses on audio and video data, using hardware such as Raspberry Pi 4 for data collection and a server for storage and visualisation. Despite offering five visualisations for educators to assess collaborative activities, further improvements in design and evaluation are needed for effective use by stakeholders. This highlights the ongoing need for MMLA architectures that provide contextualised, meaningful interfaces to support teachers and students, underlining the importance of developing accessible and explanatory data stories within these systems. 3. YarnSense: Automated Educational Data Storytelling Architecture YarnSense is a multi-tiered architecture, that automatically distills insights from multimodal behavioural data, gathered via sensors worn by students and human observations, translating these into data stories that reflect teachers’ educational goals. This system architecture helps students reflect on their learning activities. It comprises four main tiers, as shown in Figure 1: Figure 1: YarnSense: Multi-tier architecture for automated Data Storytelling 3.1. User Interfaces: The Context Modeller This tier provides user interfaces for experts (e.g., teachers or researchers) to input what is known of the learning activity and the teacher’s pedagogical intentions. Knowledge of the learning activity corresponds to the specifics of a learning activity (e.g., nursing simulations) based on the learning design. Key features to be identified from the learning design and captured in these interfaces include: i) Actions of interest expected during the activity, such as critical moments or milestones (e.g., patient adverse reaction). ii) Information on physical resources in the learning space, including the positions of manikins, trolleys, or sensors (e.g., number of beds in a simulation ward). iii) Meta-information, such as the role of team members and devices to be worn during the activity (e.g., an auxiliary nurse wearing a microphone). In addition, this tear allows teachers to input the pedagogical intentions of the learning activity into the system. It defines the teacher assessment criteria into rules that will be used for the system to interrogate multimodal data and create stories. Considerations for implementation. Web-based platforms are a suitable choice for implementing this tier. Technologies such as HTML, CSS, and JavaScript, in conjunction with frameworks like React or Angular, can be used to develop intuitive and responsive user interfaces. 3.2. Multimodal Sensor Data Capture This component focusses on collecting data from both machine sensing (e.g., wearable sensors) and human sensing (e.g., observations). For wearable sensors, in particular, this tier considers multiple sensor data captured independently and in a loosely coupled manner. Key features for collecting data from sensors include: i) a recommended design pattern of ‘pipe and filters’ for independent data collection and processing. Pipe and filter patterns are commonly used in signal processing and remote sensing applications [48]. According to this pattern, filters are designed independently and are typically well defined as services or functions, and pipes are conduits of information. ii) According to the previous architectural feature, this tier captures sensor data in parallel with automated functions for start / stop processes, helping to synchronise and scalability of the data. That way, each data modality is cleaned and stored in its more convenient format (e.g., json, csv, mp4). in various formats per sensor, with flexibility for real-time processing or batch collection based on context needs. For human sensing data, this tear provides a user interface for users to log information into the system. Web and mobile applications are making it more accessible for users to capture additional observations during their learning activities. The data provided by the users are used to label actions that would otherwise be hard to detect using sensing technology. Considerations for implementation. For machine sensing, wearable sensor technologies are ideal for implementing this tier. Devices like smartwatches, fitness trackers, or custom wearables, equipped with sensors for physiological data, indoor positioning, and audio capture, can be used. A comprehensive list of sensors used in educational data capture is detailed in the literature review by Chango et al. [49]. To handle parallel data processing, multi threading or multiprocessing capabilities in programming languages such as Python, Java, or C++ can be employed. Additionally, parallel processing frameworks like Apache Kafka or RabbitMQ can be used for efficient data stream management. For human sensing, mobile applications developed for platforms like Android or iOS would enable users to conveniently log data during their learning activities. 3.3. Multimodal Modelling Considering the quantitative ethnography (QE) approach [50] and the multimodal matrix (MM) concept [51], this tier enhances low-level data with contextual insights from the context modeler. It transforms sensor data into meaningful constructs by coding multimodal observations into a data structure for analysis against the assessment criteria. These constructs are crucial for analysing multiple data modalities. For example, in physiological data, arousal peaks (indicative of changes in skin conductance levels) are interpreted as stress level indicators [45]. Additionally, for indoor positioning data, the theory of proxemics helps to identify interactional spaces and social formations during learning activities [52]. This theory is also applied to model the combination of modalities, such as positioning data and audio, to detect co-located speech events [53]. To do that, this tier implements custom software scripts to filter, combine, aggregate, or summarise the multimodal matrices according to the teacher’s pedagogical intentions previously defined. As a result of this analysis, a Learner Model is generated. The Learner Model is a structured representation of student performance, misconceptions, or difficulties. The Learner Model assesses if the team achieved the pedagogical intentions defined by the teachers. Considerations for implementation. To implement this tier, data analysis software like R or Python, equipped with libraries such as Pandas, NumPy, and SciPy, can be used for processing and analysing multimodal data based on specific constructs. Additionally, data visualisation libraries like Matplotlib, Seaborn, or D3.js are useful for visualising data during the analysis phase, which assists in refining the Learner Model. 3.4. Data Storytelling Generator The final tier uses the Leaner Model and the teacher’s pedagogical intentions and communicates insights through data, visualisations, and narratives. The key features of this tier include: i) enhancing the data visualisations with DS principles (Section 2.2), such as highlighting important elements, colour schemes, and removing unnecessary elements to focus on relevant aspects of the learning model. ii) Generating visual stories that provide individual or team outcomes in an easily interpretable format for students. The Narratives are capture from the teacher’s pedagogical intentions where they can incorporate textual feedback via the user interface. Data from the Lerner model are visualised and combined with narratives to convey a story for an individual student or a team. Considerations for implementation. This tier can be implemented by integrating data visualisation tools such as Tableau, Qlik, or D3.js, which allows visual enhancements through the DS principles. Alternatively, custom visualisation software can be developed using programming languages such as JavaScript (with libraries like Chart.js or Three.js) or Python (with libraries like Matplotlib or Seaborn), customised to meet the specific requirements of the teacher’s pedagogical intentions. Another option is the use of narrative generation tools, Natural Language Processing (NLP) libraries in Python, to partly automate the creation of narratives based on the data (e.g., Large Language Models -LLM). Each tier of YarnSense plays a crucial role in transforming complex multimodal data into insightful and accessible data stories, supporting reflective learning in educational settings. 4. Reference Implementation Having introduced our architecture in general terms above, we now turn to an illustrative example of how the whole architecture can be implemented in a specific learning activity. This architecture was implemented in an authentic clinical setting in nursing healthcare and was reported in Fernández-Nieto et al. [54]. Data stories in this clinical context were created using a completely automated process. 4.1. Learning context and data collection The clinical scenario provides an opportunity for students to practice teamwork, communication, and prioritisation skills in the setting of a deteriorating patient. The clinical scenario was run in 38 classes by different instructors. A total of 254 students in their third/fourth year volunteered to participate in the data collection. The goal of the clinical scenario was to provide care to four patients and prioritise the care of each bed as a team. According to the assessment criteria established by the subject coordinator, a highly effective team should have performed the following five actions in the main bed (useful information for the context modeller tier): i) administer oxygen after patient respiratory depression; ii) assess vital signs every 5 minutes; iii) cease PCA (patient-controlled analgesia) after patient altered conscious state; iv) activate MET (Medical Emergency Team) calls after patient deterioration; and v) administer Naloxone timely. Additionally, students are supposed to take care of the other 3 beds; they have to prioritise care. 4.2. YarnSense: Implementation of Nursing Simulations in the Wild YarnSense was implemented and deployed in the 38 simulation classes following the learning activity described above. Details of the implementation in the wild are presented in Table 1. In our reference implementation, we present two different types of automated data stories. The first type highlights errors made by students in simulations. From the positioning data and observations, using the teachers’ pedagogical intentions, we automatically identified three error categories, as described in Fernández-Nieto et al. [45] and Fernandez-Nieto et al. [52]: i) Sequence Errors: Occur when a team performs a critical action in the wrong sequence. ii) Timeliness Errors: Identified when students respond too slowly, executing actions later than recommended by healthcare guidelines. iii) Frequency Errors: Detected by calculating the time difference between two key logged actions that should be performed repeatedly. The second type of data stories, called positioning graphs, focuses on the physical interactions of nurses. These stories provide insights into how much time nurses spend on patients’ bedsides and in close proximity to other nurses during the simulation. These data help to understand spatial dynamics and collaboration patterns within the nursing team. Table 1 YarSense implementation in a in-the-wild nursing simulation Tier Implementation Details Technology Used Users Researchers, 6 Teachers, 254 Students NA Context Pedagogical intentions: Definition of five Web-based platform using Express Modeller rules according to the five actions expected Node.js framework and hosted on from students during the learning activity (Sec- an Amazon Elastic Compute Cloud tion 4.1). instance (Amazon EC2). Knowledge of the Learning Activity: roles: 2 Graduate Nurses and 2 Ward Graduate Nurses. Physical resources: 4 beds Multimodal Machine sensing: Indoor positioning data1 , Apache Kafka for parallel data col- sensor data physiological data (Empatica e4), audio, and lection and custom scripts for data capture video. processing. Human sensing: actions performed by stu- dents (action log) Multimodal QE modelling: The theory of Proxemics was Custom scripts in Python to create Modelling used to identify interpersonal spaces between the multimodal matrix. Matplotlib nurses. Only indoor positioning data and log to visualise bar graphs and Python- data were used for modelling. igraph to visualise graphs. Vis- timeline to visualise a timeline2 . Data Five data stories were fully automatically ren- Custom scripts in Python and Storytelling dered, combining data visualisations and the JavaScripts to visualise the data Generator feedback created by the teacher in the peda- stories. gogical intentions. Figure 2: Types of automated data stories: a. Nurses’ proximity to beds; b. Nursing team working in close proximity; c. Error committed by students during the simulation 5. Discussion, Future Work and Limitations 5.1. Assessing the Efficacy of Architectural Decisions Our architecture emphasises the involvement of educational users, including teachers, re- searchers, and students. Accordingly, our Context Modeller tier enables users to input rules and contextual information, guiding data analysis and the automated generation of data sto- ries. These features align with user experience (UX) best practices, which advocate for user input and the ability of users to exert control in human-computer interactions [55]. With this architecture, teachers can incorporate feedback and adapt the visual representations according to their learning designs, an approach that is an integral part of the current research agenda in Learning Analytics (LA) as discussed in Ez-Zaouia [56]. Additionally, our approach fosters opportunities for user-AI collaboration [57], allowing teachers to stay engaged by modifying rules for the Multimodal Modelling tier to analyse and generate outcomes tailored to teachers and students needs. This approach empowers teachers and students with agency. The architecture decision for the use of paralell data collection and processing provides flexibility for researchers to decide which technology adapts to certain learning contexts and needs. The integration of various data modalities presents unique challenges and opportunities for deeper insight into team and individual dynamics in physical training settings. Employing mature parallel processing frameworks, such as Apache Kafka and MapReduce, helps mitigate the complexity associated with handling multiple data sources [58]. Finally, incorporating Data Storytelling (DS) principles to aid user interpretation is consis- tent with the existing literature, highlighting the need for data representations that are both comprehensive and interpretable in educational contexts [59–61]. 5.2. Reflections on the Reference Implementation Process Our reference implementation highlights the importance and complexity of effectively visu- alising multimodal data to provide evidence for professional development and training. The integration of a tool to generate data stories as evidence of what students achieve in their learning activity plays a pivotal role in enhancing the learning experience and ensuring the effectiveness of training programs. In our approach, data storytelling transcends traditional data presentation methods by weaving complex data into coherent narratives that align with the teacher’s pedagogical intentions. Automated data stories not only aid in the comprehension of intricate concepts, but also foster an immersive and intuitive learning experience that sup- ports deeper reflections on the learning activity. Such integration is particularly invaluable in professional training, where the assimilation of practical and theoretical knowledge is critical. We observed that while real-time data processing is not always necessary in educational settings, near-real-time solutions can greatly benefit both teachers and students. For example, in nursing simulations, clinical debriefs typically follow team simulations, prompting students to reflect on their performance. These debrief sessions have been proven to be effective in helping students identify misconceptions and errors during simulations [62]. Therefore, our architecture aims to provide timely feedback through data stories, facilitating post-activity discussions and reflections, and thus enhancing the overall learning experience. 5.3. Learning from the Pilot: Insights and Future Enhancements The Multimodal Modelling tier necessitates an initial exploration of learning theories, such as the theory of proxemics, to understand how data modalities can be effectively used for learning. Our reference implementation drew upon prior research that explored quantitative ethnography (QE) modelling of positioning data and human observations [52, 63]. Consequently, future studies should aim to refine QE analysis and data visualisation for additional data modalities like audio and video, thereby optimising their usefulness and accessibility for educators and learners. Furthermore, more comprehensive evaluations are needed to identify the most effective visualisations that can cohesively represent diverse data sets, enabling a thorough understanding of learning activities, particularly in the context of data fusion [34]. While human-centred design can help address these challenges [64], there lies a promising opportunity in employing Large Language Models (LLM) to generate explanations for complex visualisations, thereby aiding user interpretation. The LLM image-to-text functionality, specifi- cally in its role for data storytelling in education, is an avenue worth exploring. Moreover, the potential of LLMs to assist in creating narratives that render visualisations more self-explanatory deserves thorough investigation. Future research should also further explore the role of user-AI collaboration. One of the main limitations of our approach is the automation of certain data modalities, particularly physiological data. There is a need for further development to fully automate this process, ensuring seamless integration and analysis of all data types. The architecture, originally used for Nursing Simulations, requires careful adaptation for different contexts. Successful integration into Learning Design calls for collaboration with educators to align it with assessment intentions and promote reflective thinking with added narratives. 6. Conclusion In conclusion, our work makes a significant contribution to the field of educational data analysis by detailing an architecture that automates the generation of data stories in real-world environ- ments. Our reference implementation, conducted in a large-scale, in-the-wild setting, not only demonstrates practical application but also acknowledges the challenges and limitations that future research must address to further refine architectures supporting physical activities and the provision of data storytelling. References [1] A. Petrosoniak, R. Almeida, L. D. Pozzobon, C. Hicks, M. Fan, K. White, M. McGowan, P. Trbovich, Tracking workflow during high-stakes resuscitation: the application of a novel clinician movement tracing tool during in situ trauma simulation, BMJ Simulation and Technology Enhanced Learning 5 (2019) 78–84. doi:10.1136/bmjstel- 2017- 000300 . [2] M. Viktorelius, C. Sellberg, The lived body and embodied instructional practices in maritime basic safety training, Vocations and Learning 15 (2022) 87–109. doi:10.1007/ s12186- 021- 09279- z . [3] A. B. Cooper, E. J. Tisdell, Embodied aspects of learning to be a surgeon, Medical teacher 42 (2020) 515–522. doi:10.1080/0142159X.2019.1708289 . [4] M. Kelly, R. Ellaway, A. Scherpbier, N. King, T. Dornan, Body pedagogics: embodied learning for the health professions, Medical education 53 (2019) 967–977. doi:10.1111/ medu.13916 . [5] L. Eyal, E. Gil, Hybrid Learning Spaces — A Three-Fold Evolving Perspective, Springer International Publishing, Cham, 2022, pp. 11–23. doi:10.1007/978- 3- 030- 88520- 5_2 . [6] M. A. Rosen, A. S. Dietz, T. Yang, C. E. Priebe, P. J. Pronovost, An integrative framework for sensor-based measurement of teamwork in healthcare, Journal of the American Medical Informatics Association 22 (2014) 11–18. doi:10.1136/amiajnl- 2013- 002606 . [7] G. Dafoulas, C. Cardoso Maia, A. Ali, J. Augusto, Collecting sensor-generated data for as- sessing teamwork and individual contributions in computing student teams, EDULEARN18 Proceedings 1 (2018) 11156–11162. doi:10.21125/EDULEARN.2018.2759 . [8] J. S. Cha, D. Athanasiadis, N. E. Anton, D. Stefanidis, D. Yu, Measurement of Nontech- nical Skills During Robotic-Assisted Surgery Using Sensor-Based Communication and Proximity Metrics, JAMA Network Open 4 (2021) e2132209–e2132209. doi:10.1001/ JAMANETWORKOPEN.2021.32209 . [9] D. O. Olguín, P. A. Gloor, A. Pentland, Wearable sensors for pervasive healthcare man- agement, in: 2009 3rd International Conference on Pervasive Computing Technolo- gies for Healthcare - Pervasive Health 2009, PCTHealth 2009, 2009. doi:10.4108/ICST. PERVASIVEHEALTH2009.6033 . [10] I. Winder, D. Delaporte, S. Wanaka, K. Hiekata, Sensing teamwork during multi-objective optimization, in: 2020 IEEE 6th World Forum on Internet of Things (WF-IoT), 2020, pp. 1–6. doi:10.1109/WF- IoT48130.2020.9221086 . [11] E. Watanabe, T. Ozeki, T. Kohama, Analysis of interactions between lecturers and students, in: Proceedings of the 8th International Conference on Learning Analytics and Knowledge, ACM, New York, NY, USA, 2018, pp. 370–374. doi:10.1145/3170358.3170360 . [12] E. Chng, M. R. Seyam, W. Yao, B. Schneider, Toward capturing divergent collaboration in makerspaces using motion sensors, Information and Learning Sciences (2022). doi:10. 1108/ILS- 08- 2020- 0182 . [13] S. K. D’Mello, N. Bosch, H. Chen, Multimodal-multisensor affect detection, in: The Handbook of Multimodal-Multisensor Interfaces: Signal Processing, Architectures, and Detection of Emotion and Cognition-Volume 2, 2018, pp. 167–202. doi:10.1145/3107990. 3107998 . [14] I. Villanueva, B. D. Campbell, A. C. Raikes, S. H. Jones, L. G. Putney, A multimodal explo- ration of engineering students’ emotions and electrodermal activity in design activities, Journal of Engineering Education 107 (2018) 414–441. doi:10.1002/jee.20225 . [15] M. Raca, L. Kidzinski, P. Dillenbourg, Translating Head Motion into Attention - Towards Processing of Student’s Body-Language, EDM (2015). URL: https://api.semanticscholar. org/CorpusID:15798760. [16] K. Ahuja, D. Kim, F. Xhakaj, V. Varga, A. Xie, S. Zhang, J. E. Townsend, C. Harrison, A. Ogan, Y. Agarwal, Edusense: Practical classroom sensing at scale, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 3 (2019). doi:10.1145/3351229 . [17] N. Saquib, A. Bose, D. George, S. Kamvar, Sensei: sensing educational interaction, Proceed- ings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1 (2018) 1–27. doi:10.1145/3161172 . [18] R. Martinez-Maldonado, V. Echeverria, K. Mangaroska, A. Shibani, G. Fernandez-Nieto, J. Schulte, S. Buckingham Shum, Moodoo the tracker: Spatial classroom analytics for char- acterising teachers’ pedagogical approaches, International Journal of Artificial Intelligence in Education (2021) 1–27. doi:10.1007/s40593- 021- 00276- w . [19] M. Worsley, R. Martinez-Maldonado, C. D’Angelo, A new era in multimodal learning analytics: Twelve core commitments to ground and grow mmla, Journal of Learning Analytics 8 (2021) 10–27. doi:10.18608/jla.2021.7361 . [20] V. Echeverria, R. Martinez-Maldonaldo, S. Buckingham Shum, K. Chiluiza, R. Granda, C. Conati, Exploratory versus Explanatory Visual Learning Analytics: Driving Teachers’ Attention through Educational Data Storytelling, Journal of Learning Analytics 5 (2018) 72–97. doi:https://doi.org/10.18608/jla.2018.53.6 . [21] R. Martinez-Maldonado, V. Echeverria, G. Fernandez-Nieto, S. B. Shum, From Data to Insights: A Layered Storytelling Approach for Multimodal Learning Analytics, in: CHI ’20, 2020, p. 15. doi:https://doi.org/10.1145/3313831.3376148 . [22] G. M. Fernandez-Nieto, V. Echeverria, S. Buckingham Shum, K. Mangaroska, K. Kitto, E. Palominos, C. Axisa, R. Martinez-Maldonado, Storytelling With Learner Data: Guiding Student Reflection on Multimodal Team Data, IEEE Transactions on Learning Technologies (2021) 1–14. doi:10.1109/TLT.2021.3131842 . [23] H. Khosravi, J. Kay, S. Sadiq, S. Buckingham Shum, R. Martinez-Maldonado, S. Knight, G. Chen, Y.-S. Tsai, C. Conati, D. Gasevic, Explainable artificial intelligence in education, Computers Education: Artificial Intelligence 3 (2022) 1–31. doi:10.1016/j.caeai.2022. 100074 . [24] A. V. Maltese, J. A. Harsh, D. Svetina, Data visualization literacy: Investigating data interpretation along the novice—expert continuum, Journal of College Science Teaching 45 (2015) 84–90. URL: https://www.jstor.org/stable/43631889. [25] J. E. Raffaghelli, B. Stewart, Centering complexity in ‘educators’ data literacy’ to support future practices in faculty development: a systematic review of the literature, Teaching in Higher Education 25 (2020) 435–455. doi:10.1080/13562517.2019.1696301 . [26] M. Worsley, Multimodal Learning Analytics ’ Past , Present , and , Potential Futures (2018) 1–16. URL: http://ceur-ws.org/Vol-2163/paper5.pdf. [27] S. K. Milligan, P. Griffin, Understanding learning and learning design in moocs: A measurement-based interpretation, Journal of Learning Analytics 3 (2016) 88–115. doi:10.18608/jla.2016.32.5 . [28] V. Shute, M. Ventura, Stealth assessment: Measuring and supporting learning in video games, MIT press, 2013. doi:10.7551/mitpress/9589.001.0001 . [29] S. S. Alhadad, Visualizing data to support judgement, inference, and decision making in learning analytics: Insights from cognitive psychology and visualization science, Journal of Learning Analytics 5 (2018) 60–85. doi:10.18608/jla.2018.52.5 . [30] S. Knight, A. Gibson, A. Shibani, Implementing learning analytics for learning impact: Taking tools to task, The Internet and Higher Education 45 (2020) 100729. doi:10.1016/j. iheduc.2020.100729 . [31] V. Echeverria, R. Martinez-Maldonado, S. Buckingham Shum, Towards Collaboration Translucence: Giving Meaning to Multimodal Group Data, Proceedings of the CHI (2019) 39:1–39:16. doi:10.1145/3290605.3300269 . [32] B. Dykes, Data storytelling: What it is and how it can be used to effectively communicate analysis results, Applied Marketing Analytics (2015). URL: https://hstalks.com/article/619/ data-storytelling-what-it-is-and-how-it-can-be-use/. [33] L. Ryan, The visual imperative: creating a visual culture of data discovery, Kaufmann, Morgan, Massachusetts, 2016. [34] L. Yan, L. Zhao, D. Gasevic, R. Martinez-Maldonado, Scalability, sustainability, and ethicality of multimodal learning analytics, in: LAK22: 12th International Learning Analytics and Knowledge Conference, 2022, pp. 13–23. doi:10.1145/3506860.3506862 . [35] K. Sharma, M. Giannakos, Multimodal data capabilities for learning: What can multimodal data tell us about learning?, British Journal of Educational Technology 51 (2020) 1450–1484. doi:10.1111/bjet.12993 . [36] B. Schneider, G. Sung, E. Chng, S. Yang, How can high-frequency sensors capture collab- oration? a review of the empirical links between multimodal metrics and collaborative constructs, Sensors 21 (2021) 8185. doi:10.3390/s21248185 . [37] K. Kurihara, M. Goto, J. Ogata, Y. Matsusaka, T. Igarashi, Presentation sensei: A presen- tation training system using speech and image processing, in: Proceedings of the 9th International Conference on Multimodal Interfaces, ICMI ’07, Association for Computing Machinery, New York, NY, USA, 2007, p. 358–365. doi:10.1145/1322192.1322256 . [38] M. Worsley, K. Anderson, N. Melo, J. Jang, Designing Analytics for Collaboration Literacy and Student Empowerment., Journal of Learning Analytics 8 (2021) 30–48. doi:10.18608/ jla.2021.7242 . [39] Q. Chen, Z. Li, T.-C. Pong, H. Qu, Designing narrative slideshows for learning analytics, in: 2019 IEEE Pacific Visualization Symposium (PacificVis), 2019, pp. 237–246. doi:10.1109/ PacificVis.2019.00036 . [40] C. N. Knaflic, Storytelling with data: A data visualization guide for business professionals, 12 ed., John Wiley Sons, New Jersey, 2017. [41] N. Gershon, W. Page, What storytelling can do for information visualization, Commun. ACM 44 (2001) 31–37. doi:10.1145/381641.381653 . [42] W. Wojtkowski, W. Wojtkowski, W. G. Wojtkowski, Storytelling: its role in information visualization, Proceedings of European Systems Science Congress (2002) 1–5. doi:10.1.1. 99.4771 . [43] E. Segel, J. Heer, Narrative visualization: Telling stories with data, IEEE Transactions on Visualization and Computer Graphics 16 (2010) 1139–1148. doi:10.1109/TVCG.2010.179 . [44] V. Echeverria, R. Martinez-Maldonado, R. Granda, K. Chiluiza, C. Conati, S. Buckingham Shum, Driving data storytelling from learning design, in: ACM, Association for Computing Machinery, 2018, pp. 131–140. doi:https://doi.org/10.1145/3170358.3170380 . [45] G. M. Fernández-Nieto, S. Buckingham Shum, K. Kitto, R. Martínez-Maldonado, Beyond the Learning Analytics Dashboard: Alternative Ways to Communicate Student Data Insights Combining Visualisation, Narrative and Storytelling, LAK 2022 5 (2022) 1–16. doi:10.1145/3506860.3506895 . [46] S. K. Shankar, L. P. Prieto, M. J. Rodríguez-Triana, A. Ruiz-Calleja, A review of multimodal learning analytics architectures, in: 2018 IEEE 18th International Conference on Advanced Learning Technologies (ICALT), 2018, pp. 212–214. doi:10.1109/ICALT.2018.00057 . [47] R. Noël, D. Miranda, C. Cechinel, F. Riquelme, T. T. Primo, R. Munoz, Visualizing collabora- tion in teamwork: A multimodal learning analytics platform for non-verbal communication, Applied Sciences 12 (2022). doi:10.3390/app12157499 . [48] A. B. Pillai, Software architecture with Python : design and architect highly scalable, robust, clean, and high performance applications in Python, Packt Publishing, 2017. [49] W. Chango, J. A. Lara, R. Cerezo, C. Romero, A review on data fusion in multimodal learning analytics and educational data mining, Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 12 (2022) e1458. doi:10.1002/widm.1458 . [50] D. W. Shaffer, Quantitative ethnography, Cathcart Press, 2017. [51] S. Buckingham Shum, V. Echeverria, R. Martinez-Maldonado, The multimodal matrix as a quantitative ethnography methodology, in: B. Eagan, M. Misfeldt, A. Siebert-Evenstone (Eds.), Advances in Quantitative Ethnography, Springer International Publishing, Cham, 2019, pp. 26–40. Doi: https://doi.org/10.1007/978-3-030-33232-7_3. [52] G. Fernandez-Nieto, R. Martinez-Maldonado, V. Echeverria, K. Kitto, P. An, S. Buckingham Shum, What Can Analytics for Teamwork Proxemics Reveal About Positioning Dynamics In Clinical Simulations?, Proceedings of the ACM on Human-Computer Interaction 5 (2021) 1–24. doi:10.1145/3449284 . [53] L. Zhao, L. Yan, D. Gasevic, S. Dix, H. Jaggard, R. Wotherspoon, R. Alfredo, X. Li, R. Martinez-Maldonado, Modelling co-located team communication from voice detection and positioning data in healthcare simulation, in: LAK22: 12th International Learn- ing Analytics and Knowledge Conference, LAK22, Association for Computing Machin- ery, New York, NY, USA, 2022, p. 370–380. URL: https://doi.org/10.1145/3506860.3506935. doi:10.1145/3506860.3506935 . [54] G. M. Fernández-Nieto, R. Martinez-Maldonado, V. Echeverria, D. G. Kirsty Kitto, S. B. Shum, Data storytelling editor: A teacher-centred tool for customising learning analytics dashboard narratives, in: 14th Learning Analytics and Knowledge Conference (LAK ’24). ACM, 2024. doi:10.1145/3636555.3636930 . [55] J. Bergström, J. Knibbe, H. Pohl, K. Hornbæk, Sense of agency and user experience: Is there a link?, ACM Trans. Comput.-Hum. Interact. 29 (2022). URL: https://doi.org/10.1145/3490493. doi:10.1145/3490493 . [56] M. Ez-Zaouia, Teacher-Centered Dashboards Design Process, in: LAK20, Frankfurt, Germany, 2020. URL: https://hal.science/hal-02516815. [57] A. F. Wise, S. Knight, S. B. Shum, Collaborative Learning Analytics, in: U. Cress, C. Rosé, A. F. Wise, J. Oshima (Eds.), International Handbook of Computer-Supported Collaborative Learning, Springer International Publishing, Cham, 2021, pp. 425–443. doi:10.1007/978- 3- 030- 65291- 3_23 . [58] L. Yao, Z. Ge, Distributed parallel deep learning of hierarchical extreme learning machine for multimode quality prediction with big process data, Engineering Applications of Artificial Intelligence 81 (2019) 450–465. doi:10.1016/j.engappai.2019.03.011 . [59] S. Pozdniakov, R. Martinez-Maldonado, Y.-S. Tsai, V. Echeverria, N. Srivastava, D. Gasevic, How do teachers use dashboards enhanced with data storytelling elements according to their data visualisation literacy skills?, in: LAK23: 13th International Learning Analytics and Knowledge Conference, LAK2023, Association for Computing Machinery, New York, NY, USA, 2023, p. 89–99. doi:10.1145/3576050.3576063 . [60] D. Hooshyar, K. Tammets, T. Ley, K. Aus, K. Kollom, Learning analytics in supporting student agency: A systematic review, Sustainability 15 (2023). doi:10.3390/su151813662 . [61] E. Villalobos, I. Hilliger, M. Pérez-Sanagustín, C. González, S. Celis, J. Broisin, Analyzing learners’ perception of indicators in student-facing analytics: A card sorting approach, in: O. Viberg, I. Jivet, P. J. Muñoz-Merino, M. Perifanou, T. Papathoma (Eds.), Responsive and Sustainable Educational Futures, Springer Nature Switzerland, Cham, 2023, pp. 430–445. doi:10.1007/978- 3- 031- 42682- 7_29 . [62] E. Palominos, T. Levett-Jones, T. Power, R. Martinez-Maldonado, ‘We learn from our mistakes’: Nursing students’ perceptions of a productive failure simulation, Collegian 29 (2022) 708–712. doi:10.1016/j.colegn.2022.02.006 . [63] V. Echeverria, R. Martinez-Maldonado, L. Yan, L. Zhao, G. Fernandez-Nieto, D. Gašević, S. B. Shum, Huceta: A framework for human-centered embodied teamwork analytics, IEEE Pervasive Computing 22 (2023) 39–49. doi:10.1109/MPRV.2022.3217454 . [64] R. Alfredo, V. Echeverria, Y. Jin, Z. Swiecki, D. Gašević, , R. Martinez-Maldonado, Slade: A method for designing human-centred learning analytics systems, in: LAK24: 14th International Learning Analytics and Knowledge Conference, 2023, p. 16. doi:10.1145/ 3636555.3636847 .