Learning by doing is essential in sectors like emergency response [1], safety training [2], and healthcare [3], where professionals gain knowledge through practical experiences, including bodily interactions and emotional responses [4]. However, capturing critical events or errors during fast-paced training scenarios is challenging.
Integrating digital technologies and sensing devices in physical learning spaces offers a solution to improve teaching and learning [5]. These technologies, including infrared sensors, video and audio recorders, and wearables, capture multimodal behaviour data in real time , supporting the understanding of processes such as teamwork [6][7][8] and communication [9][10][11][12], as well as the impact of emotions on learning [13,14]. They also assist in analysing teacherstudent interactions [15][16][17][18]. Multimodal data can be difficult to interpret if the intention is to open these data up to educational users, ultimately closing the feedback loop [19]. Thus, researchers have started to use InfoVis and visual design principles to unpack and communicate insights coming from multimodal data to non-expert users, such as teachers and students.
Previous research has explored the use of data storytelling (DS) in learning analytics dashboards (LAD) for conveying insights to educational users [20][21][22]. These studies have shown promising outcomes, demonstrating that DS elements effectively aid in interpreting complex data. However, in these studies, the integration of pedagogical intentions with DS elements has been manually conducted. Researchers typically engage in an inquiry process with educational stakeholders to identify these pedagogical intentions, which are then mapped to DS elements in the LADs. While advancements have been notable, the field still lacks integrated, automated solutions that are tailored to both students and teachers, incorporating educators' instructional strategies and offering custom data interfaces suited to their specific teaching skills and requirements [20,[23][24][25][26][27][28][29][30][31].
To overcome these challenges, we introduce YarnSense, a system architecture that employs data storytelling, an approach combining data, visuals, and narrative [32,33], to simplify and communicate insights from multimodal behaviour data in dynamic and collocated settings. YarnSense includes a context modeller for educators to guide analysis, an automated sensor data capture, a multimodal modeller to translate sensor data into meaningful constructs, and a data storytelling generator for learner-facing interfaces. We demonstrate its application through a reference implementation in a clinical nursing healthcare setting with 254 students and six teachers, showing how YarnSense helps define and interpret the pedagogical intentions.
The integration of digital technologies into learning spaces has led to the use of various sensing devices such as infrared sensors and physiological wearables to capture multimodal behaviour data in educational settings [5,[34][35][36]. These data help to study key learning processes such as effective teamwork [6][7][8] and communication [9][10][11][12]. However, creating effective user interfaces for non-data experts remains a challenge. Current implementations, such as EduSense [16] and Sensai [37], have been used to provide feedback in educational contexts, but often lack the ability to simplify complex multimodal data for end users, such as students and teachers. Recent efforts have aimed to address these challenges by developing tools that can elucidate complex team dynamics using collecting audio and user interactions in an online setting (BLINC [38]) and narrative visualisations for MOOCs [39]. Despite these advances, the need to automatically generate user-friendly interfaces that can translate sensor data into meaningful insights for educational purposes remains unmet.
Data Storytelling (DS) has emerged as an effective technique for communicating complex data insights through a combination of data, visuals, and narrative [32,33,[40][41][42]. DS transforms data into intuitive visualisations and narratives, making it easier for non-experts to grasp complex information. They identified key data storytelling principles of effective data storytelling: (1) focus on purposeful communication, (2) drive audience attention through meaningful visual elements, (3) select appropriate visuals for different purposes, (4) adhere to the basic principles of information visualisation design, such as removing unnecessary elements and using captions, space, shape, and colour wisely, and (5) incorporate narrative structures, as in narrative visualisation or visual narratives [20,43]. In education, the application of the principles of DS has shown promise in enhancing multimodal data visualisation. For instance, Martinez-Maldonado et al. [21] used a layered storytelling approach to categorise multimodal data into meaningful information structures.
However, most existing DS applications in education, including those by Martinez-Maldonado et al. [21], Echeverria et al. [44], Fernández-Nieto et al. [45], are not fully automated and have been tested primarily in high-fidelity prototypes or controlled settings. Although previous research has investigated how to automatically generate visual outcomes using multimodal data, there remains a gap in the generation of meaningful automated interfaces guided by teacher's pedagogical intentions and data storytelling principles to facilitate the communication of complex multimodal data. This paper presents an architecture and its implementation to automatically generate multimodal data storytelling interfaces to support students' reflection in a nursing simulation setting.
Recent efforts in automating Multimodal Learning Analytics (MMLA) interfaces have been reviewed by Shankar et al. [46], focusing on nine different architectures through the Data Value Chain (DVC) framework. This framework includes data discovery, integration, and exploitation. In data discovery, all architectures leveraged multiple data sources, such as physiological data and posture data. Most included data preparation steps such as pretransformation and organising data relevant to the learning context. For data integration, over half of the architectures incorporated mechanisms to merge data from specific modalities, with databases being a popular choice. The literature review highlights that, in terms of data exploitation, almost all architectures carried out analysis activities, including statistical analysis or machine learning, and most produced visualisations like dashboards. The review also noted that three architectures provided decision-making support, specifically targeting teachers and students. However, two main challenges were identified: the lack of learning alignment or connections to the learning context in MMLA architectures, and the complexity of MMLA data and its visualisation posing challenges for stakeholders' data literacy.
A more recent architecture by Noël et al. [47] focuses on audio and video data, using hardware such as Raspberry Pi 4 for data collection and a server for storage and visualisation. Despite offering five visualisations for educators to assess collaborative activities, further improvements in design and evaluation are needed for effective use by stakeholders. This highlights the ongoing need for MMLA architectures that provide contextualised, meaningful interfaces to support teachers and students, underlining the importance of developing accessible and explanatory data stories within these systems.
YarnSense is a multi-tiered architecture, that automatically distills insights from multimodal behavioural data, gathered via sensors worn by students and human observations, translating these into data stories that reflect teachers' educational goals. This system architecture helps students reflect on their learning activities. It comprises four main tiers, as shown in Figure 1:
This tier provides user interfaces for experts (e.g., teachers or researchers) to input what is known of the learning activity and the teacher's pedagogical intentions. Knowledge of the learning activity corresponds to the specifics of a learning activity (e.g., nursing simulations) based on the learning design. Key features to be identified from the learning design and captured in these interfaces include: i) Actions of interest expected during the activity, such as critical moments or milestones (e.g., patient adverse reaction). ii) Information on physical resources in the learning space, including the positions of manikins, trolleys, or sensors (e.g., number of beds in a simulation ward). iii) Meta-information, such as the role of team members and devices to be worn during the activity (e.g., an auxiliary nurse wearing a microphone). In addition, this tear allows teachers to input the pedagogical intentions of the learning activity into the system. It defines the teacher assessment criteria into rules that will be used for the system to interrogate multimodal data and create stories. Considerations for implementation. Web-based platforms are a suitable choice for implementing this tier. Technologies such as HTML, CSS, and JavaScript, in conjunction with frameworks like React or Angular, can be used to develop intuitive and responsive user interfaces.
This component focusses on collecting data from both machine sensing (e.g., wearable sensors) and human sensing (e.g., observations). For wearable sensors, in particular, this tier considers multiple sensor data captured independently and in a loosely coupled manner. Key features for collecting data from sensors include: i) a recommended design pattern of 'pipe and filters' for independent data collection and processing. Pipe and filter patterns are commonly used in signal processing and remote sensing applications [48]. According to this pattern, filters are designed independently and are typically well defined as services or functions, and pipes are conduits of information. ii) According to the previous architectural feature, this tier captures sensor data in parallel with automated functions for start / stop processes, helping to synchronise and scalability of the data. That way, each data modality is cleaned and stored in its more convenient format (e.g., json, csv, mp4). in various formats per sensor, with flexibility for real-time processing or batch collection based on context needs.
For human sensing data, this tear provides a user interface for users to log information into the system. Web and mobile applications are making it more accessible for users to capture additional observations during their learning activities. The data provided by the users are used to label actions that would otherwise be hard to detect using sensing technology.
Considerations for implementation. For machine sensing, wearable sensor technologies are ideal for implementing this tier. Devices like smartwatches, fitness trackers, or custom wearables, equipped with sensors for physiological data, indoor positioning, and audio capture, can be used. A comprehensive list of sensors used in educational data capture is detailed in the literature review by Chango et al. [49]. To handle parallel data processing, multi threading or multiprocessing capabilities in programming languages such as Python, Java, or C++ can be employed. Additionally, parallel processing frameworks like Apache Kafka or RabbitMQ can be used for efficient data stream management.
For human sensing, mobile applications developed for platforms like Android or iOS would enable users to conveniently log data during their learning activities.
Considering the quantitative ethnography (QE) approach [50] and the multimodal matrix (MM) concept [51], this tier enhances low-level data with contextual insights from the context modeler. It transforms sensor data into meaningful constructs by coding multimodal observations into a data structure for analysis against the assessment criteria. These constructs are crucial for analysing multiple data modalities. For example, in physiological data, arousal peaks (indicative of changes in skin conductance levels) are interpreted as stress level indicators [45]. Additionally, for indoor positioning data, the theory of proxemics helps to identify interactional spaces and social formations during learning activities [52]. This theory is also applied to model the combination of modalities, such as positioning data and audio, to detect co-located speech events [53].
To do that, this tier implements custom software scripts to filter, combine, aggregate, or summarise the multimodal matrices according to the teacher's pedagogical intentions previously defined. As a result of this analysis, a Learner Model is generated. The Learner Model is a structured representation of student performance, misconceptions, or difficulties. The Learner Model assesses if the team achieved the pedagogical intentions defined by the teachers.
Considerations for implementation. To implement this tier, data analysis software like R or Python, equipped with libraries such as Pandas, NumPy, and SciPy, can be used for processing and analysing multimodal data based on specific constructs. Additionally, data visualisation libraries like Matplotlib, Seaborn, or D3.js are useful for visualising data during the analysis phase, which assists in refining the Learner Model.
The final tier uses the Leaner Model and the teacher's pedagogical intentions and communicates insights through data, visualisations, and narratives. The key features of this tier include: i) enhancing the data visualisations with DS principles (Section 2.2), such as highlighting important elements, colour schemes, and removing unnecessary elements to focus on relevant aspects of the learning model. ii) Generating visual stories that provide individual or team outcomes in an easily interpretable format for students. The Narratives are capture from the teacher's pedagogical intentions where they can incorporate textual feedback via the user interface. Data from the Lerner model are visualised and combined with narratives to convey a story for an individual student or a team.
Considerations for implementation. This tier can be implemented by integrating data visualisation tools such as Tableau, Qlik, or D3.js, which allows visual enhancements through the DS principles. Alternatively, custom visualisation software can be developed using programming languages such as JavaScript (with libraries like Chart.js or Three.js) or Python (with libraries like Matplotlib or Seaborn), customised to meet the specific requirements of the teacher's pedagogical intentions. Another option is the use of narrative generation tools, Natural Language Processing (NLP) libraries in Python, to partly automate the creation of narratives based on the data (e.g., Large Language Models -LLM).
Each tier of YarnSense plays a crucial role in transforming complex multimodal data into insightful and accessible data stories, supporting reflective learning in educational settings.
Having introduced our architecture in general terms above, we now turn to an illustrative example of how the whole architecture can be implemented in a specific learning activity. This architecture was implemented in an authentic clinical setting in nursing healthcare and was reported in Fernández-Nieto et al. [54]. Data stories in this clinical context were created using a completely automated process.
The clinical scenario provides an opportunity for students to practice teamwork, communication, and prioritisation skills in the setting of a deteriorating patient. The clinical scenario was run in 38 classes by different instructors. A total of 254 students in their third/fourth year volunteered to participate in the data collection. The goal of the clinical scenario was to provide care to four patients and prioritise the care of each bed as a team. According to the assessment criteria established by the subject coordinator, a highly effective team should have performed the following five actions in the main bed (useful information for the context modeller tier): i) administer oxygen after patient respiratory depression; ii) assess vital signs every 5 minutes; iii) cease PCA (patient-controlled analgesia) after patient altered conscious state; iv) activate MET (Medical Emergency Team) calls after patient deterioration; and v) administer Naloxone timely. Additionally, students are supposed to take care of the other 3 beds; they have to prioritise care.
YarnSense was implemented and deployed in the 38 simulation classes following the learning activity described above. Details of the implementation in the wild are presented in Table 1.
In our reference implementation, we present two different types of automated data stories. The first type highlights errors made by students in simulations. From the positioning data and observations, using the teachers' pedagogical intentions, we automatically identified three error categories, as described in Fernández-Nieto et al. [45] and Fernandez-Nieto et al. [52]: i) Sequence Errors: Occur when a team performs a critical action in the wrong sequence. ii) Timeliness Errors: Identified when students respond too slowly, executing actions later than recommended by healthcare guidelines. iii) Frequency Errors: Detected by calculating the time difference between two key logged actions that should be performed repeatedly.
The second type of data stories, called positioning graphs, focuses on the physical interactions of nurses. These stories provide insights into how much time nurses spend on patients' bedsides and in close proximity to other nurses during the simulation. These data help to understand spatial dynamics and collaboration patterns within the nursing team.
Machine sensing: Indoor positioning data 1 , physiological data (Empatica e4), audio, and video.
Human sensing: actions performed by students (action log)
Apache Kafka for parallel data collection and custom scripts for data processing.
Multimodal Modelling
The theory of Proxemics was used to identify interpersonal spaces between nurses. Only indoor positioning data and log data were used for modelling.
Custom scripts in Python to create the multimodal matrix. Matplotlib to visualise bar graphs and Pythonigraph to visualise graphs. Vistimeline to visualise a timeline 2 .
Five data stories were fully automatically rendered, combining data visualisations and the feedback created by the teacher in the pedagogical intentions.
Custom scripts in Python and JavaScripts to visualise the data stories.
Our architecture emphasises the involvement of educational users, including teachers, researchers, and students. Accordingly, our Context Modeller tier enables users to input rules and contextual information, guiding data analysis and the automated generation of data sto-ries. These features align with user experience (UX) best practices, which advocate for user input and the ability of users to exert control in human-computer interactions [55]. With this architecture, teachers can incorporate feedback and adapt the visual representations according to their learning designs, an approach that is an integral part of the current research agenda in Learning Analytics (LA) as discussed in Ez-Zaouia [56]. Additionally, our approach fosters opportunities for user-AI collaboration [57], allowing teachers to stay engaged by modifying rules for the Multimodal Modelling tier to analyse and generate outcomes tailored to teachers and students needs. This approach empowers teachers and students with agency.
The architecture decision for the use of paralell data collection and processing provides flexibility for researchers to decide which technology adapts to certain learning contexts and needs. The integration of various data modalities presents unique challenges and opportunities for deeper insight into team and individual dynamics in physical training settings. Employing mature parallel processing frameworks, such as Apache Kafka and MapReduce, helps mitigate the complexity associated with handling multiple data sources [58].
Finally, incorporating Data Storytelling (DS) principles to aid user interpretation is consistent with the existing literature, highlighting the need for data representations that are both comprehensive and interpretable in educational contexts [59][60][61].
Our reference implementation highlights the importance and complexity of effectively visualising multimodal data to provide evidence for professional development and training. The integration of a tool to generate data stories as evidence of what students achieve in their learning activity plays a pivotal role in enhancing the learning experience and ensuring the effectiveness of training programs. In our approach, data storytelling transcends traditional data presentation methods by weaving complex data into coherent narratives that align with the teacher's pedagogical intentions. Automated data stories not only aid in the comprehension of intricate concepts, but also foster an immersive and intuitive learning experience that supports deeper reflections on the learning activity. Such integration is particularly invaluable in professional training, where the assimilation of practical and theoretical knowledge is critical.
We observed that while real-time data processing is not always necessary in educational settings, near-real-time solutions can greatly benefit both teachers and students. For example, in nursing simulations, clinical debriefs typically follow team simulations, prompting students to reflect on their performance. These debrief sessions have been proven to be effective in helping students identify misconceptions and errors during simulations [62]. Therefore, our architecture aims to provide timely feedback through data stories, facilitating post-activity discussions and reflections, and thus enhancing the overall learning experience.
The Multimodal Modelling tier necessitates an initial exploration of learning theories, such as the theory of proxemics, to understand how data modalities can be effectively used for learning. Our reference implementation drew upon prior research that explored quantitative ethnography (QE) modelling of positioning data and human observations [52,63]. Consequently, future studies should aim to refine QE analysis and data visualisation for additional data modalities like audio and video, thereby optimising their usefulness and accessibility for educators and learners. Furthermore, more comprehensive evaluations are needed to identify the most effective visualisations that can cohesively represent diverse data sets, enabling a thorough understanding of learning activities, particularly in the context of data fusion [34].
While human-centred design can help address these challenges [64], there lies a promising opportunity in employing Large Language Models (LLM) to generate explanations for complex visualisations, thereby aiding user interpretation. The LLM image-to-text functionality, specifically in its role for data storytelling in education, is an avenue worth exploring. Moreover, the potential of LLMs to assist in creating narratives that render visualisations more self-explanatory deserves thorough investigation. Future research should also further explore the role of user-AI collaboration.
One of the main limitations of our approach is the automation of certain data modalities, particularly physiological data. There is a need for further development to fully automate this process, ensuring seamless integration and analysis of all data types. The architecture, originally used for Nursing Simulations, requires careful adaptation for different contexts. Successful integration into Learning Design calls for collaboration with educators to align it with assessment intentions and promote reflective thinking with added narratives.
In conclusion, our work makes a significant contribution to the field of educational data analysis by detailing an architecture that automates the generation of data stories in real-world environments. Our reference implementation, conducted in a large-scale, in-the-wild setting, not only demonstrates practical application but also acknowledges the challenges and limitations that future research must address to further refine architectures supporting physical activities and the provision of data storytelling.