Multimodal Temporal Network Analysis to Improve Learner Support and Teaching

Abstract

A learning process involves interactions between learners, teachers, machines and formal and/or informal learning environments. These interactions are relational, interdependent and temporal. The emergence of rich multimodal learner data suggests the development of methods that can capture time-stamped data from multiple sources (e.g., heart rate data and eye tracking data), thus allowing researchers to examine learning as a continuous process rather than a static one. This leads us to propose a new methodological approach, the Multimodal Temporal Network Analysis to: i) measure temporal learner data deriving from the relevant interactions and ii) ultimately support learners and their teachers in learning and/or teaching activities.

Keywords

• Multimodal learning analytics
• temporal networks
• social network analysis

INTRODUCTION

Learning occurs across both formal and informal learning settings and evolves as students interact with each other, machines, and/or with teachers, as they engage with multifaceted learning tasks. Such interactions are self- and socially regulated, temporal and interdependent (Järvelä et al., 2014). As a socially regulated process, learners’ activities are facilitated or constrained by peers while they negotiate their roles, tasks and work together for the achievement of their shared goals (Malmberg, Järvelä, & Järvenoja, 2017) . As a temporal process, learning follows the universal law of time, and so are the interactions and learning activities, they are forward moving, unidirectional and uniform (Saqr, Fors, & Nouri, 2019) . As an interdependent process, learning activities and events are largely interdependent. To understand learning as an outcome, we need to understand the processes and sequences of past events, i.e., learning as a continuous process, which is multidimensional, complex and rich (Malmberg et al., 2017) . An adequate understanding of such a process requires new innovative methods that can capture learning and its related activities as a continuous process rather than a static one. Multimodal learning analytics (MMLA) have emerged to address this issue. 1.1

Background

MMLA uses multiple synchronized sensing modalities to record learners’ interactions, spatial data, physiological indicators as well as eye- and body movements. For example, physiological measurements such as heart rate data can be linked to certain learner’s experiences (Ochoa & Worsley, 2016) . Multimodal data can be recorded in real-time and amass unprecedented volumes of high resolution learner temporal data. As researchers try to make sense of these complex data, they have used several approaches for analysis either separately or in combination. Such approaches include traditional statistics, machine learning and qualitative methods (Viberg, Hatakka, Bälter, & Mavroudi, 2018) . The complex interactions among learners - and learning resources - were earlier studied using well established network representations (Cela, Sicilia, & Sánchez, 2014) , which employ network methods (i.e., powerful tools for the study of the relational data). They have been used successfully by educational researchers to for example, intuitively map interactions in simple understandable visual graphs, to reveal the structural dynamics of groups of learners, and to identify roles and influencers in a collaborative environment (Cela et al., 2014) . To represent the relations as a network, researchers often aggregate all interactions in what is known as an ‘aggregate’ or static network (i.e., a compilation of all interactions). In doing so, the static network representation ignores the time aspect, considers that relations are permanent, and disregards the dynamics of the represented interaction process and related learning activities (Holme, 2015) . As such, static network representations are much limited in terms of a holistic understanding of learning as a continuous process occurring when students interact with: each other, teachers, the available learning resources and involved learning environments. Compressing the time dimension is reductionist and arguably simplistic. Earlier learning analytics studies have shown the importance of taking time into account when analyzing learning events (e.g., Chen, Resendes, Chai, & Hong, 2017; Malmberg et al., 2017; Molenaar & Järvelä, 2014; Saqr et al., 2019) . 2

METHOD PROPOSAL

We argue that extending the current approach by retaining the temporal dimension and its related information is beneficial to: i) understand the continuous nature of the learning process, and ii) further suggest related actions aimed at improving student learning outcomes and relevant learner support and teaching. A multimodal temporal network analytical approach is thus believed to have the potential to help researchers to unravel the timeline of learning events, the sequence of interactions and the relational properties of the learning process; most importantly, its evolving nature. The captured multimodal data from multiple streams are both temporal and relational as they capture time-stamped interactions. Consequently, temporal networks could offer a solid model for representing multimodal data in meaningful ways. Nowadays, research in temporal networks methods have given rise to a growing set of visual and mathematical methods. Such methods have contributed to the understanding of complex phenomena such as information spread, modelling disease contagion and brain connectivity, for a review please see (e.g., Holme, 2015; Holme & Saramäki, 2012) . For education, temporal network analysis of multimodal data offers powerful representations and modeling of the temporal dimensions (e.g., timing of interactions among learners and teachers, timing of interactions with learning resources, timing of interactions with learning environment/s) that underpin learning- and teaching processes. While other methods of Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 Internationa2l (CC BY 4.0). temporal analysis, such as using time series analysis offer a rich tool set for temporal analysis, they do not fully cover the relational continuous nature of interactions in a learning environment. Nonetheless, both methods are complimentary, and recent research is exploring methods to combine the strengths of each method. We propose a three step approach to Multimodal Temporal Network Analysis to improve learner support and teaching. Such an approach is suggested to include three key mutually constituting parts: data, representations and analysis (Figure 1)

Data

• • • • •

Multimodal data: spatial and proximity data Audio data and discourse capturing Video data Log-file data Physiological measurements such as eye movement, electro-dermal activity (galvanize skin response) Representation Networks enable the representation and modeling of the collected data • proximity, audio, computer mediated interactions and spatial data be represented as networks of interactions among learners • proximity, eye interaction with the elements of learning environment such as equipment, artefacts or laboratory tools could be represented as affiliation networks. • physiological data: o as networks of physiological synchronization among collaborators o physiological data such as heart rate could be incorporated as edge weights or signs.

Analysis

•

Temporal networks methods offers several models for the visualization (i.e., learner-and teacher support mechanisms) and the mathematical analysis of networks such as the spread of information, the evolution of communities, influencers, and the key drivers of the process. Future research directions By applying multimodal temporal network analysis, we suggest that we can better understand multifaceted aspects of temporal learning processes occurring in learners’ interactions with each other and/or teachers, as well as the interactions with the involved learning environments and learning resources in use.

Some examples of potential research questions that could be addressed include: Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 Internationa3l (CC BY 4.0). • How can we understand the social regulation of collaborative learning that unfolds and develops over time? • How do successful teams of learners manage learning tasks, and what characterizes a successful team process? • Can temporal network representation offer an accurate model for the understanding of group dynamics, and if so, how? • What are the temporal characteristics of productive collaboration considering the interplay between stress levels (biometric data), communication (audio or text), and eye-movements (video)? Example: capturing multimodal data of a group of learners, audio data can be used to obtain a network of students’ interactions; eye tracking and video data could be used to obtain another network of eye contact; physiological sensors could be used to capture levels of physiological arousal. Mapping these multiple signals together one could understand the interactions that lead to successful social regulation of teamwork, when they happened and how they progressed.

All in all, we propose to develop and adopt a new methodological approach for MMLA research, the Multimodal Temporal Network Analysis that, on the one hand, incorporates temporal aspects of learning as an analytical lens in order to capture learning as a continuous process, and on the other hand, combines it with network analysis as an analytical method in order to also capture the interdependent nature of learning interactions. By doing so, we argue that MMLA research is enhanced with a stronger ability to represent and model the complex interdependent multimodal learning interactions and processes that take place in space as well as in time.

REFERENCES

Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 Internationa4l (CC BY 4.0). Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 Internationa5l (CC BY 4.0).

References

Cela , K. L. , Sicilia , M. Á. , & Sánchez , S. ( 2014 ). Social Network Analysis in E-Learning Environments: A Preliminary Systematic Review . Educational Psychology Review , 27 ( 1 ), 219 - 246 . https://doi.org/10.1007/s10648-014-9276-0

Chen , B. , Resendes , M. , Chai , C. S. , & Hong , H.-Y. ( 2017 ). Two tales of time: uncovering the significance of sequential patterns among contribution types in knowledge-building discourse . Interactive Learning Environments , 25 ( 2 ), 162 - 175 . https://doi.org/10.1080/10494820. 2016 .1276081

Holme , P. ( 2015 ). Modern temporal network theory: a colloquium . European Physical Journal B , 88 ( 9 ). https://doi.org/10.1140/epjb/e2015-60657-4

Holme , P. , & Saramäki , J. ( 2012 ). Temporal networks . Physics Reports , 519 ( 3 ), 97 - 125 . https://doi.org/10.1016/j.physrep. 2012 . 03 .001

Järvelä , S. , Kirschner , P. A. , Panadero , E. , Malmberg , J. , Phielix , C. , Jaspers , J. , … Järvenoja, H. ( 2014 ). Enhancing socially shared regulation in collaborative learning groups: designing for CSCL regulation tools . Educational Technology Research and Development , 63 ( 1 ), 125 - 142 . https://doi.org/10.1007/s11423-014-9358-1

Malmberg , J. , Järvelä , S. , & Järvenoja , H. ( 2017 ). Capturing temporal and sequential patterns of self-, co-, and socially shared regulation in the context of collaborative learning . Contemporary Educational Psychology , 49 , 160 - 174 . https://doi.org/10.1016/j.cedpsych. 2017 . 01 .009

Molenaar , I. , & Järvelä , S. ( 2014 ). Sequential and temporal characteristics of self and socially regulated learning . Metacognition and Learning , 9 ( 2 ), 75 - 85 . https://doi.org/10.1007/s11409-014-9114-2

Ochoa , X. , & Worsley , M. ( 2016 ). Augmenting Learning Analytics with Multimodal Sensory Data . Journal of Learning Analytics , 3 ( 2 ), 213 - 219 . https://doi.org/10.18608/jla. 2016 . 32 .10

Saqr , M. , Fors , U. , & Nouri , J. ( 2019 ). Time to focus on the temporal dimension of learning: a learning analytics study of the temporal patterns of students' interactions and self-regulation . International Journal of Technology Enhanced Learning , 11 ( 4 ), 398. https://doi.org/10.1504/ijtel. 2019 .10020597

Viberg , O. , Hatakka , M. , Bälter , O. , & Mavroudi , A. ( 2018 ). The current landscape of learning analytics in higher education . Computers in Human Behavior , 89 ( July ), 98 - 110 .