The first workshop on Multimodal Learning Analytics took place in 2012 at the International Conference of Multimodal Interaction in Santa Monica, California. Since then researchers have been organizing annual workshops, tutorials, webinars, conferences and data challenges. This paper examines the body of research that has emerged from these workshops, as well as in other published proceedings and journals. Within this review, a distinction is drawn between empirical and position/review/dataset papers, and looks at their rate of publication over the past several years. Additionally, key characteristics of empirical papers as well as current trends from non-empirical papers provide key insights from which to reflect on the progress of MMLA to date, and identify potential future opportunities. Specifically this review suggests that greater attention should be paid to using deep learning, developing simpler data collection tools, and finding ways to use MMLA to support accessibility.
At its essence MMLA aims to leverage data from non-traditional modalities in order to study and
analyze student learning in complex learning environments. In an effort to advance research within
this domain, researchers have been organizing workshops and data challenges for the past five years
This review of the MMLA literature includes 82 papers. The first step in examining these papers was
to determine which were empirical. This classification was based on whether or not the paper included
an explicit study and analysis, versus those that present a position, a dataset or a review. 46 papers
After looking at the year each paper was published, empirical papers were coded for the modalities captured, analytic techniques utilized, dependent variable, location of the study (ecological versus laboratory), whether the study was computer-mediated, the age group of the participants and whether or not the task and analysis were collaborative.
18 16 14 12 10 8 6 4 2 0 2011 2012 2013 2014 2015 2016 2017
2018 Empirical
Non-Empirical Study design encapsulates the nature of the task, the participants, and the location of the study. The study design analysis begins by considering collaboration. Collaboration was coded at two levels. Specifically, the coding process recorded if the task that students completed were collaborative as well as whether or not the analysis looked at group level outcomes or individual level outcomes. Looking at the level of collaboration in the analysis will be presented later. The empirical papers included an even split between collaborative and individual tasks.
Ecological studies, i.e. those that took place outside of laboratory contexts, represented 13 of the empirical papers, while the remaining 35 were conducted in laboratory settings (Figure 2). Additionally, 33 of the studies involved computer-mediated activities while the remaining 15 involved non-computer mediated activities.
Finally, only two of the papers worked with elementary school students. The remaining 44 working with either high school or college students (at both the undergraduate and graduate levels). 13
35 Laboratory
Ecological Consistent with the goals of multimodal learning analytics, researchers have drawn upon a large number of modalities that include audio, video, gestures, electro-dermal activation, emotions, cognitive load and several others. The five most frequently utilized modalities are as audio, video, biophysiology, eye tracking and digital interactions. Figure 3 includes a pie chart describing how frequently these modalities are used among the 46 empirical papers. 20 18 16 14 12 10 8 6 4 2 0
Audio
Video
Biophys. Eye Trackers Digital Log
data Additionally, 27 of the 46 papers use at least three modalities, with the vast majority using at least two modalities. To put these modalities in context, researchers frequently used audio to analyze participant speech, and video to study body language, using both real-time and post-hoc gesture and posture tracking. Similarly, the bio-physiological measures are regularly used to study student arousal and/or affective state. Eye tracking and audio analysis are the two modalities that researchers frequently use as single modality analyses. That said, both eye tracking and audio (speech) can be easily analyzed for a variety of features (i.e., cognitive load, arousal, sentiment, entrainment). Importantly, some of the non-empirical papers present novel data collection tools. Most notable is the Multimodal Selfies work that features synchronous two-channel audio, video and digital pen input through a Raspberry Pi. 2.3
Researchers have looked at a number of constructs within their respective studies. These constructs often reflect the theoretical orientation that the researchers are following. Nonetheless, the coding of dependent variables identified several common classes of dependent variables. The constructs that emerged across multiple papers include: learning, multimodal behavior/engagement, expertise, collaboration quality, presentation quality, joint attention, affect and success (Figure 4). Understandably, the ways that researchers instantiate each of these constructs is highly variable, with some relying on human coding, while others utilize heuristics. Similarly, researchers utilize a number of different modalities to ascertain the same construct. For example, some researchers used speech signals to study affect, while others used facial expressions. 14 12 10 8 6 4 2 0
In considering the aforementioned learning constructs, approximately 22% looked at the group as the primary unit of analysis, 48% looked at just the individual, and 30% looked at both individuals and groups.
Some preliminary data was coded regarding the analytic techniques used to analyze the different data
streams. Several of the analyses relied on custom developed scripts, though most leveraged existing
code bases and/or toolkits to conduct the analyses. Examples of existing tools that researchers used
include: Linguistic Inquiry Word Count (LIWC)
In summary, prior work in multimodal learning analytics has been heavily geared towards studying groups of learners using a broad number of modalities completing computer-mediated tasks. The majority of the studies describe work completed in laboratory settings, and primarily includes high school and college students. Taken together, the past studies highlight the feasibility to capture and analyze multimodal data, but demonstrate this capability within a somewhat limited set of contexts and with a limited set of participant types. Turning to more present day analyses, researchers are aiming to address some of these limitations.
Note: A number of papers included within this review were based on data sets distributed in conjunction with data challenges and workshops. In particular, the 2013 and 2014 Multimodal Learning Analytics Grand Challenge Workshops were centered around the Multimodal Math Data Corpus and the Public Speaking Corpus. 3
3.1
Consideration of present work in MMLA is based on ideas raised in non-empirical papers from 2015
2017. Each document was open-coded for the central ideas that emerged. After the initial coding
process like terms were grouped into categories. This section presents a summary of those categories.
A central idea that emerged from several works
Another central component of contemporary MMLA is the development of frameworks and models
that offer better generalizability and applicability
Researchers are also looking to address challenges of data visualization, integration with existing data
analysis tools, and the creation of new data analysis tools
Another theme is opportunities to conduct research that sits at the intersection of human-computer
collaboration
This category reflects current work
The current orientation of CrossMMLA reflects one of the current trends within the MMLA community. Specifically, researchers are increasingly engaged in using MMLA to study student learning across different digital and physical spaces, and in increasingly ecological, or real world, contexts. Conducting such work presents a number of novel challenges in terms of data collection, interoperability and standardization.
These categories by no means represent the entirety of current research in MMLA. However, they do touch on several of the cross-cutting ideas that are being advanced by multiple researchers within this field. 4
Having considered the past and the present, this paper now turns to considering potential futures. Reasonably, there are several potential directions that MMLA research could take. Here, I highlight areas that may be fruitful for advancing the field, especially given the overall motivation and prior work in MMLA. 4.1
Absent from current work in MMLA is considerations for how these technologies can promote
accessibility and inclusivity in learning. Put differently, MMLA has the potential to create novel
learning experiences for people with disabilities
In examining the many analytic techniques currently employed in MMLA research, there appears to be significant underrepresentation of deep learning algorithms, especially given the extent to which deep learning is transforming the field of artificial intelligence. As we endeavor to stay on the cuttingedge, it will be important for the field of MMLA to find ways to leverage deep learning. In particular, several existing deep learning algorithms have the ability to easily be adapted to a specific domain on context by retraining the final layers of a deep neural network for example. In the case of computer vision, for example, the initial layers are, ostensibly, beneficial for extracting common features from an image, while the later layers are trained to handle the peculiarities of a given dataset. Researchers are also leveraging deep learning to conduct much more complex gesture tracking within large groups of people. Outside of computer vision, deep learning is also proving to be quite useful for natural language processing, both in building language models and, potentially for speaker identification. Taking advantage of these capabilities could offer a significant boost in MMLA research. 4.3
In addition to thinking about accessibility and deep learning, the field is still in need of significantly simplified data collection, and analysis, tools. At present, the challenges associated with synchronously collecting multimodal data, is a significant impediment for many researchers. Too many researchers continue to rely on custom developed scripts and manual data alignment for MMLA to be tractable and accessible for those who are not already invested in this type of research. 5
This paper presented a preliminary literature review about Multimodal Learning Analytics. It used a number of criterion to identify trends in MMLA, as well as opportunities for future development. While the process for coding these papers could easily be extended to consider more details about specific data collection tools, the best approaches for multimodal triangulation or the types of analyses completed, the current literature review suggests that past and present work in multimodal learning analytics has laid a strong foundation for on-going research. Importantly, researchers have demonstrated the ability to collect multimodal data from groups of students in ecological settings, and to conduct analyses at both the individual and group levels. Moving forward the field appears to be poised to continue working in ecological settings, and to now expand towards collecting data across digital and physical spaces, while also taking advantage of the affordances of mobile technology. Furthermore, we are likely to see the development of more robust frameworks, simplified data collection and analysis tools and agreed upon standards. The field can advance this work further by taking full advantage of and contributing to deep learning. More importantly, the field would benefit for considering the ways that MMLA can positively contribute to accessibility.