Multimodal Learning Analytics (MMLA) has enabled researchers to address learning in physical settings which have long been either overlooked or studied using observational methods. With the use of sensors, researchers have been able to understand learning through an entirely new perspective (e.g., analyzing heart-rate variability to find collaboration indicators). Consequently, MMLA has grown significantly in the past few years, moving from a nascent stage towards a more mature field. It raises a question on how the MMLA researcher can move further, i.e., the transition towards practice which started getting researchers' attention. This paper discusses the challenges we faced while conducting MMLA studies in classroom settings over four years and potential solutions to realize the goal of transitioning MMLA research to educational practice. This paper aims to start a discussion in the field of MMLA over the transition of research to practice.
Traditional log-based Learning Analytics (LA) has utilized digital traces of students’ activities to
understand learning and its context, which only provides a partial picture of a learning situation
that occurs across spaces (e.g., collaborative learning in face-to-face settings). To address this
issue, researchers have started employing various types of data sources (e.g., audio, video) to
complement log-based LA analysis [
Researchers have employed MMLA for studying a range of learning scenarios (e.g.,
projectbased learning [
MMLA Researchers have explored the relationship between various data features and
collaboration constructs (e.g., collaboration quality) [
The majority of current MMLA research in CSCL research has been conducted in laboratory settings [10] which allow more controlled settings than classroom settings. However, students may not necessarily behave in a similar way when participating in a learning activity in laboratory and classroom settings [11]. Furthermore, in our previous vignette study where we investigated teachers’ trust in the MMLA dashboard for CSCL, teachers indicated trusting the data from the dashboard and proposing a variety of interventions in the imagined classroom [12]. However, we were unable to replicate those findings in authentic settings. These concerns raise the need of conducting MMLA research in classroom settings. However, given its technological complexity, the use/implementation of MMLA in classroom settings is not straightforward.
In this paper, we share our experience of conducting MMLA research in authentic settings and building MMLA tools for supporting teachers [13, 14]. We conducted two iterations of the MMLA study in Estonian high school and vocational school. We collected multimodal educational data to build automated models for collaboration behavior to build a guiding tool to support teachers with intervention strategies during collaborative learning.
The goal of our research was to build automated models which can estimate collaboration quality and its underlying dimensions (e.g., argumentation) as per Rummel et al. [15] framework, and integrate the developed models in intervention guiding tools to support teachers in the classrooms. We developed our prototype in two iterations (later explained). Table 1 presents the stakeholders involved in both iterations. iteration teachers students researchers schools
In our first iteration (Figure 1), we developed a Raspberry Pi-based prototype to capture the
speaking behavior of participants of a group working on a given problem in classroom settings.
The prototype was used along with Etherpad1 which allowed the group’s participants to draft
solutions to the given problem. We collected multimodal data (audio and logs) from two
classroom sessions in an Estonian High school during a collaborative learning activity with the
same teacher [
In our next iteration (Figure 2), we developed a web-based application called CoTrack2 to simplify the use of multimodal data collection in f2f and to also make it possible in online learning settings [13, 16]. With the development of CoTrack, we automated all the steps which needed 1Etherpad is a real-time collaborative text editor. 2https://www.cotrack.website manual setup in the first prototype (i.e., time synchronization, and feature pre-processing). In our second iteration, the study involved 174 students from 4 diferent schools, 9 diferent subject teachers, and 2 researchers (Table 1) [13]. The data from this study was used for developing automated collaboration estimation models for building intervention guiding tools.
Here, we outline the main challenges we faced during the enactment of our MMLA research in the classroom setting. #1. Technical expertise was needed to set up the first prototype to collect multimodal educational data.
The first challenge we faced was associated with the technological complexities inherent in MMLA. The developed prototype in the first iteration required a particular setup with a specific set of commands (e.g., running an MQTT server, starting Raspberry Pi devices, setting up an ad-hoc network, and configuring time synchronization). Thus, there was a need for a person with technical expertise whenever the prototype was used, resulting in data collection from fewer classroom sessions.
Solution: We addressed this challenge by automating most of the technological configuration steps. It resulted in a prototype (iteration-2) with an easy-to-use interface without the need for technical expertise which was needed in the first version of the prototype. The prototype allowed teachers to create collaborative learning activities with multimodal data capturing and monitoring functionality without the need for a technical expert. Another potential solution could be to use a fixed MMLA setup that is already configured in a classroom.
#2. The prototype used in classroom settings caused obtrusiveness in the learning process.
The prototype’s usage from the first iteration required a preliminary set-up phase which caused interruptions in the classroom. During the study, the setup took an additional time of (10-15 minutes) before starting the learning activity. Moreover, during the activity, the researchers present in the classroom were ensuring the device’s functioning which most likely distracted students’ attention.
Solution: We minimized the obtrusiveness by using a minimal number of devices for data collection purposes. We used the laptop’s microphone and video camera for data collection purposes.
#3. Covid-19 restrictions complicated the situation further.
The arrival of the pandemic Covid-19 put restrictions on physical participation in Estonian schools which closed all possibilities of using the developed prototype (version-1) in classroom settings for multimodal data collection.
Solution: We tackled this challenge in our second iteration by transforming our raspberry pi-based prototype into a complete web-based application. This feature allowed us to collect more data in our second iteration of the study. Furthermore, this also enabled researchers without technical expertise to use multimodal data-capturing functionality in their research.
#4. An increased number of requests sent for storing extracted multimodal features exhausted the server’s resources.
As we had integrated Voice Activity Detection (VAD) and Speech-to-Text in CoTrack, it increased the number of requests received on the server. Furthermore, CoTrack also stores media files (e.g., audio and video) that need a larger space on the server. These posed the issue of scalability. Solution: We are now exploring cloud-based solution to store our multimodal data and deal with the challenge of server resource exhaustion.
#5. Students found it dificult to follow MMLA-related instructions along with learning activities.
Though we simplified the use of CoTrack in comparison with the Raspberry Pi-based prototype, there were still a few instructions for students to follow while using CoTrack. For example, one instruction was to click on the finish button once the learning activity is done which saved the ifnal audio (or video) file on the server. However, during our data collection, we observed many instances when students simply closed the application without clicking on the finish button which caused a loss of audio/video data.
Solution: This challenge made us realize the need for the MMLA tool with minimal reliance on students for any data collection-related controls. We are now aiming to eliminate this requirement by simply saving media files (audio/video) in the background when students leave the learning activity space in CoTrack.
#6. Multimodal data collection had a lower quality of data due to the noisy settings of the classroom.
Multimodal data collected from classroom settings are most likely to be of a lower quality than data collected from a controlled setting. For example, in the case of face-to-face group activity where participants sit close to each other, the chances of audio data getting corrupted become higher.
Solution: The data from classroom settings is most likely to have noise, thus, there should be a mechanism to deal with that noise or we should employ data analysis (or modeling) techniques which are noise resistance. In this direction, the next step could be to first understand how the presence of noise in data afects results from analysis or modeling. Another potential idea to deal with noisy data is to exploit cross-modal features. For example, speaking time extracted from audio data can be corrupted as a result of loud noises in the classroom; here, we can extract the same feature also from video using computer vision (e.g., lip-activity detection [17]) and then utilize it in a complementary way. #7. Issue of variation in learning contexts on multiple aspects.
The learning contexts from where the datasets were collected varied on multiple aspects, e.g., schools, subjects, educational level, communication language, teacher, etc. This raised our concern over the applicability of analysis results to a broader context.
Solution: To deal with this challenge of context variance, broader data gathering in multiple learning contexts is needed which could further enable a stricter evaluation of the generalizability of analysis results, e.g., across contexts evaluation [8, 13]. #8. Complexity of multimodal data visualization.
During our initial phases of prototype development, we involved teachers in the design phase [18]. Teachers reported having freedom in selecting what analytics to see rather than having all the data plotted for them. In addition, they also suggested having some abstract representation for the group’s participation or to show collaboration quality levels.
Solution: We addressed this challenge by incorporating teachers’ suggestions in our current version of CoTrack. We now plan to investigate teachers’ perspectives on the usefulness of CoTrack in monitoring and guiding collaborative learning activities in the classroom.
This paper shares our experience of conducting MMLA research in classroom settings in terms of the challenges faced and how we handled them. We also presented some open challenges (e.g., noisy data from the classroom) with some ideas on their potential solution. We like to mention that our solutions are the result of general engineering and are not MMLA-specific. Some solutions (e.g., for #5) can be related to already existing design guidelines for edTech, e.g., minimalism principle for classroom orchestration [19].
In the workshop, how other MMLA researchers have solved challenges in authentic settings could be discussed. The knowledge from the such discussion could result in a manual or tutorial about applying MMLA in authentic settings which we hope can help researchers to conduct MMLA studies in authentic settings. oration? A review of the empirical links between multimodal metrics and collaborative constructs, Sensors 21 (2021). [8] P. Chejara, L. P. Prieto, A. Ruiz-Calleja, M. J. Rodríguez-Triana, S. K. Shankar, R. Kasepalu, Efar-mmla: An evaluation framework to assess and report generalizability of machine learning models in mmla, Sensors 21 (2021) 2863. doi:10.3390/s21082863. [9] L. Yan, L. Zhao, D. Gasevic, R. Martinez-Maldonado, Scalability, sustainability, and ethicality of multimodal learning analytics, in: LAK22: 12th International Learning Analytics and Knowledge Conference, ACM, New York, NY, USA, 2022, pp. 13–23. [10] M. Worsley, Multimodal learning analytics’ past , present , and , potential futures, in: 2nd Multimodal Learning Analytics Across (Physical and Digital) Spaces, CrossMMLA, 2018, pp. 1–16. [11] A. Kharrufa, M. Balaam, P. Heslop, D. Leat, P. Dolan, P. Olivier, Tables in the wild: Lessons learned from a large-scale multi-tabletop deployment, Conference on Human Factors in Computing Systems - Proceedings (2013) 1021–1030. doi:10.1145/2470654.2466130. [12] R. Kasepalu, P. Chejara, L. P. Prieto, T. Ley, Do teachers find dashboards trustworthy, actionable and useful? a vignette study using a logs and audio dashboard, Technology, Knowledge and Learning 27 (2022) 971–989. [13] P. Chejara, L. P. Prieto, M. J. Rodriguez-Triana, R. Kasepalu, A. Ruiz-Calleja, S. K. Shankar, How to build more generalizable models for collaboration quality? lessons learned from exploring multi-context audio-log datasets using multimodal learning analytics, in: LAK23: 13th International Learning Analytics and Knowledge Conference, LAK2023, ACM, New York, NY, USA, 2023, pp. 111–121. [14] R. Kasepalu, L. P. Prieto, T. Ley, P. Chejara, Teacher artificial intelligence-supported pedagogical actions in collaborative learning coregulation: A wizard-of-oz study, Frontiers in Education 7 (2022). URL: https://www.frontiersin.org/articles/10.3389/feduc.2022.736194. doi:10.3389/feduc.2022.736194. [15] N. Rummel, A. Deiglmayr, H. Spada, G. Kahrimanis, N. Avouris, Analyzing collaborative interactions across domains and settings: An adaptable rating scheme, Springer US, Boston, MA, 2011, pp. 367–390. [16] P. Chejara, L. P. Prieto, M. J. Rodriguez-Triana, A. Ruiz-Calleja, M. Khalil, Impact of window size on the generalizability of collaboration quality estimation models developed using multimodal learning analytics, in: LAK23: 13th International Learning Analytics and Knowledge Conference, LAK2023, ACM, New York, NY, USA, 2023, pp. 559–565. [17] S. Siatras, N. Nikolaidis, M. Krinidis, I. Pitas, Visual lip activity detection and speaker detection using mouth region intensities, IEEE Transactions on Circuits and Systems for Video Technology 19 (2009) 133–137. doi:10.1109/TCSVT.2008.2009262. [18] P. Chejara, R. Kasepalu, A. Ruiz-Calleja, M. J. Rodriguez-Triana, L. P. Prieto, Codesigning a multimodal dashboard for collaborative analytics, in: A. Weinberger, W. Chen, D. Hernández-Leo, B. Chen (Eds.), 15th International Conference on Computer Supported Collaborative Learning, CSCL 2022, Hiroshima, Japan, June 6 –10, 2022, International Society of the Learning Sciences, 2022. URL: https://repository.isls.org/bitstream/1/7849/1/ 577-578.pdf. [19] P. Dillenbourg, Design for classroom orchestration, Computers & education 69 (2013) 485–492.