=Paper=
{{Paper
|id=Vol-2610/paper5
|storemode=property
|title=Designing and implementing multimodal data collection in classroom to capture metacognition in collaborative learning
|pdfUrl=https://ceur-ws.org/Vol-2610/paper5.pdf
|volume=Vol-2610
|authors=Jonna Malmberg,Sanna Järvelä, Hanna Järvenoja,Eetu Haataja,Ahsen Cini
|dblpUrl=https://dblp.org/rec/conf/lak/MalmbergJJHC20
}}
==Designing and implementing multimodal data collection in classroom to capture metacognition in collaborative learning==
https://ceur-ws.org/Vol-2610/paper5.pdf
Companion Proceedings 10th International Conference on Learning Analytics & Knowledge (LAK20)
Designing and implementing multimodal data collection in
classroom to capture metacognition in collaborative learning
Jonna Malmberg1, Sanna Järvelä1, Hanna Järvenoja1, Eetu Haataja1, Héctor J. Pijeira-Díaz2,
Ahsen Cini1
1University of Oulu
2 Maastricht University
Jonna.malmberg@oulu.fi, sanna.jarvela@oulu.fi, hanna.jarvenoja@oulu.fi,
eetu.haataja@oulu.fi, h.pijeiradiaz@,aastrichtuniversity.nl, Ahsen.Cini@oulu.fi
h.pijeiradiaz@maastrichtuniversity.nl
ABSTRACT: While prominent empirical research exploring the possibilities to utilize different
data channels in the research of regulation in collaborative learning is emerging, we are still
in the process of discovering the relevant combinations of different data sources and proper
ways to combine data from different channels. This is the case particularly with
metacognition. The potential of using multiple data channels lies also in their power to be
transferred as a tool for providing learners ‘on the fly’ support for regulation when needed.
However, an advanced understanding of the regulated learning in collaborative learning
contexts, and particularly on metacognitive processes is essential to harness the benefits of
technology in supporting these processes in collaborative learning.
Keywords: Metacognition, collaborative learning, multimodal data
1 INTRODUCTION
Learning processes are hard to predict or model, since learning is always situated, dependent on
the learning context and the learner’s individual metacognition. Metacognitive knowledge involves
learners' perceptions of a task. It draws to prior knowledge in terms of same types of tasks and
procedures needed to perform those (Winne & Hadwin, 1998). Another component of
metacognition are metacognitive experiences. Metacognitive experiences constitute, for example
learners’ perceptions of task difficulty. Unlike task understanding, which is thoughtful and
cognitive, perception about task difficulty is reactive, and is also informative for Self-Regulation of
Learning (SRL) (Winne & Hadwin, 1998). Multimodal data (e.g., physiological measures, videos, and
situated self-reports) can provide a new unobtrusive way to capture learners’ metacognition without
interrupting learning process (Järvelä et al.,2019). Currently, there is an accumulating evidence on
how physiological measures can be used to track learning. Recent studies have shown that the level
of students’ physiological arousal is related to learners’ metacognition (Hajcak, McDonald, &
Simons, 2003) and achievement (Pijeira-Diaz et al. 2018). Metacognition, in turn, is related to
learners’ perceptions of tasks, self and learning situations (Flavell, 1979). Yet, current research lack
methods to capture the situated nature of task perceptions in the context of collaborative learning
over time.
Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International
(CC BY 4.0).
Companion Proceedings 10th International Conference on Learning Analytics & Knowledge (LAK20)
In this paper, the focus is to (1) introduce collaborative learning model designed to study processes
focusing on metacognition and promoting awareness of metacognition in a secondary school science
classroom, (2) describe multimodal data collection procedure implemented in secondary school
science classroom and (3) illustrate with two case examples how multimodal data has been used to
capture learner’s metacognition. Participants of the study were (N = 94) upper elementary school
students aged 13 to 14 (58 females, 36 males) enrolled in compulsory physics course consisting
altogether five lessons. In each lesson, the students collaborated in the same groups of three to four
students based on the collaborative learning model. Altogether, the students had four 90 min
physics lessons, once in a week and the last lesson was a collaborative exam. In addition, after each
lesson, the students filled in a multiple-choice knowledge test consisting of five questions related on
topics they had just learned.
2 COLLABORATIVE LEARNING MODEL
The collaborative learning model designed for science class is based on a self-regulated learning
framework that provides opportunities and awareness for self-initiated regulation among individual
learners and collaborative groups (authors). It utilizes technology-based environment called Qridi®
(https://kokoa.io/products/qridi), which was designed to structure collaboration. The collaborative
learning model is built on the idea of a ‘flipped classroom.’ Recently, the flipped classroom concept
has been gaining considerable attention due to its potential to facilitate the regulation of learning
(Jovanovic et al., 2019). The use of a flipped classroom in collaborative learning creates a learning
setting in which students are provided opportunities to take responsibility for their own learning by
familiarizing themselves with the content knowledge beforehand to prepare for collaborative
learning. In the current study, the flipped classroom structure and the collaborative work were
coordinated by using a Qridi® (Figure 1). However, the learning materials were not provided via
Qridi®, but the students used their own regular physics books.
In the Qridi® environment, students were able to check, for example, the phase of the lesson. In our
learning model, Qridi® was tailored to increase students’ awareness of the collaborative learning
task phases in general and, specifically, supporting their awareness of the regulation of learning. For
example, Qridi® involved a 6Q tool designed to promote students’ situation-specific metacognitive
awareness related to task understanding and task difficulty before and after the collaborative
learning. In practice, the 6Q tool consists of two 0–100 slider-scale questions where students
estimate their task understanding (Schraw and Dennison, 1994), perceived task difficulty (Efklides et
al. 1998).
Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International
(CC BY 4.0).
Companion Proceedings 10th International Conference on Learning Analytics & Knowledge (LAK20)
Figure 1. Collaborative learning model
2.1 Multimodal data collection
As the students study according to collaborative learning model, multiple data sources were
collected. Prior to the study, the participating students responded to trait-type questionnaires such
as Metacognitive Awareness Inventory (MAI) (Schraw and Dennison, 1994) that captured their
individual metacognitive beliefs. During the seven-week multichannel data collection process,
students’ collaborative work was followed by video recordings. Shimmer3 GSR+ sensors with 128hz
sampling rate were used to measure learners’ electrodermal activity (EDA) indicating arousal. The
sensors were automatically synchronized with each other in the dock station before the start of each
session. Students fitted the devices at the beginning of each lesson and took if off at the end. In this
way, continuous EDA data was obtained for each student during the entire lesson. As one of the
multiple data sources, we used the 6Q tool implemented in Qridi® to collect students’ situation-
specific interpretations of their metacognition in terms of task understanding and task difficulty
related to each collaborative session before and after the collaborative work. Altogether the
students had five physics lessons and the last one was collaborative exam.
2.2 Processing physiological data
First, files having contact issues were removed from the dataset. Second, Butterworth low pass filter
with frequency 1 and order 5 was used to remove small movement artifacts from the signal. Third,
Ledalab toolbox and through-to-peak analysis with minimum amplitude of 0.05μS was used for peak
Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International
(CC BY 4.0).
Companion Proceedings 10th International Conference on Learning Analytics & Knowledge (LAK20)
detection (Benedek & Kaernbach, 2010). Number of non-specific skin conductance responses per
minute (NS.SCR/min) for the session was used as a marker of arousal (Boucsein, 2012).
3 CASE EXAMPLES – WHAT ABOUT METACOGNITION?
The case examples provide insight on how to use multimodal data to investigate fluctuation of task
difficulty and task understanding during collaborative learning. The first case example illustrates
how individual learners’ metacognitive beliefs and situation specific perceptions of task difficulty
and task understanding are related on learning outcomes in the context of collaborative learning.
The second case example instead focuses on exploring how individual learners’ situation specific
interpretations of task difficulty and task understanding are related in physiological arousal in the
context of collaborative learning.
3.1 Analysis
In both of the case examples, Generalized Estimating Equations (GEE) was used. GEEs enable a
general method for analyzing clustered variables and ease several assumptions of traditional
regression models (Diggle, 2002; Liang & Zeger, 1998; Zeger & Liang, 1986). The GEE method does
not explicitly model between-cluster variation, rather it estimates its counterpart, the within-cluster
similarity of the residuals, and then uses this estimated correlation to re-estimate the regression
parameters and to calculate standard error. To estimate the validity of the GEE, QIC statistics
proposed by Pan (2001) allow comparisons of GEE models and selection of a correlation structure. In
both case examples, normal distributions with the log link function were selected because they
yielded the lowest quasi-likelihood under the independence criterion (QIC) values
3.2 How metacognitive beliefs and situated task perceptions relate for learning
outcomes?
With regard to first case example, generalized estimating equations (GEE) examine the effects of
individual metacognitive beliefs (MAI) and task perceptions which are task understanding (TU) and
task difficulty (TD) on upper elementary school students’ learning outcomes measured after each
lesson.
Table 1 shows that only learners’ interpretations on post-task understanding (Post TU) score can
effectively predict different actualized knowledge tests that were measured after each lesson. The
model fit statistics (QIC) scores was 258,986.
Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International
(CC BY 4.0).
Companion Proceedings 10th International Conference on Learning Analytics & Knowledge (LAK20)
Table 1. GEE results model using a normal distribution with a log link function predicting students’
learning outcomes
Dependent variable
Knowledge Tests
Independent variables B (95%CI) p value
Pre TU 0,001 (-0,001;0,003) 0,431
Post TU 0,002 (0;0,004) 0,038
Pre TD -0,0000576 (-0,002;0,002) 0,949
Post TD -0,001(-0,003: 0) 0,116
MAI 0,002 (-7,39E-05:0,004) 0,059
To summarize, learner individual metacognitive beliefs (which are quite static) do not predict
learning outcomes, but rather learner’s situation specific interpretations of the task after the
collaborative learning session predicts learning outcomes at individual level.
3.3 How individuals task perceptions relate for physiological arousal when
collaborative learning context is not or is considered?
With regard to second case example, generalized estimating equations (GEE) was used to examine
the effects of individual understanding (TU) and task difficulty (TD) on upper elementary school
students’ physiological arousal (NS.SCRs in minute) during collaborative exam first at individual level
(independent from the group) and second at collaborative level (exchangeable in the group).
In the light of the second case example, the results show, that when the collaborative learning
context is not considered, task perceptions dos not predict physiological arousal. The model fit
statistics (QIC) scores was 10,1.
However, when the group is considerd as exhchanceable, the results show that learners
interpretations before the task (pre-TU) score can effectively predict physiological arousal (NS.SRCs
in minute) (Table 2). The model fit statistics (QIC) score was 8,943.
Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International
(CC BY 4.0).
Companion Proceedings 10th International Conference on Learning Analytics & Knowledge (LAK20)
Table 2. GEE results model using a normal distribution with a log link function predicting students’
physiological arousal
Dependent variable
NS.SCRs / minute
Independent variables B (95%CI) p value
Pre TU 0,006 (0,001;0,10) 0,012
Post TU -0,004 (-0,010;0,002) 0,190
Pre TD -0,007 (-0,016;0,001) 0,086
Post TD -0,001(-0,10;0,008) 0,746
These two case examples shed a light in the process of discovering the relevant combinations of
different data sources and proper ways to combine data to investigate metacognition. The first case
example illustrates, that student characteristics, in terms of their metacognitive beliefs does not
predict learning outcomes. However, the way students perceive the task after the learning situation
predicts their learning outcomes.
The second example shows that when social context is taken account, task understanding predicts
physiological arousal. In both examples, learner’s situation specific interpretations of a task were
used as an indicator of metacognition. It can be concluded, that finding (relatively) unobtrusive ways
to measure and detect variations in learners task understanding as the learning proceeds, provides a
fruitful venue to explore ways to implement learning analytics and to provide learners feedback and
support for regulation when needed.
4 THE WORKSHOP PRESENTATION
To conclude, this presentation focuses on workshop theme: Examples of CrossMMLA research
designs and case examples by presenting 1) collaborative learning model designed to capture and
promote awareness of metacognition, 2) multimodal data collection implemented in science
classroom and 3) providing two representative case examples of multimodal data use to capture
metacognition focusing on task perceptions of learners. In the workshop, the aim is to illustrate in
detail how the collaborative learning model and multimodal data collection has been designed to
capture metacognition in the light of theories of regulated learning.
Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International
(CC BY 4.0).
Companion Proceedings 10th International Conference on Learning Analytics & Knowledge (LAK20)
REFERENCES
Boucsein, W. (2012). Electrodermal activity (2nd ed.). New York, NY: Springer.
Diggle, P. J., Heagerty, P., Liang, K.-Y., & Zeger, S. L. (2002). Longitudinal data analysis (2nd ed.).
Oxford, UK: Oxford University Press.
Benedek, M., & Kaernbach, C. (2010). Decomposition of skin conductance data by means of
nonnegative deconvolution. Psychophysiology, 47(4), 647–658.
https://doi.org/10.1111/j.1469-8986.2009.00972.x
Efklides, A., Papadaki, M., Papantoniou, G., & Kiosseoglou, G. (1998). Individual differences in
feelings of difficulty: The case of school mathematics. European Journal of Psychology of
Education, 13(2), 207-226.
Flavell, J. H. (1979). Metacognition and cognitive monitoring: A new area of cognitive–
developmental inquiry. American psychologist, 34(10), 906.
Ghisletta, P., & Spini, D. (2004). An introduction to generalized estimating equations and an
application to assess selectivity effects in a longitudinal study on very old individuals. Journal
of Educational and Behavioral Statistics, 29(4), 421-437.
Hajcak, G., McDonald, N., & Simons, R. F. (2003). To err is autonomic: Error-related brain potentials,
ANS activity, and post-error compensatory behavior. Psychophysiology, 40(6), 895–903.
https://doi.org/10.1111/1469-8986.00107
Jovanovic, J., Mirriahi, N., Gašević, D., Dawson, S., and Pardo, A. (2019). Predictive power of
regularity of pre-class activities in a flipped classroom. Computers & Education, 134, 156–
168.
Järvelä, S., Malmberg, J., Haataja, E., Sobocinski, M., & Kirschner, P. A. (2019). What multimodal data
can tell us about the students’ regulation of their learning process. Learning and Instruction,
https://doi. org/10.1016/j.
Liang, K.-Y., & Zeger, S. L. (1986). Longitudinal data analysis using Generalized LinearModels.
Biometrika, 73,13–22.
Pijeira-Díaz, H. J., Drachsler, H., Kirschner, P. A., & Järvelä, S. (2018). Profiling sympathetic arousal in
a physics course: How active are students?. Journal of Computer Assisted Learning, 34(4),
397-408.
Schraw, G., & Dennison, R. S. (1994). Assessing metacognitive awareness. Contemporary
EducationalPpsychology, 19(4), 460-475.
Winne, P.H., & Hadwin, A.F. (1998). Studying as self-regulated learning. In D.J. Hacker, J. Dunlosky, &
A.C. Graesser (Eds.), Metacognition in educational theory and practice (pp. 277–304).
Mahwah: Lawrence Erlbaum.
Zeger, S. L., & Liang, K.-Y. (1986). Longitudinal data analysis for discrete and continuous outcomes.
Biometrics, 42,121–130.
Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International
(CC BY 4.0).