Designing and implementing multimodal data collection classroom to capture metacognition in collaborative learning in

Abstract

While prominent empirical research exploring the possibilities to utilize different data channels in the research of regulation in collaborative learning is emerging, we are still in the process of discovering the relevant combinations of different data sources and proper ways to combine data from different channels. This is the case particularly with metacognition. The potential of using multiple data channels lies also in their power to be transferred as a tool for providing learners 'on the fly' support for regulation when needed. However, an advanced understanding of the regulated learning in collaborative learning contexts, and particularly on metacognitive processes is essential to harness the benefits of technology in supporting these processes in collaborative learning.

Keywords

• Metacognition
• collaborative learning
• multimodal data

INTRODUCTION

Learning processes are hard to predict or model, since learning is always situated, dependent on the learning context and the learner’s individual metacognition. Metacognitive knowledge involves learners' perceptions of a task. It draws to prior knowledge in terms of same types of tasks and procedures needed to perform those (Winne & Hadwin, 1998) . Another component of metacognition are metacognitive experiences. Metacognitive experiences constitute, for example learners’ perceptions of task difficulty. Unlike task understanding, which is thoughtful and cognitive, perception about task difficulty is reactive, and is also informative for Self-Regulation of Learning (SRL) (Winne & Hadwin, 1998) . Multimodal data (e.g., physiological measures, videos, and situated self-reports) can provide a new unobtrusive way to capture learners’ metacognition without interrupting learning process (Järvelä et al.,2019) . Currently, there is an accumulating evidence on how physiological measures can be used to track learning. Recent studies have shown that the level of students’ physiological arousal is related to learners’ metacognition (Hajcak, McDonald, & learners’ perceptions of tasks, self and learning situations (Flavell, 1979) . Yet, current research lack methods to capture the situated nature of task perceptions in the context of collaborative learning over time. In this paper, the focus is to (1) introduce collaborative learning model designed to study processes focusing on metacognition and promoting awareness of metacognition in a secondary school science classroom, (2) describe multimodal data collection procedure implemented in secondary school science classroom and (3) illustrate with two case examples how multimodal data has been used to capture learner’s metacognition. Participants of the study were (N = 94) upper elementary school students aged 13 to 14 (58 females, 36 males) enrolled in compulsory physics course consisting altogether five lessons. In each lesson, the students collaborated in the same groups of three to four students based on the collaborative learning model. Altogether, the students had four 90 min physics lessons, once in a week and the last lesson was a collaborative exam. In addition, after each lesson, the students filled in a multiple-choice knowledge test consisting of five questions related on topics they had just learned. 2

COLLABORATIVE LEARNING MODEL

The collaborative learning model designed for science class is based on a self-regulated learning framework that provides opportunities and awareness for self-initiated regulation among individual learners and collaborative groups (authors). It utilizes technology-based environment called Qridi® (https://kokoa.io/products/qridi), which was designed to structure collaboration. The collaborative learning model is built on the idea of a ‘flipped classroom.’ Recently, the flipped classroom concept has been gaining considerable attention due to its potential to facilitate the regulation of learning (Jovanovic et al., 2019) . The use of a flipped classroom in collaborative learning creates a learning setting in which students are provided opportunities to take responsibility for their own learning by familiarizing themselves with the content knowledge beforehand to prepare for collaborative learning. In the current study, the flipped classroom structure and the collaborative work were coordinated by using a Qridi® (Figure 1). However, the learning materials were not provided via Qridi®, but the students used their own regular physics books.

In the Qridi® environment, students were able to check, for example, the phase of the lesson. In our learning model, Qridi® was tailored to increase students’ awareness of the collaborative learning task phases in general and, specifically, supporting their awareness of the regulation of learning. For example, Qridi® involved a 6Q tool designed to promote students’ situation-specific metacognitive awareness related to task understanding and task difficulty before and after the collaborative learning. In practice, the 6Q tool consists of two 0–100 slider-scale questions where students estimate their task understanding (Schraw and Dennison, 1994) , perceived task difficulty (Efklides et al. 1998) .

Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). As the students study according to collaborative learning model, multiple data sources were collected. Prior to the study, the participating students responded to trait-type questionnaires such as Metacognitive Awareness Inventory (MAI) (Schraw and Dennison, 1994) that captured their individual metacognitive beliefs. During the seven-week multichannel data collection process, students’ collaborative work was followed by video recordings. Shimmer3 GSR+ sensors with 128hz sampling rate were used to measure learners’ electrodermal activity (EDA) indicating arousal. The sensors were automatically synchronized with each other in the dock station before the start of each session. Students fitted the devices at the beginning of each lesson and took if off at the end. In this way, continuous EDA data was obtained for each student during the entire lesson. As one of the multiple data sources, we used the 6Q tool implemented in Qridi® to collect students’ situationspecific interpretations of their metacognition in terms of task understanding and task difficulty related to each collaborative session before and after the collaborative work. Altogether the students had five physics lessons and the last one was collaborative exam. 2.2

Processing physiological data

First, files having contact issues were removed from the dataset. Second, Butterworth low pass filter with frequency 1 and order 5 was used to remove small movement artifacts from the signal. Third, Ledalab toolbox and through-to-peak analysis with minimum amplitude of 0.05μS was used for peak Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). detection (Benedek & Kaernbach, 2010) . Number of non-specific skin conductance responses per minute (NS.SCR/min) for the session was used as a marker of arousal (Boucsein, 2012) . 3

CASE EXAMPLES – WHAT ABOUT METACOGNITION?

The case examples provide insight on how to use multimodal data to investigate fluctuation of task difficulty and task understanding during collaborative learning. The first case example illustrates how individual learners’ metacognitive beliefs and situation specific perceptions of task difficulty and task understanding are related on learning outcomes in the context of collaborative learning. The second case example instead focuses on exploring how individual learners’ situation specific interpretations of task difficulty and task understanding are related in physiological arousal in the context of collaborative learning. 3.1

Analysis

In both of the case examples, Generalized Estimating Equations (GEE) was used. GEEs enable a general method for analyzing clustered variables and ease several assumptions of traditional regression models (Diggle, 2002; Liang & Zeger, 1998; Zeger & Liang, 1986) . The GEE method does not explicitly model between-cluster variation, rather it estimates its counterpart, the within-cluster similarity of the residuals, and then uses this estimated correlation to re-estimate the regression parameters and to calculate standard error. To estimate the validity of the GEE, QIC statistics proposed by Pan (2001) allow comparisons of GEE models and selection of a correlation structure. In both case examples, normal distributions with the log link function were selected because they yielded the lowest quasi-likelihood under the independence criterion (QIC) values

3.2 How metacognitive beliefs and situated task perceptions relate for learning outcomes?

With regard to first case example, generalized estimating equations (GEE) examine the effects of individual metacognitive beliefs (MAI) and task perceptions which are task understanding (TU) and task difficulty (TD) on upper elementary school students’ learning outcomes measured after each lesson.

Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).

Dependent variable

Knowledge Tests

Independent variables

Pre TU

Post TU

Pre TD

Post TD MAI

To summarize, learner individual metacognitive beliefs (which are quite static) do not predict learning outcomes, but rather learner’s situation specific interpretations of the task after the collaborative learning session predicts learning outcomes at individual level.

3.3 How individuals task perceptions relate for physiological arousal when collaborative learning context is not or is considered?

With regard to second case example, generalized estimating equations (GEE) was used to examine the effects of individual understanding (TU) and task difficulty (TD) on upper elementary school students’ physiological arousal (NS.SCRs in minute) during collaborative exam first at individual level (independent from the group) and second at collaborative level (exchangeable in the group). In the light of the second case example, the results show, that when the collaborative learning context is not considered, task perceptions dos not predict physiological arousal. The model fit statistics (QIC) scores was 10,1.

However, when the group is considerd as exhchanceable, the results show that learners interpretations before the task (pre-TU) score can effectively predict physiological arousal (NS.SRCs in minute) (Table 2). The model fit statistics (QIC) score was 8,943.

Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).

Dependent variable

NS.SCRs / minute

Independent variables

Pre TU

Post TU

Pre TD

Post TD

These two case examples shed a light in the process of discovering the relevant combinations of different data sources and proper ways to combine data to investigate metacognition. The first case example illustrates, that student characteristics, in terms of their metacognitive beliefs does not predict learning outcomes. However, the way students perceive the task after the learning situation predicts their learning outcomes.

The second example shows that when social context is taken account, task understanding predicts physiological arousal. In both examples, learner’s situation specific interpretations of a task were used as an indicator of metacognition. It can be concluded, that finding (relatively) unobtrusive ways to measure and detect variations in learners task understanding as the learning proceeds, provides a fruitful venue to explore ways to implement learning analytics and to provide learners feedback and support for regulation when needed. 4

THE WORKSHOP PRESENTATION

To conclude, this presentation focuses on workshop theme: Examples of CrossMMLA research designs and case examples by presenting 1) collaborative learning model designed to capture and promote awareness of metacognition, 2) multimodal data collection implemented in science classroom and 3) providing two representative case examples of multimodal data use to capture metacognition focusing on task perceptions of learners. In the workshop, the aim is to illustrate in detail how the collaborative learning model and multimodal data collection has been designed to capture metacognition in the light of theories of regulated learning.

Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).

References

Boucsein , W. ( 2012 ). Electrodermal activity (2nd ed .). New York, NY: Springer.

Diggle , P. J. , Heagerty , P. , Liang , K.-Y., & Zeger , S. L. ( 2002 ). Longitudinal data analysis (2nd ed .). Oxford, UK: Oxford University Press.

Benedek , M. , & Kaernbach , C. ( 2010 ). Decomposition of skin conductance data by means of nonnegative deconvolution . Psychophysiology , 47 ( 4 ), 647 - 658 . https://doi.org/10.1111/j.1469- 8986 . 2009 . 00972 .x

Efklides , A. , Papadaki , M. , Papantoniou , G. , & Kiosseoglou , G. ( 1998 ). Individual differences in feelings of difficulty: The case of school mathematics . European Journal of Psychology of Education , 13 ( 2 ), 207 - 226 .

Flavell , J. H. ( 1979 ). Metacognition and cognitive monitoring: A new area of cognitivedevelopmental inquiry . American psychologist , 34 ( 10 ), 906 .

Ghisletta , P. , & Spini , D. ( 2004 ). An introduction to generalized estimating equations and an application to assess selectivity effects in a longitudinal study on very old individuals . Journal of Educational and Behavioral Statistics , 29 ( 4 ), 421 - 437 .

Hajcak , G. , McDonald , N. , & Simons , R. F. ( 2003 ). To err is autonomic: Error-related brain potentials, ANS activity, and post-error compensatory behavior . Psychophysiology , 40 ( 6 ), 895 - 903 . https://doi.org/10.1111/ 1469 - 8986 . 00107

Jovanovic , J. , Mirriahi , N. , Gašević , D. , Dawson , S. , and Pardo , A. ( 2019 ). Predictive power of regularity of pre-class activities in a flipped classroom . Computers & Education , 134 , 156 - 168 .

Järvelä , S. , Malmberg , J. , Haataja , E. , Sobocinski , M. , & Kirschner , P. A. ( 2019 ). What multimodal data can tell us about the students' regulation of their learning process . Learning and Instruction , https://doi. org/10.1016/j.

Liang , K.-Y., & Zeger , S. L. ( 1986 ). Longitudinal data analysis using Generalized LinearModels . Biometrika, 73 , 13 - 22 .

Pijeira-Díaz , H. J. , Drachsler , H. , Kirschner , P. A. , & Järvelä , S. ( 2018 ). Profiling sympathetic arousal in a physics course: How active are students? . Journal of Computer Assisted Learning , 34 ( 4 ), 397 - 408 .

Schraw , G. , & Dennison , R. S. ( 1994 ). Assessing metacognitive awareness . Contemporary EducationalPpsychology , 19 ( 4 ), 460 - 475 .

Winne , P.H. , & Hadwin , A.F. ( 1998 ). Studying as self-regulated learning . In D.J. Hacker , J. Dunlosky , & A.C. Graesser (Eds.), Metacognition in educational theory and practice (pp. 277 - 304 ). Mahwah: Lawrence Erlbaum.

Zeger , S. L. , & Liang , K.-Y. ( 1986 ). Longitudinal data analysis for discrete and continuous outcomes . Biometrics , 42 , 121 - 130 .