Learning processes are hard to predict or model, since learning is always situated, dependent on the learning context and the learner's individual metacognition. Metacognitive knowledge involves learners' perceptions of a task. It draws to prior knowledge in terms of same types of tasks and procedures needed to perform those (Winne & Hadwin, 1998). Another component of metacognition are metacognitive experiences. Metacognitive experiences constitute, for example learners' perceptions of task difficulty. Unlike task understanding, which is thoughtful and cognitive, perception about task difficulty is reactive, and is also informative for Self-Regulation of Learning (SRL) (Winne & Hadwin, 1998). Multimodal data (e.g., physiological measures, videos, and situated self-reports) can provide a new unobtrusive way to capture learners' metacognition without interrupting learning process (Järvelä et al.,2019). Currently, there is an accumulating evidence on how physiological measures can be used to track learning. Recent studies have shown that the level of students' physiological arousal is related to learners' metacognition (Hajcak, McDonald, & Simons, 2003) and achievement (Pijeira-Diaz et al. 2018). Metacognition, in turn, is related to learners' perceptions of tasks, self and learning situations (Flavell, 1979). Yet, current research lack methods to capture the situated nature of task perceptions in the context of collaborative learning over time.
In this paper, the focus is to (1) introduce collaborative learning model designed to study processes focusing on metacognition and promoting awareness of metacognition in a secondary school science classroom, (2) describe multimodal data collection procedure implemented in secondary school science classroom and (3) illustrate with two case examples how multimodal data has been used to capture learner's metacognition. Participants of the study were (N = 94) upper elementary school students aged 13 to 14 (58 females, 36 males) enrolled in compulsory physics course consisting altogether five lessons. In each lesson, the students collaborated in the same groups of three to four students based on the collaborative learning model. Altogether, the students had four 90 min physics lessons, once in a week and the last lesson was a collaborative exam. In addition, after each lesson, the students filled in a multiple-choice knowledge test consisting of five questions related on topics they had just learned.
The collaborative learning model designed for science class is based on a self-regulated learning framework that provides opportunities and awareness for self-initiated regulation among individual learners and collaborative groups (authors). It utilizes technology-based environment called Qridi® (https://kokoa.io/products/qridi), which was designed to structure collaboration. The collaborative learning model is built on the idea of a 'flipped classroom.' Recently, the flipped classroom concept has been gaining considerable attention due to its potential to facilitate the regulation of learning (Jovanovic et al., 2019). The use of a flipped classroom in collaborative learning creates a learning setting in which students are provided opportunities to take responsibility for their own learning by familiarizing themselves with the content knowledge beforehand to prepare for collaborative learning. In the current study, the flipped classroom structure and the collaborative work were coordinated by using a Qridi® (Figure 1). However, the learning materials were not provided via Qridi®, but the students used their own regular physics books.
In the Qridi® environment, students were able to check, for example, the phase of the lesson. In our learning model, Qridi® was tailored to increase students' awareness of the collaborative learning task phases in general and, specifically, supporting their awareness of the regulation of learning. For example, Qridi® involved a 6Q tool designed to promote students' situation-specific metacognitive awareness related to task understanding and task difficulty before and after the collaborative learning. In practice, the 6Q tool consists of two 0-100 slider-scale questions where students estimate their task understanding (Schraw and Dennison, 1994), perceived task difficulty (Efklides et al. 1998).
As the students study according to collaborative learning model, multiple data sources were collected. Prior to the study, the participating students responded to trait-type questionnaires such as Metacognitive Awareness Inventory (MAI) (Schraw and Dennison, 1994) that captured their individual metacognitive beliefs. During the seven-week multichannel data collection process, students' collaborative work was followed by video recordings. Shimmer3 GSR+ sensors with 128hz sampling rate were used to measure learners' electrodermal activity (EDA) indicating arousal. The sensors were automatically synchronized with each other in the dock station before the start of each session. Students fitted the devices at the beginning of each lesson and took if off at the end. In this way, continuous EDA data was obtained for each student during the entire lesson. As one of the multiple data sources, we used the 6Q tool implemented in Qridi® to collect students' situationspecific interpretations of their metacognition in terms of task understanding and task difficulty related to each collaborative session before and after the collaborative work. Altogether the students had five physics lessons and the last one was collaborative exam.
First, files having contact issues were removed from the dataset. Second, Butterworth low pass filter with frequency 1 and order 5 was used to remove small movement artifacts from the signal. Third, Ledalab toolbox and through-to-peak analysis with minimum amplitude of 0.05μS was used for peak detection (Benedek & Kaernbach, 2010). Number of non-specific skin conductance responses per minute (NS.SCR/min) for the session was used as a marker of arousal (Boucsein, 2012).
The case examples provide insight on how to use multimodal data to investigate fluctuation of task difficulty and task understanding during collaborative learning. The first case example illustrates how individual learners' metacognitive beliefs and situation specific perceptions of task difficulty and task understanding are related on learning outcomes in the context of collaborative learning.
The second case example instead focuses on exploring how individual learners' situation specific interpretations of task difficulty and task understanding are related in physiological arousal in the context of collaborative learning.
In both of the case examples, Generalized Estimating Equations (GEE) was used. GEEs enable a general method for analyzing clustered variables and ease several assumptions of traditional regression models (Diggle, 2002;Liang & Zeger, 1998;Zeger & Liang, 1986). The GEE method does not explicitly model between-cluster variation, rather it estimates its counterpart, the within-cluster similarity of the residuals, and then uses this estimated correlation to re-estimate the regression parameters and to calculate standard error. To estimate the validity of the GEE, QIC statistics proposed by Pan ( 2001) allow comparisons of GEE models and selection of a correlation structure. In both case examples, normal distributions with the log link function were selected because they yielded the lowest quasi-likelihood under the independence criterion (QIC) values
With regard to first case example, generalized estimating equations (GEE) examine the effects of individual metacognitive beliefs (MAI) and task perceptions which are task understanding (TU) and task difficulty (TD) on upper elementary school students' learning outcomes measured after each lesson.
Table 1 shows that only learners' interpretations on post-task understanding (Post TU) score can effectively predict different actualized knowledge tests that were measured after each lesson. The model fit statistics (QIC) scores was 258,986.
Table 1. GEE results model using a normal distribution with a log link function predicting students' learning outcomes
Pre TU 0,001 (-0,001;0,003) 0,431
Post TU 0,002 (0;0,004) 0,038
Pre TD -0,0000576 (-0,002;0,002) 0,949
Post TD -0,001(-0,003: 0) 0,116 MAI 0,002 (-7,39E-05:0,004) 0,059
To summarize, learner individual metacognitive beliefs (which are quite static) do not predict learning outcomes, but rather learner's situation specific interpretations of the task after the collaborative learning session predicts learning outcomes at individual level.
With regard to second case example, generalized estimating equations (GEE) was used to examine the effects of individual understanding (TU) and task difficulty (TD) on upper elementary school students' physiological arousal (NS.SCRs in minute) during collaborative exam first at individual level (independent from the group) and second at collaborative level (exchangeable in the group).
In the light of the second case example, the results show, that when the collaborative learning context is not considered, task perceptions dos not predict physiological arousal. The model fit statistics (QIC) scores was 10,1. However, when the group is considerd as exhchanceable, the results show that learners interpretations before the task (pre-TU) score can effectively predict physiological arousal (NS.SRCs in minute) (Table 2). The model fit statistics (QIC) score was 8,943. Pre TU 0,006 (0,001;0,10) 0,012
Post TU -0,004 (-0,010;0,002) 0,190
Pre TD -0,007 (-0,016;0,001) 0,086
Post TD -0,001(-0,10;0,008) 0,746
These two case examples shed a in the process of discovering the relevant combinations of different data sources and proper ways to combine data to investigate metacognition. The first case example illustrates, that student characteristics, in terms of their metacognitive beliefs does not predict learning outcomes. However, the way students perceive the task after the learning situation predicts their learning outcomes.
The second example shows that when social context is taken account, task understanding predicts physiological arousal. In both examples, learner's situation specific interpretations of a task were used as an indicator of metacognition. It can be concluded, that finding (relatively) unobtrusive ways to measure and detect variations in learners task understanding as the learning proceeds, provides a fruitful venue to explore ways to implement learning analytics and to provide learners feedback and support for regulation when needed.
To conclude, this presentation focuses on workshop theme: Examples of CrossMMLA research designs and case examples by presenting 1) collaborative learning model designed to capture and promote awareness of metacognition, 2) multimodal data collection implemented in science classroom and 3) providing two representative case examples of multimodal data use to capture metacognition focusing on task perceptions of learners. In the workshop, the aim is to illustrate in detail how the collaborative learning model and multimodal data collection has been designed to capture metacognition in the light of theories of regulated learning.