=Paper=
{{Paper
|id=Vol-2902/paper3
|storemode=property
|title=Different Modality, Different Design, Different Results: Exploring Self-regulated Learner Clusters’ Engagement Behaviours at Individual, Group and Cohort Activities
|pdfUrl=https://ceur-ws.org/Vol-2902/paper3.pdf
|volume=Vol-2902
|authors=Qi Zhou,Wannapon Suraworachet,Mutlu Cukurova
|dblpUrl=https://dblp.org/rec/conf/aied/ZhouSC21
}}
==Different Modality, Different Design, Different Results: Exploring Self-regulated Learner Clusters’ Engagement Behaviours at Individual, Group and Cohort Activities==
https://ceur-ws.org/Vol-2902/paper3.pdf
Different modality, different design, different results:
Exploring self-regulated learner clusters’ engagement
behaviours at individual, group and cohort activities
Qi Zhou1, Wannapon Suraworachet1, Mutlu Cukurova1
1 University College London, London, UK
qtnvqz3@ucl.ac.uk
Abstract. Self-Regulated Learning (SRL) competence is an important aspect of
online learning. SRL is an internal process, but analytics can offer an externali-
sation trigger to allow for observable effects on learner behaviours. The purpose
of this paper is to explore the relationship between students’ SRL competence
and their learning engagement behaviours observed in multimodal data. In a post-
graduate course with 42 students, eighteen features from three types of data in
seven learning activities were extracted to investigate multi-level SRL compe-
tence students’ engagement behaviours. The results revealed that students with
different SRL competence clusters might exhibit different behaviours in individ-
ual, group, and cohort level learning activities. Also, students with similar SRL
competence might exhibit significantly different engagement behaviours in dif-
ferent learning activities, depending on the learning design. Therefore, while us-
ing engagement data in AIED systems; the modality of the data, specific analysis
techniques used to process it, and the contextual particularities of the learning
design should all be explicitly presented. So that, they can be considered in the
interpretations of automated decisions about student achievement.
Keywords: Self-regulated learning, multimodality, social planes, learning de-
sign
1 Introduction and Background
Online learning is considered to have the advantages of breaking the barrier of time and
space, which may provide more flexibility and accessibility for students [1, 2]. Even
though online learning has great potential, its effectiveness depends on student active
engagement as well as how they plan, control, and reflect on their learning activities in
these settings [3, 4]. This process has been broadly referred to as self-regulated learning
(SRL) [5]. Self-regulated learning has been defined as the process of modulating emo-
tion, cognition and behaviours to accomplish desired goals which could lead to a higher
level of academic achievement [6-9]. However, as SRL is an internal and hidden pro-
cess, representative measures such as behavioural proxies or self-report data are re-
quired for probing SRL [10].
Self-report questionnaires are extensively used to assess students’ SRL characteris-
tics due to their advantage in capturing learners’ perceptions. However, it is questiona-
ble whether this subjective measure can comprehensively represent SRL. Rather than
viewing SRL as an aptitude, many researchers support that SRL is a dynamic process
of individual learners responding to the constraints imposed by the learning environ-
ment in accordance with their personal goals and the task at hand [11-13]. Although
alternative measures such as think-aloud and event-based interviews are more respon-
sive to the situation, their time-demanding nature and imposition of cognitive load dur-
ing the task engagement for learners, leave self-reports less practical and unsatisfactory
Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
2
instruments to assess SRL [14]. With the rise in computer-based learning environments,
digital traces—how students selectively interact with the available digital resources, are
other available measures to observe SRL, especially in terms of how and when students
deploy an SRL strategy within the environment.
Research has shown that self-report measures and online traces are equitable to cap-
ture SRL from different perspectives [14-16]. Hadwin et al. [15] found that while stu-
dents did not report their actual behaviours, trace data can be applied to study the stu-
dent's actual enactment of SRL strategies. Similarly, Jamieson-Noel and Winne [16]
also concluded that students used different criteria when reporting SRL compared to
their deployed tactics that emerged through log data. van Halem et al. [14] suggested
that self-report is better at capturing student’s prior performance whereas trace data is
better at explaining variance in student’s subsequent performance. Therefore, self-re-
ports can be used as a complement to trace data. The method that integrates multiple
data sources to analyze learners’ interactions and examine complex learning processes
is known as Multimodal Learning Analytics (MMLA) [17]. MMLA aims to generate
tailoring and timely support for learners and aid teachers’ decision-making in the learn-
ing contexts [18]. Previous research has reinforced the usual higher predictive accuracy
of models generated from multimodal compared to unimodal data [19-21]
SRL research with MMLA has been introduced in many contexts such as individual
distance learning [22-24] and collaborative learning [25-28]. For example, perceived
individual versus group challenges for regulation was studied in collaborative contexts
by Järvelä et al. [29] whereas emotional regulation of students in an agent-based indi-
vidual learning platform was investigated by Azevedo et al. [30]. However, most avail-
able studies focus on a single learning context and generate results based on learners’
interaction data from a single digital environment or a learning activity. Learning de-
sign refers to a set of learning activities or learning interactions, including resources
and materials prepared for students to achieve learning goals [31]. Learner behaviours
are likely to vary depending on the learning design, domain, method of assessment,
expected learning outcomes as well as the social plane in which students operate (e.g.,
individual, group, and cohort levels). Although the impact of learning design on learner
behaviours is well-established in the learning sciences literature, cross-context and
cross-modality differences of students’ SRL behaviours in digital environments are un-
derexplored. Here, we investigate the differences in student’s engagement behaviours
based on their SRL competence clustering when they are engaging in learning activities
alone, while they are in small groups, and while they are interacting at the cohort level.
More specifically, we investigate two research questions:
1) To what extent do students with different SRL competence exhibit different en-
gagement behaviours in learning activities?
2) To what extent do students with the same SRL competence engage differently in
learning activities from different social planes (individual, group, and cohort)?
This study aims to provide insights into how SRL behaviours might potentially differ
depending on the learning activities and social planes in which these activities are im-
plemented. Through the identification of such potential differences, our ultimate goal
is to support future designs of AI and analytics solutions that may provide appropriate
support in different learning activities and at different social planes.
3
2 Context of Study
2.1 The Educational Context
Forty-two postgraduate students from a UK university enrolled and completed the 10-
week course in Design and Use Technology for Education. All of them have consented
to participate in the study which has obtained ethical approval from the institution. Dur-
ing the course, students participated in learning activities that require them to interact
at different social planes: individual level, group level and cohort level. At the individ-
ual level, each week students were asked to (1) finish the readings available on the
online platform, (2) watch the video lectures about the weekly topic, and (3) write in-
dividual reflections on what they have learnt. At the cohort level, students (4) partici-
pated in an online asynchronous debate on a debate platform and (5) voluntarily at-
tended a live Q&A session for clarification. Besides, a forum was also available for
asynchronous discussion. For the group level, students were divided into ten groups
(with 3-5 members) who had interdisciplinary backgrounds, mixed genders, and were
from similar timezones for conveniently scheduling synchronous group meetings. They
had to select an educational challenge and carry out an educational technology design
case to solve the challenge they identified. Weekly, the group (6) had an online syn-
chronous group meeting to discuss and (7) completed the group tasks in a co-design
environment. During the meeting, the speaker’s timestamp was recorded to automati-
cally generate students’ speech time pie charts and their turn-taking networks [32].
These graphs were manually combined with narrative feedback from two teaching as-
sistants and sent to students via weekly emails.
The assignment consisted of two parts: Part A focused on students’ individual re-
flections (40%) and Part B was a 2500-word essay (60%) on critical analysis of the key
issues on the design case. The feedbacks were provided as (1) Part A’s formative feed-
back at mid-term (week 6), (2) feedback on the draft of part B before the final submis-
sion, and (3) summative feedback at the end of the module (week 15). The final sub-
missions were marked by three reviewers. Double marking was applied to 35% of the
final submissions, achieving 96% of inter-rater reliability in the final grades given.
2.2 Measuring students’ SRL competence
Before the course started, all students have completed the questionnaire which used to
evaluate the students’ SRL competence. The questionnaire was adapted from a meta-
review in the SRL field concerning multiple SRL dimensions in relation to academic
achievement including metacognition (metacognitive strategies, goal-setting, self-mon-
itoring and self-control), attention, learning strategies, time management, environmen-
tal structuring, help-seeking, motivation, emotion control, and attributions [9]. The
adapted version can be found here. Cronbach’s alpha (0.771) was used to test the reli-
ability. In this study, students were divided into high (n=13, Mean = 4.18, SD = 0.14),
medium (n=14, Mean = 3.80, SD = 0.15) and low (n=15, Mean = 3.37, SD = 0.14) SRL
competence clusters according to the mean score of all dimensions of the questionnaire.
3 Methodology
3.1 Data Collection and Preprocess
In total, three types of data were collected from seven resources: 1) timestamp data was
collected from Zoom meetings to investigate students’ speech behaviours within the
group contexts; 2) log data was collected from Moodle, Ligilo, Echo360, Miro and
4
Google Docs; and 3) qualitative manual coding data was recorded from live Q&A ses-
sions to investigate students’ engagement in the sessions. Since the submission deadline
for individual reflection was five weeks after the course ended, there were 15-week log
data from Google Docs. For other sources, there were a total of 11-week log data which
included the reading week, a mid-term break at week 6. All data has been anonymized
according to the requirements of the institutional ethical approval before its analysis.
Timestamp Data. During the ten-week collaboration, groups used Zoom
(https://zoom.us) as an online synchronous meeting platform to meet, plan and com-
plete their design tasks. In each meeting which lasted around 1-2 hours, a system was
used to collect timestamp data from Zoom. Once a user’s microphone has been detected
as active, the system would capture the timestamp and the user’s ID. If none of the
user’s microphone has been detected as active, the system would also capture the
timestamp and mark the user's ID as 0. For each meeting, the system also records all
users’ IDs and names, as well as meetings’ IDs and titles. All this data has been inte-
grated into one JSON file and stored in a cloud database. In total, 130000 lines of
timestamp data were successfully collected from eighty-five remote group meetings
(lost data due to technical issues). The timestamps were 13-digit and accurate to milli-
seconds.
During data preprocessing, initially, the data was extracted from the JSON files and
restructured into a CSV file. For each line in the CSV file, it presented the week num-
ber, the group number, speakers’ student ID, timestamp of speech beginning, timestamp
of speech end, and duration of the speech. The meetings’ IDs and titles were used to
identify the week number and group number. The speakers’ usernames were used to
identify who the speaker was. Since the system only records the timestamps of speech
beginning, the timestamp of a new speaker appearing was taken as the end of the last
speech. The duration of one speech was calculated by the difference between the
timestamp of a speech beginning and the timestamp of a speech ending.
Log Data. In this study, log data was collected from five online platforms/tools:
Moodle, Ligilo, Echo360, Miro and Google Docs. The log data from Moodle
(https://moodle.org) consisted mostly of the number of page viewing, posts creating,
and task completion. Ligilo (https://www.go-ligilo.com) hosted the online asynchro-
nous debate activity in the course. It is an online discussion platform designed specifi-
cally for peer learning which enables students to connect their posts with others’
through a specified relationship. The data from Ligilo recorded the number of post-
viewing and post creating. Echo360 (https://echo360.com) is a video platform on which
students can view all pre-record lectures in the course. It recorded how many lectures
each student viewed and the duration of each lecture viewing. Miro (https://miro.com)
is an online visual collaboration platform that supports, for example, brainstorming,
mind mapping and prototyping. Every week, each group was asked to finish pre-set
tasks related to the weekly topic on Miro. The platform recorded actions with timestamp
made by students. Google Docs (https://docs.google.com) was used as a tool for stu-
dents to write individual reflections. An open-source Chrome extension, Draftback
(http://draftback.com), was used to extract log data from the edit history of google docs.
The data provided information about 1) what types of change (inserting or deleting) has
been made; 2) the start and end index of the document in which the change has been
made; 3) when the change has been made; 4) the unique identifier of the change called
‘revision number’; and 5) what actual contents has been inserted or deleted. During the
5
data preprocessing, all log data was restructured into CSV files. Activity data from
teachers, course administrators, and dropped students were removed. Then, duplicated
data has been identified by the timestamps: if the same action has been made more than
one time in a very short period, a single record was kept as the actual action.
Qualitative Observation Data. To investigate students’ engagement in Q&A ses-
sions, manual coding was applied to record: 1) students’ attendance, and 2) whether
students have made relevant contributions to the discussion in the Q&A sessions. For
instance, the researchers recorded the list of participants in one Q&A session three
times: at the beginning, middle and the end of the session to be able to count late at-
tending and early leaving students. Students have been marked as “attended” only if
they were in at least two of the participant lists. Meanwhile, during the sessions, stu-
dents would be marked as “made contributions” if they have asked questions, answered
questions, or shared their opinions on the content of the course.
3.2 Behavioural Features
Behavioural features were generated from the different modalities of data described
above to evaluate students’ engagement in different learning activities across the whole
course. Table 1 shows the extracted behavioural features and their description.
Table 1. Extracted behavioural features and description according to the task levels and tools.
Task Tool Extracted Feature Description
level
Cohort Ligilo Ligilo_View The total number of times that a student viewed a post
level on Ligilo
Ligilo_Post The total number of posts that a student created on
Ligilo
Qualitative Q&A_Attendence The total number of sessions that a student has attended
Observation
data Q&A_Speech The total number of sessions that a student made a
speech.
Moodle Moodle_Discussion The total number of times that a student viewed a post
View on Moodle
Moodle_Discussion The total number of posts that a student created on Moo-
Create dle
Group Zoom Zoom_SpeechTime The average time of one student’s speech time in a 60-
level seconds window during the online synchronous meet-
ings across 10 weeks
Zoom_SpeechFre- The average number of times one student made a speech
quency in a 60-seconds window during the online synchronous
meetings across 10 weeks
Miro Miro_Engagement A student’s total number of actions on the Miro plat-
form
Individ- Echo360 Lecture_View A student’s total number of viewed lectures
ual Level
Lecture_Review A student's total number of times they reviewed lectures
on echo360
Google Reflection_TotalRevi- The total number of revisions that a student made in the
Docs sion reflection document
Reflection_Final- The total string count of a student’s reflection docu-
StringCount ment.
Reflection_Avg- The average string count of the revisions that a student
StrCountPerDay made per day in the reflection document
6
3.3 Comparison tests
To explore the relationship between students’ SRL competence levels and their engage-
ment in different learning activities, significance tests of difference were used. Stu-
dents’ general SRL competence levels were examined as independent factors, while the
eighteen features described above were considered as dependent factors. Tests for ho-
mogeneity of variance were conducted for each feature to determine the normality of
data distributions and the appropriate tests were administered accordingly. If the p-
value is less than 0.05, it means a violation of the assumption, Kruskal-Wallis test for
nonparametric comparisons was applied. Otherwise, One-way ANOVA was used to
explore whether there are differences between different SRL competence groups. Bon-
ferroni multiple comparison corrections were conducted in post-hoc.
3.4 Time Series Analysis
Due to the time-dependent nature of the data being collected, also a time-series analysis
was applied. Time-series analysis is useful to explore the timely behaviours and extract
time series components to better understand and model future behaviours [33]. This
analysis is based on the assumption that data points in the previous time are good pre-
dictors of the later data points [33]. The main time-series components we used within
the focus of this study was the trend. The trend represents the long-term direction of the
dataset which can be an increasing/upward trend or a decreasing/downwards trend. In
the module, most of the tasks e.g., cohort debates and group discussion were advised to
be completed on a specific day of the week except the individual reflection in which
students were free to complete on any day of the week until the optional formative
feedback was given at mid-term. Therefore, students’ weekly behaviours on individual
reflection tasks were investigated using time-series analysis.
4 Results
4.1 Comparison Tests on Students’ Engagement
Cohort Level. The first columns of Table.2 show the mean value and standard devia-
tion for each behavioural feature at the cohort level. An ANOVA on the value for the
Moodle_DiscussionView produced a significant difference, F(2,39) = 5.237, p < 0.05,
partial η2 = 0.212. Follow up post-hoc tests (with Bonferroni correction) showed that
for the Moodle_DiscussionView value, the high SRL group was significantly higher
than the medium SRL group, p = 0.013. It means that high SRL competence students
have viewed the discussions on Moodle more frequently than medium SRL competence
students. Furthermore, there is no significant difference in Moodle_DiscussionCreate,
Ligilo_View, Ligilo_Post, Q&A_Attendance, and Q&A_Speech. It means that students
with different SRL competence levels did not exhibit different engagements in online
asynchronous debate activities and live Q&A sessions.
Table 2. Comparison tests of cohort-level behavioural features
High SRL Medium SRL Low SRL ANOVA
M SD M SD M SD Sig.
Ligilo_View 504.92 288.22 359.57 304.59 389.80 167.19 0.32
Ligilo_Post 13.54 12.93 10.86 10.48 13.8 9.61 0.74
Q&A_Attendance 6.38 2.96 7.00 2.83 6.00 3.53 0.69
Q&A_Speech 2.54 2.93 3.14 3.86 2.80 3.49 0.90
7
Moodle_Discus-
sionView 67.23 40.74 31.64 21.17 59.87 27.34 0.01
Moodle_Discussion-
Create 5.00 5.13 1.86 3.55 5.07 6.51 0.19
Group Level. The first columns of Table.3 show the mean value and standard deviation
for each behavioural feature at the group level. An ANOVA on the value for the
Zoom_SpeechFrequency produced a significant difference, F(2,39) = 3.452, p < 0.05,
partial η2 = 0.15. Follow up post-hoc tests (with Bonferroni correction) showed that for
the Zoom_SpeechFrequency value, the high SRL group was significantly higher than
the low SRL group, p = 0.046. It means that students with low SRL competence exhib-
ited a higher frequency of speech in online group meetings than students with high SRL
competence. Apart from this, at students’ group-level interactions, no statistical differ-
ences were observed in Zoom_SpeechTime and Miro_Engagement behaviours.
Table 3. Comparison tests of group-level behavioural features
High SRL Medium SRL Low SRL ANOV
A Sig.
M SD M SD M SD
Zoom_SpeechTime 39.80 17.14 52.85 20.00 54.20 19.40 0.11
Zoom_SpeechFre-
quency 12.30 5.11 16.15 5.83 17.27 4.60 0.04
Miro_Engagement 142.31 107.07 233.93 233.11 297.2 217.24 0.33
Individual Level. At the individual level, we investigated two learning activities: lec-
ture viewing and individual reflective writing. The first columns of Table.4 show the
mean value and standard deviation for each behavioural feature at this level. The result
of ANOVA analysis in the last column shows that there is no statistical difference found
in these learning activities. Students with different SRL competence levels did not ex-
hibit statistically significant differences in engagement behaviours at this level. Given
the time-dependent nature of the reflection task, time series analysis was conducted to
investigate time-series components.
Table 4. Comparison tests of individual-level behavioural features
Medium SRL
ANOV
High SRL Group Group Low SRL Group
A Sig.
M SD M SD M SD
Lecture_View 9.23 3.11 10.50 1.23 9.47 2.42 0.717
Lecture_Review 10.69 11.37 10.79 12.96 5.40 3.78 0.268
Reflection_TotalRevi- 15317. 15996. 16739. 23707. 10388. 8838.0
sion 55 06 79 71 27 9 0.588
Reflection_Final- 32367. 10231. 32418. 9173.6 28694. 10702.
StringCount 09 82 93 9 13 91 0.539
Reflection_Avg- 11191. 6478.0 7085.4 3600.3 7771.8 4322.9
StrCountPerDay 38 1 5 7 8 3 0.094
8
4.2 Time series analysis of reflection behaviours
Fig. 1. The average number of edited strings per day between students with high (blue line),
medium (orange line) and low (green line) SRL competence across 15 weeks
The average number of edited strings per day was selected as a proxy to represent stu-
dent reflection behaviours. Figure.1 shows the average number of edited strings per day
of students with high, medium, and low SRL competency across 15 weeks. Despite
similar trends of the three groups productively reflected more towards the end of the
course (submission date on week 15), the graph visually suggested that students with
high SRL competence and students with medium SRL competence tend to be more
active in reflecting their learning. For instance, in figure 1, these two groups of students
are observed with more peaks and higher peaks than the students with low SRL com-
petence. These differences are significant in the week after students received their mid-
term feedback and the week before the submission deadline. A more detailed discussion
of time-series analysis could be found in [34].
5 Discussion
In this study, eighteen features were extracted from three different types of data in seven
learning activities to investigate different SRL competence students’ engagement be-
haviours at cohort, group, and individual levels. The results illustrated significant dif-
ferences in students’ behaviours at different social planes.
The first research question explored whether students with different SRL compe-
tence levels exhibit different engagement behaviours in the learning activities. At the
cohort level, a significant difference has been observed in students’ engagement with
the discussion activities on Moodle. The students with high SRL competence tended to
view more posts in the discussion forums than the students with low SRL competence.
Further investigations into the details of the log data from Moodle revealed that these
engagement behaviours mainly focused on sharing individual reflections and asking
questions to teachers. This particularly links to the recent findings from Jivet et al.[35]
that only the high SRL skills, specifically high help-seeking skills will value the dis-
cussion forum as a sort of help. It was also observed that students with high SRL com-
petence have a higher frequency of viewing others’ reflection input as well as viewing
discussions between students and teachers. In terms of online asynchronous debate and
live Q&A sessions, students with different SRL competence did not appear to exhibit
9
different behaviours. At the group level, students with different SRL competence did
not exhibit significantly different behaviours in their online design platform activities
(Miro). Similarly, there was no difference in speech time during the online group meet-
ings between students with different SRL competence. However, high SRL students
had a lower frequency of speaking during the group meetings. It means that the speech
made by high SRL students may be longer than the speech made by low SRL students.
One potential interpretation of this result might be that low SRL competence students
tend to ask more questions with short sentences while high SRL competence students
used long sentences to share and explain their opinions.
In terms of the learning activities at the individual level, viewing lectures and writing
individual reflections, there was no statistical difference found between different SRL
groups. This indicates that they might have similar levels of engagement in these two
activities during the course at the accumulated data over the course time. However, the
time series analysis showed that students with different SRL competence seemed to use
different behaviours in individual reflective writings. For instance, students with high
and medium levels of SRL competence tended to edit their individual reflections im-
mediately after they received the mid-term feedback. But reactions to feedback by low
SRL students were given two weeks later, if at all. Furthermore, students with high and
medium SRL competence levels were more active than the students with low SRL com-
petence in the week before the submission deadline.
Our second research question investigated whether students with the same SRL com-
petence exhibit the same engagement behaviours in learning activities from different
social planes (individual, group, and cohort). The results show that high SRL students
were more active in learning at the cohort level activities and individual levels but less
active in group level activities. Although high SRL students did not create a signifi-
cantly larger number of posts on Moodle or Ligilo, they viewed a significantly larger
number of posts on Moodle. Similarly, at the individual level, the high SRL students
exhibited a higher number of engagement behaviours than the low SRL students. Time
series analysis illustrated that students with high SRL were more active in some specific
weeks even though no significant difference was found at the accumulated data analy-
sis. However, compared with individual-level activities and cohort-level activities, stu-
dents with high SRL competence exhibited different engagement behaviours at group-
level activities. As mentioned above, students with high SRL competence tended to
have a lower frequency of speaking in remote group meetings which contradicted the
results from Cho, Kim and Choi [36] that high SRL competence tended to perceive a
greater sense of community of inquiry (CoI), i.e., beliefs in knowledge acquisition
through social inquiry within a community. In other words, it can be inferred that high
regulators are expected to participate more in the group meeting. However, in their
study, the authors only considered students’ self-perception of CoI without observing
their actual behaviours unlike in our study. Moreover, these results also highlight the
importance of considering contextual specificity in studying students’ SRL behaviours
from trace data. In different learning activities and different social planes, high compe-
tence SRL students might exhibit different engagement behaviours.
Looking at the learning activities we investigated in this study, the strategic im-
portance of assessment also becomes clear. High SRL students tended to be more active
in viewing Moodle discussions and individual reflective writing activities. Students of-
ten used the Moodle forums to share their individual reflections or ask questions about
10
final submissions. It is important to note that both of these learning activities are
strongly tied to the course's summative assessment and student grades. On the other
hand, in Ligilo discussions, which were strongly encouraged but were not part of the
summative evaluation, different SRL groups did not exhibit different engagement be-
haviours. Similarly, high SRL competence students appeared to be less active in remote
group meetings for which feedback was provided to promote equality in the meeting
contributions, but there was no summative evaluation. These results may indicate that
high SRL competence students tend to engage more in the activities that are strategic
for their performance measures (i.e., are related to the final assessment). The relation-
ship between assessment and self-regulation has long been affirmed in many studies
e.g., an entanglement between teachers’ assessment criteria and students’ learning goals
and the effects of formative/summative assessment on student’s monitoring of progress
[37].
6 Conclusion
This study has some implications for the future design and implementation of multi-
modal AIED tools for promoting students SRL. First, it shows that data from different
channels and modalities may bring in different interpretations of student engagement
behaviours and their connection to students’ SRL competence. Furthermore, it under-
lines the value of bringing in temporal data analysis approaches compared to accumu-
lative ones for investigating dynamic student behaviours. To be more specific, we
showed that a time series analysis can reveal potential differences in students’ individ-
ual reflective writing behaviours, which were not found in the analysis of students’
accumulated data. Third, perhaps, more importantly, the same group of students with
similar SRL competence might exhibit significantly different engagement behaviours
in different learning activities, and different social planes, depending on the learning
design requirements such as the assessment and expected learning outcomes. If we are
to design AI systems that automatically detect, monitor, and support students with dif-
ferent levels of SRL competence, the modality of data that is fed into the system, spe-
cific data analysis techniques used, as well as the contextual particularities of the learn-
ing design should also be taken into account since they can have significant implica-
tions on the decisions made by the systems and their interpretations by humans.
References:
1. Waschull, S.B.: The online delivery of psychology courses: attrition, performance, and eval-
uation. Teaching of Psychology. 28(2), 143–147 (2001).
https://doi.org/10.1207/S15328023TOP2802_15.
2. Ku, D.T., Chang, C.-S.: The effect of academic discipline and gender difference on taiwan-
ese college students’ learning styles and strategies in web-based learning environments. The
Turkish Online Journal of Educational Technology. 10(3), 265-272 (2011).
3. Wang, C.-H., Shannon, D.M., Ross, M.E.: Students’ characteristics, self-regulated learning,
technology self-efficacy, and course outcomes in online learning. Distance Education. 34(3),
302–323 (2013). https://doi.org/10.1080/01587919.2013.835779.
4. Ally, M.: Foundations of educational theory for online learning. In: Anderson, T., Elloumi,
F. (eds.) Theory and Practice of Online Learning. Athabasca University, Athabasca (2004).
11
5. Zimmerman, B.J.: Investigating self-regulation and motivation: historical background,
methodological developments, and future prospects. American Educational Research Jour-
nal. 45(1), 166–183 (2008). https://doi.org/10.3102/0002831207312909.
6. Zimmerman, B.J.: Becoming a self-regulated learner: which are the key subprocesses? Con-
temporary Educational Psychology. 11(4), 307–313 (1986). https://doi.org/10.1016/0361-
476X(86)90027-5.
7. Pintrich, P.R.: The role of goal orientation in self-regulated learning. In: Handbook of Self-
Regulation. pp. 451–502. Elsevier (2000). https://doi.org/10.1016/B978-012109890-
2/50043-3.
8. Boekaerts, M., Maes, S., Karoly, P.: Self‐regulation across domains of applied psychology:
is there an emerging consensus? Applied Psychology. 54(2), 149–154 (2005).
https://doi.org/10.1111/j.1464-0597.2005.00201.x.
9. Sitzmann, T., Ely, K.: A meta-analysis of self-regulated learning in work-related training
and educational attainment: what we know and where we need to go. Psychological Bulletin.
137(3), 421–442 (2011). https://doi.org/10.1037/a0022777.
10. Boekaerts, M., Corno, L.: Self-regulation in the classroom: a perspective on assessment and
intervention. Applied Psychology: An International Review. 54(2), 199–231 (2005).
https://doi.org/10.1111/j.1464-0597.2005.00205.x.
11. Veenman, M.V.J.: Alternative assessment of strategy use with self-report instruments: a dis-
cussion. Metacognition Learning. 6, 205–211 (2011). https://doi.org/10.1007/s11409-011-
9080-x.
12. Winne, P.H., Perry, N.E.: Measuring self-regulated learning. In: Handbook of Self-Regula-
tion. pp. 531–566. Elsevier (2000). https://doi.org/10.1016/B978-012109890-2/50045-7.
13. Boekaerts, M., Pintrich, P.R., Zeidner, M.: Self-regulation: An introductory overview.
Handbook of Self-regulation. 1–9 (2000).
14. van Halem, N., van Klaveren, C., Drachsler, H., Schmitz, M., Cornelisz, I.: Tracking Pat-
terns in Self-Regulated Learning Using Students’ Self-Reports and Online Trace Data. FLR.
8(3), 140–163 (2020). https://doi.org/10.14786/flr.v8i3.497.
15. Hadwin, A.F., Nesbit, J.C., Jamieson-Noel, D., Code, J., Winne, P.H.: Examining trace data
to explore self-regulated learning. Metacognition and Learning. 2, 107–124 (2007).
https://doi.org/10.1007/s11409-007-9016-7.
16. Jamieson-Noel, D., Winne, P.H.: Comparing Self-Reports to Traces of Studying Behavior
as Representations of Students’ Studying and Achievement. Zeitschrift für Pädagogische
Psychologie / German Journal of Educational Psychology. 17, 159–171 (2003).
https://doi.org/10.1024/1010-0652.17.34.159.
17. Mitri, D.D., Schneider, J., Specht, M., Drachsler, H.: From signals to knowledge: A concep-
tual model for multimodal learning analytics. Journal of Computer Assisted Learning. 34(4),
338–349 (2018). https://doi.org/10.1111/jcal.12288.
18. Blikstein, P.: Multimodal Learning Analytics. In: Proceedings of the Third International
Conference on Learning Analytics and Knowledge. pp. 102–106. Association for Compu-
ting Machinery, New York, NY, USA (2013). https://doi.org/10.1145/2460296.2460316.
19. Cukurova, M., Kent, C., Luckin, R.: Artificial intelligence and multimodal data in the ser-
vice of human decision‐making: a case study in debate tutoring. British Journal of Educa-
tional Technology. 50(6), 3032–3046 (2019). https://doi.org/10.1111/bjet.12829.
20. Cukurova, M., Zhou, Q., Spikol, D., Landolfi, L.: Modelling collaborative problem-solving
competence with transparent learning analytics: is video data enough? In: Proceedings of
the Tenth International Conference on Learning Analytics & Knowledge. pp. 270–275.
ACM, Frankfurt Germany (2020). https://doi.org/10.1145/3375462.3375484.
12
21. Giannakos, M.N., Sharma, K., Pappas, I.O., Kostakos, V., Velloso, E.: Multimodal data as
a means to understand the learning experience. International Journal of Information Man-
agement. 48, 108–119 (2019). https://doi.org/10.1016/j.ijinfomgt.2019.02.003.
22. Khalil, M.: MOLAM: A Mobile Multimodal Learning Analytics Conceptual Framework to
Support Student Self-Regulated Learning. In Prinsloo, P., Slade, S., Khalil, M. (Eds.),
Learning analytics in open and distributed learning: Potentials and challenges.
SpringerOpen. (2020).
23. Lim, L.-A., Gentili, S., Pardo, A., Kovanović, V., Whitelock-Wainwright, A., Gašević, D.,
Dawson, S.: What changes, and for whom? A study of the impact of learning analytics-based
process feedback in a large course. Learning and Instruction. 72, 101202 (2021).
https://doi.org/10.1016/j.learninstruc.2019.04.003.
24. Viberg, O., Khalil, M., Baars, M.: Self-Regulated Learning and Learning Analytics in
Online Learning Environments: A Review of Empirical Research. (2020).
https://doi.org/10.1145/3375462.3375483.
25. Gašević, D., Joksimović, S., Eagan, B.R., Shaffer, D.W.: SENS: Network analytics to com-
bine social and cognitive perspectives of collaborative learning. Computers in Human Be-
havior. 92, 562–577 (2019). https://doi.org/10.1016/j.chb.2018.07.003.
26. Noroozi, O., Alikhani, I., Järvelä, S., Kirschner, P.A., Juuso, I., Seppänen, T.: Multimodal
data to design visual learning analytics for understanding regulation of learning. Computers
in Human Behavior. 100, 298–304 (2019). https://doi.org/10.1016/j.chb.2018.12.019.
27. Winne, P., Hadwin, A., Perry, N.: Metacognition and computer-supported collaborative
learning. International handbook of collaborative learning. 462–479 (2013).
28. Järvelä, S., Hadwin, A.F.: New Frontiers: Regulating Learning in CSCL. Educational Psy-
chologist. 48(1), 25–39 (2013). https://doi.org/10.1080/00461520.2012.748006.
29. Järvelä, S., Järvenoja, H., Malmberg, J., Hadwin, A.F.: Exploring Socially Shared Regula-
tion in the Context of Collaboration. Journal of Cognitive Education and Psychology. 12,
267–286 (2013). https://doi.org/10.1891/1945-8959.12.3.267.
30. Azevedo, R., Taub, M., Mudrick, N.V., Millar, G.C., Bradbury, A.E., Price, M.J.: Using
Data Visualizations to Foster Emotion Regulation During Self-Regulated Learning with Ad-
vanced Learning Technologies. In: Buder, J. and Hesse, F.W. (eds.) Informational Environ-
ments. pp. 225–247. Springer International Publishing, Cham (2017).
https://doi.org/10.1007/978-3-319-64274-1_10.
31. Beetham, H., Sharpe, R.: Rethinking pedagogy for a digital age: designing for 21st Century
Learning. Routledge, New York (2013).
32. Zhou, Q., Suraworachet, W., Pozdniakov, S., Martinez-Maldonado, R., Bartindale, T., Chen,
P., Richardson, D., Cukurova, M.: Investigating students’ experiences with collaboration
analytics for remote group meetings. In: Proceedings of the International Conference of Ar-
tificial Intelligence in Education, Springer, Cham (2021).
33. Box, G. E. P., Jenkins, G. M., Reinsel, G. C.: Introduction In: Box, G. E. P., Jenkins, G. M.,
Reinsel, G. C. (eds.) Time series analysis forecasting and control. pp. 7-18. John Wiley &
Sons, Inc., Hoboken, N.J. (2008).
34. Suraworachet, W., Villa-Torrano, C., Zhou, Q., Asensio-Pérez, J.I., Dimitriadis, Y., Cuku-
rova, M.: Examining the relationship between reflective writing behaviour and self-regu-
lated learning competence: A time-series analysis. In: Proceedings of the 16th European
Conference on Technology Enhanced Learning, Springer, Cham (2021).
35. Jivet, I., Wong, J., Scheffel, M., Valle Torre, M., Specht, M., Drachsler, H.: Quantum of
Choice: How learners’ feedback monitoring decisions, goals and self-regulated learning
skills are related. In: LAK21: 11th International Learning Analytics and Knowledge
13
Conference. pp. 416–427. ACM, Irvine CA USA (2021).
https://doi.org/10.1145/3448139.3448179.
36. Cho, M.-H., Kim, Y., Choi, D.: The effect of self-regulated learning on college students’
perceptions of community of inquiry and affective outcomes in online learning. The Internet
and Higher Education. 34, 10–17 (2017). https://doi.org/10.1016/j.iheduc.2017.04.001.
37. Andrade, H., Brookhart, S.M.: The role of classroom assessment in supporting self-regulated
learning. In: Laveault, D. and Allal, L. (eds.) Assessment for Learning: Meeting the Chal-
lenge of Implementation. pp. 293–309. Springer International Publishing, Cham (2016).
https://doi.org/10.1007/978-3-319-39211-0_17