=Paper=
{{Paper
|id=Vol-1841/E01_26
|storemode=property
|title=Assessing the Effectiveness of Self-Regulated Learning in MOOCs Using Macro-level Behavioural Sequence Data
|pdfUrl=https://ceur-ws.org/Vol-1841/E01_26.pdf
|volume=Vol-1841
|authors=Lan Min,Lu Jingyan
|dblpUrl=https://dblp.org/rec/conf/emoocs/LanL17
}}
==Assessing the Effectiveness of Self-Regulated Learning in MOOCs Using Macro-level Behavioural Sequence Data==
https://ceur-ws.org/Vol-1841/E01_26.pdf
Proceedings of EMOOCs 2017:
Work in Progress Papers of the Experience and Research Tracks and Position Papers of the Policy Track
Assessing the Effectiveness of Self-Regulated Learning in
MOOCs using Macro-level Behavioural Sequence Data
Lan Min1, Lu Jingyan2
1,2
The University of Hong Kong, Faculty of Education, Hong Kong, China
minlan@hku.hk, jingyan@hku.hk
Abstract. Learners are demanded more self-regulatory capability to carry on
effective online learning. Online course instructors attempt to stimulate online
learners’ effective self-regulated learning (SRL) to support effective learning
and enhance achievement. Knowing how the online learners learn in SRL loop
will contribute to the effective course design and scaffolding. In this study, the
learners from an edX MOOC were differentiated into more effective self-
regulated learners (SRLers) and less effective SRLers based on the criteria of
three SRL phases behavioural sequence patterns. The clickstream data of 5764
learners was analyzed on macro-level behavioural learning sequence through n-
gram algorithm. Persistence and grade were compared among the different
types of learners. The results showed us that more effective SRLers persisted
longer and performed better than less effective SRLers on a significant level.
Keywords: Self-regulated Learning, Behavioural Sequence Pattern, MOOC,
Persistence, Learning Achievement
1 Introduction
Recently, massive open online courses (MOOCs) offer people more opportunities to
access the free educational resources. Learners are required more self-regulation
capability to self-learning with fewer instructors’ assistance. Various emerging
problems on learning and teaching in MOOCs challenge the MOOCs development.
For instance, the prominent high attrition phenomenon [1] leads the persistence issue
to become one of the controversial topics for MOOC study. According to Wigfield,
Klauda, and Cambria [2], persistence is a key behavioural indicator of self-regulatory
capacity in the monitoring and control phases of self-regulation. Most previous
persistence research in SRL field was conducted in face to face situation using a self-
reported questionnaire or experiment [3-4]. In some e-learning settings, for instance,
Azevedo and his colleagues [5] developed a tool named Metatutor to assess and
convey SRL. These studies suggested that the adaptive scaffolding based on learners’
navigation path was necessary to fostering learners’ learning and use of key SRL
processes. However, these fine-grained scaffoldings stressed on time-sensitivity that
remains a challenge for researchers [5], because micro-level click actions from whole
course learning behaviour perspectives generated uncountable learners’ types, which
are inapplicable for scaffolding from SRL aspects. When it comes to the natural
learning processes in the online environment, to accurately detect, track, and model
students’ SRL processes will remain a challenge as well [6-7]. In this study, through
1
Proceedings of EMOOCs 2017:
Work in Progress Papers of the Experience and Research Tracks and Position Papers of the Policy Track
exploring the clickstream data in an edX MOOC, based on the edX platform feature
and course design feature, the learning processes were detected to see how the SRL
behavioural sequences patterns reflect persistence and achievement.
2 Related Work
Studies show that self-regulatory processes are different among students and the
differences lead to the variance in achievement [8-9]. Schunk [10-11] states that self-
regulated learning (SRL) is an effective means to improve achievement. According to
self-regulated learning theory, students are self-regulated to the degree they are
metacognitively, motivationally, and behaviorally active participants in their learning
processes [12]. These metacognitive processes include goal setting, self-monitoring,
and self-evaluative feedback loops. Motivational feelings and beliefs refer to self-
regulated learners’ display of personal initiative, perseverance, and adaptive skills.
Researchers may be unable to observe the learners’ psychological states or cognitive
thinking. However, behaviorally, self-regulation refers to specific beneficial actions,
such as record keeping, environmental structuring, and help-seeking, which are
observable [12].
In traditional studies, SRL measuring included self-reported questionnaires,
structured interviews, teacher ratings, think aloud methods, error detection tasks, trace
methodologies, and observations [13]. Questionnaires and interviews are two most
widely-used measurements in MOOC studies on SRL topics. Kizilcec et al.’s studies
[3-4] provided us with some implications of how learners perform in MOOC context,
but, in terms of research method, as Winne and Perry [13] indicated, self-report
measures do not necessarily give a reliable picture of the self-regulation tactics
students actually engage in.
The fine-grained log data that were recorded by online platform give a chance to
apply trace methodology and learning analytics on every or groups of participants. In
a MOOC video study, researchers suggested that the clickstream data organized by
using n-gram algorithm method may help people understand how the online learners
process information [14]. According to topic modeling and n-gram analysis,
researchers construct the behavioural actions sequences into several categories to
classify how these learners interact with the online videos and process information
[14]. However, viewing learners’ behaviours as a whole, their online actions on other
pages or activities also reflect their cognitive thinking. Moreover, instructional
designers promise that learners who follow an underlying SRL behavioural sequence
pattern may have a better learning achievement. We assume that the more effective
SRL learners would have a relatively regular macro-level learning sequence. In other
words, when the actual online learning behaviours are calibrated with or similar to the
ideal learning sequences, they may get on the right track of SRL loop and learn better
and longer in an online course.
According to social cognitive SRL model from Zimmerman [15], the three-stage
learning process consist of forethought and planning phase, performance monitoring
phase and self-reflection phase. The more effective SRLers who experience the
complete three-stage loop are assumed to learn better than those who do not
2
Proceedings of EMOOCs 2017:
Work in Progress Papers of the Experience and Research Tracks and Position Papers of the Policy Track
experience the completed SRL loop. From this perspective, we can distinguish
effective form from ineffective forms self-regulation from the quality and quantity of
one’s self-regulatory behavioural sequence processes [15].
3 Research Questions
1. What types of learners do we have in MOOC based on macro-level SRL
behavioural sequence patterns?
2. What is the relationship between the macro-level SRL behavioural sequences
and persistence in MOOC?
3. What is the relationship between the macro-level SRL behavioural sequences
and grade in MOOC?
4 Method
4.1 Course Context
The MOOC, Epidemics, offered by The University of Hong Kong on edX was used in
this study. It composed of ten weeks of learning materials that were broken up into
four themes. There were 4-10 video lectures in each week (most videos were 5-10
minutes). Quiz was the only assessment in this MOOC. According to the navigational
bar, this MOOC has five blocks: Home (H), Course (C), Course Details (CD),
Discussion (D) and Progress (P). The learners’ click action on each block will create
one macro-level behaviour record. Under Course block, as the video watching and
quiz taking are two main activities, they are also defined as two macro-level activities.
4.2 Data Collection
Data Cleaning
Raw data were collected from 5764 learners who did at least one action in this
MOOC during the first running time. It was organized based on three items: user id
code, time, and event. Non-learners’ behaviour records (e.g. system self-generated
records and instructors’ records) were removed. Then, the micro-level behaviours
were clustered into a macro-level behaviour if the same macro-level behaviour
category was in a linear sequence based on the time series. For instance, in a time
duration 07:00:00 to 07:20:00, all the micro-level behaviours for doing the quiz that
included the actions of saving the answer, checking the answer and showing the
correct answer were marked as “Q”. Table 1 illustrates the ID-based macro-level
behaviour sequence list.
3
Proceedings of EMOOCs 2017:
Work in Progress Papers of the Experience and Research Tracks and Position Papers of the Policy Track
Table 1. Learner macro-level behaviour sequence list
ID Learning Sequence
3465 H CD C V D C H V CD D C V C V Q C V
5317 H CD D P C V C V C V Q C V
H C V C V C V Q V Q V C Q C V Q C V Q V Q V Q V Q C V C V H P H CD C H C
5879
V C V C ...
Data Analysis
In this study, the n-gram is the contiguous sequence of n items from a learner’s online
learning clickstream record on the macro-level behaviours. For instance, the
behavioural sequences “H-CD-C-V-Q-D-P” would be composed of 2-grams (e.g. H-
CD, CD-C, C-V, V-Q, Q-D, D-P), 3-grams (e.g. H-CD-C, CD-C-V, C-V-Q, V-Q-D,
Q-D-P) and 4-grams (e.g. H-CD-C-V, CD-C-V-Q, C-V-Q-D, V-Q-D-P).
Researchers predict some sequences that may reflect the SRL behaviours. During
the forethought phase, learners prepare work before performance phase on their
studying [15]. In an online setting, any navigational button that provides the
course information may attract learners’ attention to preview/overview the
course as a pre-action. In this study, Course details (CD) offers the information
about the course learning outcomes, course syllabus, course assessing types and grade
criteria, which the learners may go through before doing activities in the performance
phase. In performance phase, self-control and self-monitoring occur in the learning
process that involves learner’s attention and willpower [15]. In the relatively fixed
online setting, the behaviours in performance phase mainly happen during the
activities. The activities are mainly in the Course page (C), which include video
lecture (V), quiz (Q), and discussion (D). Learners carried out different task strategies
to monitor their learning process. Many reasonable sequences patterns can be the
combination of C, V, Q and D. Self-reflection phase happens in the final stage when
learners review their performance toward final goals [15-16]. Any course design that
helps learners review their performance or match to the final goals can be
potentially lead to actions in self-reflection phase. Progress page (P) helps the
learners check their learning progress is going on. Since the quiz is the only
assessment in this course, a possible self-reflection behavioural sequence pattern may
be the Q-P and V-Q-P or Q-P-CD because after taking the quiz the learner may
review their performance on P. Above all, three SRL behavioural sequence patterns
were proposed in Table 2. Each learner will have a count record of sequence pattern
in each of the three SRL phases. More effective SRLers are those who did action in
each SRL phase, while less effective SRLers may missed one or two phases.
Persistence is defined as the weeks the learner stays in the course (10 weeks in
total). Achievement is the grade (1 point in total). One-way ANOVA will be
conducted to observe the significant differences among the more effective SRLers
and less effective SRLers on persistence and achievement. In the relationship between
the persistence/the grade and three SRL phases, multiple regression will be applied.
4
Proceedings of EMOOCs 2017:
Work in Progress Papers of the Experience and Research Tracks and Position Papers of the Policy Track
Table 2. Three SRL Behavioural Sequence Patterns
Phase Sequences
Forethought Phase H-CD, CD-C, H-CD-C, H-C-CD-D-P (or H-CD-D-P-C or C-H-CD-
D-P)
Performance Phase C-V, V-Q, C-V-Q, C-D, V-D, C-V-D, Q-D, C-Q-D, V-Q-D, C-V-Q-D
Self-reflection Phase P-CD, Q-P, Q-P-CD, V-Q-P, V-Q-P-CD
5 Result
5.1 Macro-level SRL Behavioural Sequences Overview (RQ1)
According to the definitions of more effective SRLer and less effective SRLer
mentioned above, it is found that the number of less effective-SRLers was nearly four
times of that of more effective SRLers (See Figure 1).
5000
4000
3000
2000
1000
0
MEF-SRLer LEF-SRLer
Number 1185 4579
Fig. 1. Number of More Effective-SRLers (MEF-SRLer) and Less Effective-SRLers
(LEF-SRLer) (N=5764)
The less effective SRLers were further subdivided based on the phase(s) they
missed (See Figure 2). About 45% less effective SRLers missed both the forethought
and self-reflection phases, and around 10% and 35% less effective SRLers missed
forethought phase and self-reflection phase respectively. There were even about 10%
less effective SRLers missed the performance phase based on the SRL behaviours
criteria. These learners are regarded as browsers (“Others” in the figure).
2500
2000
1500
1000
500
0
LEF-SRL_mf LEF-SRL_msr LEF-SRL_mf&sr Others
Number 471 1615 2034 459
*LEF-SRL_mf: only miss forethought phase; LEF-SRL_msr: only miss self-
reflection phase; LEF-SRL_mf&sr: miss forethought phase and self-reflection phase;
Others: those who assess to the course but may never do any course-related activities
Fig. 2. Types of less effective SRLers (LEF-SRLers)
5
Proceedings of EMOOCs 2017:
Work in Progress Papers of the Experience and Research Tracks and Position Papers of the Policy Track
5.2 Persistence and SRL Behavioural Sequence Patterns (RQ2)
As shown in Table 3, more effective SRLers had the highest mean value of
persistence and others had the lowest. With the one/two SRL phase(s) missing, the
persistence decreases. Especially when the self-reflection phase was missing, the
persistence accelerated dropped slightly. One-way ANOVA analysis and Tukey test
showed that the persistence of the different types of learners was different
significantly (F = 886.146, p-value < 0.001***).
Table 3. Persistence and More Effective-SRLers (MEF-SRLer) and Less Effective-
SRLers (LEF-SRLer)
MEF- LEF- LEF- LEF-
Others
Persistence SRLer SRL_mf SRL_msr SRL_mf&sr
(n=459)
(n=1185) (n=471) (n=1615) (n=2034)
Mean 6.528 5.461 2.975 2.197 1.242
Std 2.966 3.000 2.384 1.816 0.684
Min 1 1 1 1 1
25% 4 3 1 1 1
50% 7 5 2 2 1
75% 9 8 4 3 1
Max 10 10 10 10 9
We found that forethought phase and performance phase had a small positive
effect on persistence through ordinary least square (OLS) multiple regression, but
they all reached the significant level on statistics [Adj. R2 (0.480), Prob (F-statistics <
0.001), the coefficients of the three independent values (b-forethought = 0.0482, r <
0.01**, b-performance = 0.0170, r < 0.01**, b-reflection = -0.0008, r < 0.878)]. Self-
reflection had a little negative effect on persistence, but it did not reach the significant
level on statistics.
5.3 Grade and SRL Behavioural sequence patterns (RQ3)
Table 4 shows that the more effective SRLers had the highest average grade, and
others hardly got mark. With one or two SRL phase(s) missing, the average grade
dropped. Especially when the self-reflection phase was missing, the grade accelerated
dropped slightly. One-way ANOVA analysis and Tukey test showed that the grade of
the different types of learners was different significantly (F = 972.546, p-value <
0.01**).
Interestingly, there were 15 less effective SRLers who got the full mark (1 point),
while only 8 more effective SRLers got the full mark. These less effective SRLers
hardly did actions in the forethought phase. Their learning behaviours focused on
Course page, video lecture, and quiz. More specifically, they regularly learn in the
course with the sequence pattern “C-V-Q”.
6
Proceedings of EMOOCs 2017:
Work in Progress Papers of the Experience and Research Tracks and Position Papers of the Policy Track
Table 4. Grade and More Effective SRLers (MEF-SRLer) and Less Effective SRLers
(LEF-SRLer)
LEF- LEF- LEF-
MEF-SRLer Others
Grade SRL_mf SRL_msr SRL_mf&sr
(n=1185) (n=459)
(n=471) (n=1615) (n=2034)
Mean 0.538 0.410 0.120 0.051 0.002
Std 0.35 0.357 0.238 0.144 0.041
Min 0 0 0 0 0
25% 0.16 0 0 0 0
50% 0.67 0.41 0 0 0
75% 0.85 0.76 0.12 0.01 0
Max 1 1 1 1 0.87
We found that forethought phase had a little negative effect on grade through OLS
regression, but it did not reach the significant level. Performance phase had a small
positive effect on grade, and it reached the significant level on statistics. Self-
reflection had a little negative effect on persistence, and it reached the significant
level on statistics [Adj. R2 (0.544), Prob (F-statistics < 0.001), the coefficients of the
three independent values (b-forethought = -0.0009, r = 0.639, b-performance =
0.0023, r < 0.05*, b-reflection = -0.0013, r < 0.05*)].
6 Discussion
6.1 Macro-Level SRL Behavioural Sequence Patterns
Good instructional design is believed to support and stimulate SRL [17]. On the one
hand, the course design should be close to the natural SRL process that supports
learners’ essential capability to self-direct; on the other hand, the more support on the
SRL strategies should facilitate different people to construct a personalised learning
model for their SRL strategies. Then, if the course was designed with more pages of
activities for preparation before performance phase and reflection after performance
phase, the learners may pay more attention to the forethought phase and self-
reflection phase. In this study, from the course design perspective, this MOOC offers
relative concise course design that participants can follow the designed steps to learn
one by one easily. Only if the participant persist on the learning activities, they may
get a good result. However, those defined less effective SRLers may still have used
other strategies that did not leave action records online in these two phases (e.g. time
manage on schedule and taking note after learning for reflection).
6.2 Persistence, Grade and SRL
In general, the more effective SRLers were significantly persisting longer and
achieving higher than the less effective SRLers. To outward seeming, with the
forethought phase and self-reflection phase missing in online learning, both
persistence and grade performance dropped dramatically. The decreasing phenomena
7
Proceedings of EMOOCs 2017:
Work in Progress Papers of the Experience and Research Tracks and Position Papers of the Policy Track
especially happened when they missed self-reflection phase. As mentioned above, the
self-reflection phase was defined as reviewing progress page after doing the quiz or
before reviewing course detail for assessing information again. Before the less
effective SRL_msr learners did something on the progress page, they have already
made the decision between keep learning in this course and other potential events. In
other words, when we see a learner missing the reflection phase in this course, he/she
may have already logged out of the course at that time. That is why the decreasing of
persistence and grade was steeper than those just miss the forethought phase.
Regarding the multiple regression results, how much the learners did in each phase
had little or no effect on both persistence and grade. For the full mark less effective
SRLers, these learners may have a clear purpose in this course, such as getting a
certificate (performance goal), because their learning sequence pattern was regularly
on “C-V-Q...C-V-Q”. And their understanding of these contents was well. However,
we did not know if they have reviewed the videos or checked other learning resources
before taking the quiz (micro-level). Therefore, exploring the micro-level behaviours
is necessary to explain the learning performance in future studies.
7 Conclusion
More effective SRLers and less effective SRLers were differentiated based on the
three SRL phases behavioural sequence patterns according to Zimmerman’s three
stages of SRL. The finding showed that more effective SRLers outperform less
effective SRLers on both persistence and grade significantly. Compared to the
Zimmerman’s SRL model, the concrete SRL behavioural sequence patterns were
constructed through n-gram method. However, the definition of the SRL behaviours
in this study was on narrow sense, which means more patterns can be involved in
other situations. In future studies, micro-level behavioural sequence patterns studies
can be explored.
8 Limitations and Future Studies
In online learning setting, participants’ learning process is the result of both internal
and external factors, including people’s motivation, emotion, psychological and
cognitive thinking, and online learning environment. In this study, we only focused
on the macro-level behavioural sequences from the superficial perspective based on a
fixed online learning environment design. According to Zimmerman’s three stages
model [15], SRL is a cycle. It is difficult to separate SRL from subsequent three
stages activities. Furthermore, n-gram algorithm helps us capture the most frequent
behavioural sequences from the long clickstream based on the pre-defined patterns.
However, it will not automatically help us identify different types of behaviours. In
future, two aspects can be explored: 1) supplemented internal factors research to
validate the impact of SRL behavioural sequences on achievement; 2) design-based
MOOC experiment to see what kind of course design or interface navigational design
can facilitate more effective SRL behavioural sequences that lead to higher
achievement, persistence and engagement.
8
Proceedings of EMOOCs 2017:
Work in Progress Papers of the Experience and Research Tracks and Position Papers of the Policy Track
References
1. Koller, D., Ng, A., Do, C., & Chen, Z.: Retention and intention in massive open online
courses: In depth. Educause review, 48(3), 62-63 (2013).
2. Wigfield, A., Klauda, S. L., & Cambria, J.: Influences on the Development of Academic
Self Regulatory Processes. In B. J. Zimmerman & D. H. Schunk (Eds.), Handbook of self-
regulation of learning and performance (pp. 40). New York: Taylor & Francis (2011).
3. Kizilcec, R. F., Pérez-Sanagustín, M., & Maldonado, J. J.. Self-regulated learning
strategies predict learner behavior and goal attainment in Massive Open Online Courses.
Computers & Education, 104, 18-33 (2017).
4. Kizilcec, R. F., Pérez-Sanagustín, M., & Maldonado, J. J.. Recommending self-regulated
learning strategies does not improve performance in a MOOC. In Proceedings of the Third
(2016) ACM Conference on Learning@ Scale (pp. 101-104). ACM (2016, April).
5. Azevedo, R., Johnson, A., Chauncey, A., & Graesser, A.: Use of Hypermedia to Assess
and Convey Self-Regulated Learning In B. J. Zimmerman & D. H. Schunk (Eds.),
Handbook of self-regulation of learning and performance (pp. 115). New York: Taylor &
Francis Group (2011).
6. Azevedo, R. Theoretical, methodological, and analytical challenges in the research on
metacognition and self-regulation: A commentary. Metacognition & Learning, 4(1), 87-95
(2009).
7. Azevedo, R., Moos, D. C., Johnson, A. M., & Chauncey, A. D.: Measuring cognitive and
metacognitive regulatory processes during hypermedia learning: Issues and challenges.
Educational Psychologist, 45(4), 210-223 (2010).
8. Zimmerman, B. J., & Martinez-Pons, M.: Development of a structured interview for
assessing student use of self-regulated learning strategies. American Educational Research
Journal, 23(4), 614-628 (1986).
9. Zimmerman, B. J., & Martinez-Pons, M.: Construct validation of a strategy model of
student self regulated learning. Journal of Educational Psychology, 80(3), 284 (1988).
10. Schunk, D. H.: Modeling and attributional effects on children's achievement: A self-
efficacy analysis. Journal of Educational Psychology, 73(1), 93-105 (1981).
11. Schunk, D. H.: Sequential attributional feedback and children's achievement behaviors.
Journal of Educational Psychology, 76(6), 1159-1169 (1984).
12. Zimmerman, B. J.: Motivational Sources and Outcomes of Self-Regulated Learning and
Performance. In B. J. Zimmerman & D. H. Schunk (Eds.), Handbook of self-regulation of
learning and performance (pp. 49). New York: Taylor & Francis Group (2011).
13. Winne, P. H., & Perry, N. E. (2000). Measuring self-regulated learning. In M. Boekaerts,
P. R. Pintrich, & M. Zeidner (Eds.), Handbook of self-regulation (pp. 531-566). San Diego,
CA, US: Academic Press.
14. Sinha, T., Jermann, P., Li, N., & Dillenbourg, P.: Your click decides your fate: Inferring
information processing and attrition behavior from mooc video clickstream interactions.
arXiv preprint arXiv:1407.7131, 1-12 (2014).
15. Zimmerman, B. J.: Attaining Self-Regulation: A Social Cognitive Perspective. In M.
Boekaerts, P. R. Pintrich, & M. Zeidner (Eds.), Handbook of self-regulation (pp. 13-39).
California: An Imprint of Elsevier (2000).
16. Phan, T., McNeil, S. G., & Robin, B. R. Students’ patterns of engagement and course
performance in a Massive Open Online Course. Computers & Education, 95, 36-44.
doi:10.1016/j.compedu.2015.11.015 (2016)
17. Jeske, D., Backhaus, J., & Stamov Roßnagel, C.: Self- regulation during e-learning: using
behavioural evidence from navigation log files. Journal of Computer Assisted Learning,
30 (3), 272-284 (2014).
9