=Paper=
{{Paper
|id=Vol-1596/paper3
|storemode=property
|title=Encouraging Metacognition & Self-Regulation in MOOCs through Increased Learner Feedback
|pdfUrl=https://ceur-ws.org/Vol-1596/paper3.pdf
|volume=Vol-1596
|authors=Dan Davis,Guanliang Chen,Ioana Jivet,Claudia Hauff,Geert-Jan Houben
|dblpUrl=https://dblp.org/rec/conf/lak/DavisCJHH16
}}
==Encouraging Metacognition & Self-Regulation in MOOCs through Increased Learner Feedback==
https://ceur-ws.org/Vol-1596/paper3.pdf
Encouraging Metacognition & Self-Regulation in MOOCs
through Increased Learner Feedback
Demonstration
Dan Davis∗, Guanliang Chen†, Ioana Jivet, Claudia Hauff and Geert-Jan Houben
Delft University of Technology
Delft, the Netherlands
{d.j.davis, guanliang.chen, i.jivet, c.hauff, g.j.p.m.houben}@tudelft.nl
ABSTRACT
Learning analytics for learners has the ability to greatly im-
prove learners’ self-regulation. Current learner dashboards
are mostly providing learners with an isolated view of their
learning behavior, while we believe learners will gain more
from a comparison of their own behavior with that of suc-
cessful peer learners. In this work-in-progress demonstration
we describe our design of a Learning Tracker widget that
provides MOOC learners with timely and goal-oriented (i.e.
towards passing the course) feedback in a manner that en-
courages reflection and self-regulation. We also present some
preliminary findings which show how exposure to feedback Figure 1: Sample edX dashboard: shows individual
can significantly increase student success and engagement. students’ weekly & total assessment scores
Keywords
Learner Feedback, Learning Analytics, Self-Regulated Learn-
ing, Study Planning
1. INTRODUCTION
The asynchronous, open nature of MOOCs presents stu-
dents with a profound sense of flexibility and freedom in
their learning experience compared to the traditional class-
room setting. They may study what they want, where they
want, and whenever they want. However, along with these Figure 2: Sample Coursera dashboard: shows indi-
ostensibly-positive affordances come major challenges. In vidual students’ grade for each quiz, whether or not
order to be successful in such a learning environment—with they passed, and the total number of quizzes passed
no pressure from teachers/parents, no financial obligations,
and no academic credit on the line—students must stay in-
credibly disciplined in both the planning and following of flow ecosystem. We believe that MOOC learners can sig-
their study habits. Dropout rates of around 95% in the av- nificantly benefit from a timely and goal-oriented feedback
erage MOOC [8] are a testimony to the challenge learners of their study habits in MOOCs. Currently, major MOOC
face in this environment. platforms provide rather generic learner feedback as seen in
The discipline for planning and following a self-imposed Figure 1 and Figure 2, which – while being timely – does
schedule does not come naturally to many learners; rather not enable learners to judge their learning behavior in abso-
it is a learned skill. And while merely releasing open edu- lute terms: are they on track to succeed in (i.e. pass) this
cational resources to the world for consumption is a great course? Are they nearly on track? Are they missing a key
start, the next step in the Open Learning movement ought ingredient to being successful?
to equip learners with the cognitive toolset they need to ef- We believe that instead of providing a general overview of
fectively self-regulate their learning experience. learner behavior, learners will be able to self-regulate better
Currently, universities, instructors, and researchers are if we provide them with a comparison of their own learning
the chief handlers of educational data generated from MOOCs. behavior against that of previous successful (in the sense
Learners do not yet form an important part of this data that they passed the course) students. We have developed a
∗
The author’s research is supported by the Leiden-Delft- first learner widget that reflects this vision, enabling learners
Erasmus Centre for Education and Learning. to compare themselves to successful learners and thus em-
† powering them to reflect on and adapt their study behavior
The author’s research is supported by the Extension School
of the Delft University of Technology. in a goal-oriented fashion.
1
Copyright © 2016 for the individual papers by the papers' authors. Copying permitted only for private and academic purposes.
This volume is published and copyrighted by its editors. LAL 2016 workshop at LAK '16, April 26, 2016, Edinburgh, Scotland.
Not only does this ease the burden of instructors (the 2.3 Increasing Learner Efficiency
learners decide how to react to and interpret the information Guo and Reinecke [4] studied to what extent students in
shown to them), it also creates a heightened awareness in MOOCs access the full offering of learning materials. Sam-
learners that they can keep with them beyond just this one pling from four edX MOOCs, they found that, on average,
course and apply in future professional or academic contexts. certificate-earning students do not access, or “ignore,” 22%
The following research question guides our line of inquiry of course materials [4]. Although instructors and instruc-
into the topic: tional designers may not be too pleased by this finding, it has
Can a comparison to previously successful learners serve as the potential to make future students more efficient in their
a helpful form of feedback to increase MOOC learners’ en- learning. If there is certain content that students repeatedly
gagement and success? skip without having their grade suffer, future students—
In this paper, we describe our prototype widget, the design maybe low on time or extrinsically motivated—can refine
decisions behind it, the setup we are currently employing in their learning plan based on this information.
our experiments, and a preliminary analysis of the results.
We find that indeed, our implemented feedback has a sig-
2.4 Open Learning Analytics
nificant positive effect on the success of the learners (in terms The Learning Tracker realizes much of the personal-level,
of grading) as well as on two out of six evaluated engagement student-facing dashboard envisioned in [14]. Along with the
metrics. three other views (educator, researcher, and institutional),
[14] proposes a dashboard in which students can see metrics
ranging from progress compared to current peers, previous
2. RELATED WORK students who took the course, their own past activities, and
instructor-defined benchmarks. Siemens et al. [14] here also
2.1 Search Dashboard suggest multiple levels of the dashboard, such as options for
The main inspiration for this research comes from Bate- “drilling down” into more detailed data visualisations.
man et al. [1] who, in the context of Web search, created Siemens [12] calls for Personal Learner Knowledge Graphs
a “Search Dashboard” that provides an interface for search to boost awareness of a student’s own current knowledge
engine users to see and reflect on their search behavior and, state in a given topic. This idea then evolved into Per-
furthermore, a comparison of this data against “archetypal sonal Learning Graphs [13], which stress the “importance of
expert profiles.” Their approach is very similar to ours in individuals owning their own learning representation” [13].
that they outline searching as something people have come While the present Learning Tracker widget is not owned
to depend on in every day life, but rarely do people consider by the learner, it empowers MOOC students to assume a
searching as a skill that may be developed and improved. more active role in shaping their own learning experience.
The same can be said about learning. Bateman et al. [1] To our knowledge, these remain undeveloped and only con-
found that people rarely change their search behavior no ceptualisations of what dashboards should be.
matter the situation, but, once exposed to the dashboard,
they become more active, aware, and critical of their search- 2.5 Dashboards as Explorable Visual Narra-
ing habits—adapting them to be more in line with those of tives
the visualized expert searchers. To see if learner feedback data visualisations can elicit
change in student behavior (similar to our research ques-
2.2 Feedback to Encourage Metacognition tion), Yousuf and Conlan [17] implemented a dashboard
The key processes underpinning our Learning Tracker (VisEN) that intended to emphasize to the student viewers
widget are that of (i) feedback prompting and (ii) metacog- a sort of “visual narrative” in the form of data visualiza-
nition, which then results in (iii) more effective self-regulated tions. There is no text-based narrative provided for the stu-
(or self-directed) [7] learning. Durall and Gros [3] and Ver- dents; rather, this dashboard, pictured in Figure 3, consists
bert et al. [16] also outline this process of providing stu- of heavily-annotated data visualizations from which students
dents with “self-knowledge” as being key to developing the were expected to draw their own narrative arc. Findings
necessary metacognitive skills for self-regulated (or directed) from their three studies, taking place over three years and in-
learning. And in order to ease the translation from data to cluding 223 students, yielded a very strong Pearson correla-
actionable knowledge, Heer and Agrawala [5] found informa- tion coefficient between dashboard views and learner engage-
tion visualization to be an effective sense-making tool due ment [17]. A fundamental difference between this approach
to its ability to synthesize complex data in a way for viewers and our Learning Tracker presented here is that VisEN in-
to quickly understand and compare. corporated tactics to directly encourage engagement such as
An early example of instructions designed to empower reminders and “bad or poor engagement notifications”. Our
learners to shape their own learning experience dates back to Learning Tracker, on the other hand, stops short of any
1965, where Keller [6] introduces and documents the result of direction-giving or motivation and merely presents the learn-
a “go-at-your-own-pace” course. This resulted in an inverted ers with a comparative view of their own behavior and that
(U-shaped), polarized (highest concentrations for Grades ’A’ of successful learners. Furthermore, the Learning Tracker
and ’F’) grade distribution at the conclusion of the courses, dashboard operates at scale in MOOCs.
making clear the difference between students who can and
cannot self-regulate effectively. Increased learner feedback 3. WIDGET DESIGN
and awareness could be the nudge some of these students
Based on prior works [11, 10, 9, 15] that have investigated
need to remain engaged and pass the course. the factors impacting learner success in MOOCs and effec-
tive feedback strategies (such as the “simple design prin-
ciples” outlined in [2]) and some subjective judgement on
2
among the middle 90% of gold standard learners—thus the
value of the outer ring increases each week, and the zero
point remains constant.
We aim to ensure that any actions learners take in re-
sponse to the widget are self-conceived. In order to do so, we
try to minimize any feelings of external judgment or assess-
ment from the visualisations by making them as “modest”
as possible—“simply making things visible that would other-
wise remain invisible” [2]. There are no red “danger zones”
or green “in-the-clear zones” on the chart as are found in
other learning dashboards such as VisEN [17] or Coursera
(Figure 2). Rather, we present a chart free of not only zones,
but also any numbers, similar to the “degraded information”
concept in [2]. All students see is their relative position com-
pared to the set of successful learners on the same plane. It
is left up to the learners how to interpret it, what to learn
from it, and how to convert this information into actionable
knowledge.
4. EXPERIMENTAL SETUP
We deployed our widget in the TU Delft MOOC Intro-
duction to Drinking Water Treatment running in its second
edition on the edX platform between January 12 and March
29, 2016 (11 course weeks in total). The first iteration of
Figure 3: Selected visualisations from a VisEN stu-
the MOOC ran in 2014, with 10,695 registered learners of
dent’s visual narrative
whom 281 (2.6%) earned a passing grade.
For this year’s edition 10,943 users enrolled before the of-
our behalf, we identified six indicators that are related to ficial start of the course and, in turn, participated in our
learner success and at the same time readily understandable experiment. Using A/B testing, we presented the Learn-
to learners: ing Tracker widget to 49.91% (5,462) of the learners. At
• Time on the platform in seconds the start of every course week, the learners were shown on
• Time watching videos in seconds the course page how they compared (up to that point in
the course) to our gold standard learners from last year.
• Fraction of time spent watching videos while on the platform:
whereas the previous two give total time commitment mea- Alongside the visualisations (concrete examples of which are
sures, this provides feedback on how they allocate their time shown in Figures 4 and 5) we also provided a short explana-
in the course tory text that included the following statement:
• Number of course videos watched
These graphs are not meant to be judgements or
• Number of graded quiz answers submitted assessments of your learning in any way; rather,
• Timeliness of quiz answer submission: how early students sub- they are a source of feedback for you, the learner,
mit answers relative to the deadline, to expose procrastinating to make you more aware of your study habits and,
behavior hopefully, help you change them for the better!
In order to enable learners to directly compare their be-
havior to successful learners, we require a set of “gold stan- This 11-week course consists of one introduction week, five
dard” successful learners. In MOOCs that are reruns (our weeks of content delivery, and two design assignments that
target in this work) we can simply consider all learners that cover the remaining five weeks. The material published in
passed one or more of the MOOC’s previous editions to make each content delivery week included an assignment with five
up this set. These successful learners do not exhibit a uni- quiz questions. The video-lectures were complemented by a
form behavior. However, if we consider the average or me- total of 63 practice quiz questions that were not graded.
dian across all these learners for each indicator, we have a In order to graduate, learners had to earn a final grade
relatively robust indicator. For each indicator, the values higher than 60. We observe an increase in the percentage of
that fall in the bottom 5% and the top 5% of the data range certificate-earning learners compared to last year’s edition
are omitted. of the MOOC: 3.18% (348 out of 10,943 learners).
Having prototyped several different visualizations of our
indicators (including bar charts, gauges and calendar charts), 5. RESULTS
we settled on the use of a spider chart as shown in Figure 4.
We now provide an overview of our preliminary findings.
Spider charts allow for (i) a concise visualisation of numer-
The results are based on all edX log traces up to and in-
ous metrics in a small space, (ii) simple legibility—data are
cluding week 9 of the MOOC1 . Due to the low number of
shown as single points along straight lines, and (iii) a visual
learners that visited the course material after the course
depiction of one’s coverage and consistency across all met-
started (3,787 - 34.6% of enrolled) and the high drop-out
rics. To allow for a consistent representation in the same
graph, all metric values are scaled in a range from 0 to 10, 1
The remaining course weeks are not included in the analy-
where 0 indicates no activity and 10 the maximum value sis, as the log traces are not yet available.
3
Figure 4: Three versions of our Learning Tracker widget with data from week 9, one showing a learner who
dropped out early in the course (left), one who dropped out in the middle of the course (middle), and one
graduate who is highly engaged with the course (right).
Figure 5: One learner’s widget for weeks 3 (left), 6 (middle) and 9 (right). These show that this particular
learner was a late starter and began engaging with the course some time between weeks 3 and 6.
rate in the first week of the course (19.26% of enrolled did In Figure 7 we present the timeliness of the two groups
not return after week one), the data distribution is highly with respect to the weekly quiz deadlines: the test group is
skewed. We analyzed active learners only, defined by having better able to self-regulate their behavior, with many learn-
spent at least five minutes in the platform. ers submitting their work well before the deadline, in con-
To explore whether our Learning Tracker widget had trast to the learners of the control group.
any effect on our learners, we ran a Mann-Whitney U test
(normal distributions not assumed) between the test (wid-
get shown) and control (widget not shown) groups. In all 600
Graded Quiz Submitters By Week
analyses that follow we set α = 0.05. Control
We perform the following analyses on the six dimensions Test
500
shown in the Learning Tracker. We find significant dif-
ferences between the two groups for the following two di- 400
mensions (and no sig. differences for the remaining four):
# Learners
• number of graded quiz answers submitted ; 300
• the timeliness of the quiz answer submission. 200
In Figure 6 we show the progression of both groups through
the course with respect to the number of learners that sub- 100
mitted answers to graded quiz questions. Consistently, a
larger number of learners in the test group submit their 0
1 2 3 4 5 6 7 8 9
work. By week 9, 34.12% (550/1612) of the active users Week #
in the test group submit graded quiz answers compared to
30.77% (485/1576) of learners in the control group. The dif-
ference between the groups becomes visible in week 3, a week Figure 6: The total number of learners, by course
after the first Learning Tracker widget was made available week, whose #quiz answers submitted > 0.
to the test group.
4
0.0045
Timeliness 0.025
Final Grades
Control Control
0.0040 Test Test
0.0035 0.020
0.0030
0.015
0.0025
Density
Density
0.0020
0.010
0.0015
0.0010 0.005
0.0005
0.0000 0.000
0 50 100 150 200 250 300 0 20 40 60 80 100
Time Ahead of Deadline (hours) Final grade
Figure 7: Kernel Density Estimation (Gaussian ker- Figure 9: Kernel Density Estimation (Gaussian ker-
nel) plot visualizing how far ahead of the dead- nel) plot visualizing the distribution of final course
line learners in each group typically submitted their grades between the two groups. Differences are sig-
weekly quiz answers. The left side of the plot is nificant at the α = 0.05 level.
indicative of procrastinating behavior, whereas the
right side indicates proactivity. Differences are sig-
nificant at the α = 0.05 level. we developed and deployed a Learning Tracker widget in
an edX MOOC with more than 10,000 learners. In an A/B
test setup, we found our widget to significantly increase
0.06
Graded Quiz Submissions Overall learner success (in terms of final grade) and two of the six
Control specified measures of learner engagement (specifically, more
Test timely assignment submissions and more assignment sub-
0.05
missions overall). We conclude that a dashboard like ours
enables learners to better self-regulate their learning behav-
0.04
ior based on a concrete anchor point for comparison (the
Density
successful learners of the past).
0.03 In future work, we plan to expand our experiments across
a number of MOOCs and a number of different Learn-
0.02 ing Tracker designs with different levels of granularity and
study dimensions to answer the following research questions:
0.01
• Can data visualization feedback elicit positive change
0.00 in MOOC learners’ study habits?
0 5 10 15 20 25
Graded Quiz Submissions • How literate are learners of this type of feedback? Are
they able to draw their own insights from simple data
Figure 8: Kernel Density Estimation (Gaussian ker- visualizations?
nel) plot visualizing the distribution of the num-
ber of submitted graded quiz answers for learners References
in each group. Differences are significant at the
[1] Scott Bateman, Jaime Teevan, and Ryen W. White.
α = 0.05 level.
The Search Dashboard: How Reflection and Compar-
ison Impact Search Behavior. In Proceedings of the
Lastly, there are differences in the percentage of passing SIGCHI Conference on Human Factors in Computing
learners per group as well: 13.17% among active learners Systems. ACM, 2012.
in the test group compared to 11.35% in the control group. [2] Pierre Dillenbourg, Guillaume Zufferey, Hamed Alavi,
According to another Mann-Whitney U test, the differences Patrick Jermann, Son Do-Lenh, and Quentin Bonnard.
between the final grades attained by the active learners in Classroom Orchestration: The Third Circle of Usabil-
both groups are statistically significant with means of ity. CSCL2011 Proceedings, pages 510–517, 2011.
14.4 for the test group and 12.7 for the control group. In
Figure 9 we plot the distributions of the final course grades; [3] Eva Durall and Begoña Gros. Learning Analytics as
the test group exhibits a consistent, positive shift in grade a Metacognitive Tool. In Proceedings of 6th Interna-
compared to the control group. tional Conference on Computer Supported Education
(CSEDU), pages 380–384, 2014.
6. CONCLUSION & FUTURE WORK [4] Philip J. Guo and Katharina Reinecke. Demographic
We have described in this paper work-in-progress in which Differences in how Students Navigate through MOOCs.
5
In Proceedings of the first ACM conference on Learn- Proceedings of the European MOOC Stakeholders Sum-
ing@ scale conference, pages 21–30. ACM, 2014. mit. Lausan: PAU Education, pages 140–146, 2014.
[5] Jeffrey Heer and Maneesh Agrawala. Design Consider- [12] George Siemens. Personal Learner Knowledge Graph.
ations for Collaborative Visual Analytics. Information In ELEARNSPACE, 2014.
visualization, 7(1):49–62, 2008.
[13] George Siemens. Personal Learning Graphs (PLeG). In
[6] Fred S Keller. Good-bye, Teacher... Journal of applied
ELEARNSPACE, 2015.
behavior analysis, 1(1):79, 1968.
[7] Malcolm S Knowles. Self-Directed Learning: A Guide [14] George Siemens, Dragan Gasevic, Caroline Haythorn-
for Learners and Teachers. 1975. thwaite, Shane Dawson, S Buckingham Shum, Re-
becca Ferguson, Erik Duval, Katrien Verbert, and Ryan
[8] Daphne Koller, Andrew Ng, Chuong Do, and Zhenghao Baker. Open Learning Analytics: an Integrated & Mod-
Chen. Retention and Intention in Massive Open Online ularized Platform. Proposal to design, implement and
Courses. Educause Review, 48(3), 2013. evaluate an open platform to integrate heterogeneous
learning analytics techniques, 2011.
[9] Eitel JM Laurı́a, Joshua D Baron, Mallika De-
vireddy, Venniraiselvi Sundararaju, and Sandeep M
[15] Kenneth David Strang. Do the Critical Success Factors
Jayaprakash. Mining Academic Data to Improve Col-
From Learning Analytics Predict Student Outcomes?
lege Student Retention: An Open Source Perspective.
Journal of Educational Technology Systems, 44(3):273–
In Proceedings of the 2nd International Conference on
299, 2016.
Learning Analytics and Knowledge. ACM, 2012.
[10] Christoph Meinel, Christian Willems, and [16] Katrien Verbert, Erik Duval, Joris Klerkx, Sten Gov-
JR Thomas Staubitz. Reflections on Enrollment aerts, and José Luis Santos. Learning Analytics Dash-
Numbers and Success Rates at the Openhpi MOOC board Applications. American Behavioral Scientist,
Platform. Proceedings of the European MOOC 2013.
Stakeholder Summit, pages 101–106, 2014.
[17] Bilal Yousuf and Owen Conlan. VisEN: Motivating
[11] Daniel T Seaton, Sergiy Nesterko, Tommy Mullaney, Learner Engagement Through Explorable Visual Nar-
Justin Reich, Andrew Ho, and Isaac Chuang. Charac- ratives. In Design for Teaching and Learning in a Net-
terizing Video Use in the Catalogue of MITx MOOCs. worked World, pages 367–380. Springer, 2015.
6