Encouraging Metacognition & Self-Regulation in MOOCs through Increased Learner Feedback

Abstract

Learning analytics for learners has the ability to greatly improve learners' self-regulation. Current learner dashboards are mostly providing learners with an isolated view of their learning behavior, while we believe learners will gain more from a comparison of their own behavior with that of successful peer learners. In this work-in-progress demonstration we describe our design of a Learning Tracker widget that provides MOOC learners with timely and goal-oriented (i.e. towards passing the course) feedback in a manner that encourages re ection and self-regulation. We also present some preliminary ndings which show how exposure to feedback can signi cantly increase student success and engagement.

1. INTRODUCTION

The asynchronous, open nature of MOOCs presents students with a profound sense of exibility and freedom in their learning experience compared to the traditional classroom setting. They may study what they want, where they want, and whenever they want. However, along with these ostensibly-positive a ordances come major challenges. In order to be successful in such a learning environment|with no pressure from teachers/parents, no nancial obligations, and no academic credit on the line|students must stay incredibly disciplined in both the planning and following of their study habits. Dropout rates of around 95% in the average MOOC [ 8 ] are a testimony to the challenge learners face in this environment.

The discipline for planning and following a self-imposed schedule does not come naturally to many learners; rather it is a learned skill. And while merely releasing open educational resources to the world for consumption is a great start, the next step in the Open Learning movement ought to equip learners with the cognitive toolset they need to effectively self-regulate their learning experience.

Currently, universities, instructors, and researchers are the chief handlers of educational data generated from MOOCs. Learners do not yet form an important part of this data The author's research is supported by the Leiden-DelftErasmus Centre for Education and Learning. yThe author's research is supported by the Extension School of the Delft University of Technology. ow ecosystem. We believe that MOOC learners can signi cantly bene t from a timely and goal-oriented feedback of their study habits in MOOCs. Currently, major MOOC platforms provide rather generic learner feedback as seen in Figure 1 and Figure 2, which { while being timely { does not enable learners to judge their learning behavior in absolute terms: are they on track to succeed in (i.e. pass) this course? Are they nearly on track? Are they missing a key ingredient to being successful?

We believe that instead of providing a general overview of learner behavior, learners will be able to self-regulate better if we provide them with a comparison of their own learning behavior against that of previous successful (in the sense that they passed the course) students. We have developed a rst learner widget that re ects this vision, enabling learners to compare themselves to successful learners and thus empowering them to re ect on and adapt their study behavior in a goal-oriented fashion.

Not only does this ease the burden of instructors (the learners decide how to react to and interpret the information shown to them), it also creates a heightened awareness in learners that they can keep with them beyond just this one course and apply in future professional or academic contexts.

The following research question guides our line of inquiry into the topic: Can a comparison to previously successful learners serve as a helpful form of feedback to increase MOOC learners' engagement and success? In this paper, we describe our prototype widget, the design decisions behind it, the setup we are currently employing in our experiments, and a preliminary analysis of the results.

We nd that indeed, our implemented feedback has a signi cant positive e ect on the success of the learners (in terms of grading) as well as on two out of six evaluated engagement metrics.

RELATED WORK

Search Dashboard

The main inspiration for this research comes from Bateman et al. [ 1 ] who, in the context of Web search, created a \Search Dashboard" that provides an interface for search engine users to see and re ect on their search behavior and, furthermore, a comparison of this data against \archetypal expert pro les." Their approach is very similar to ours in that they outline searching as something people have come to depend on in every day life, but rarely do people consider searching as a skill that may be developed and improved. The same can be said about learning. Bateman et al. [ 1 ] found that people rarely change their search behavior no matter the situation, but, once exposed to the dashboard, they become more active, aware, and critical of their searching habits|adapting them to be more in line with those of the visualized expert searchers. 2.2

Feedback to Encourage Metacognition

The key processes underpinning our Learning Tracker widget are that of (i) feedback prompting and (ii) metacognition, which then results in (iii) more e ective self-regulated (or self-directed) [ 7 ] learning. Durall and Gros [ 3 ] and Verbert et al. [ 16 ] also outline this process of providing students with \self-knowledge" as being key to developing the necessary metacognitive skills for self-regulated (or directed) learning. And in order to ease the translation from data to actionable knowledge, Heer and Agrawala [ 5 ] found information visualization to be an e ective sense-making tool due to its ability to synthesize complex data in a way for viewers to quickly understand and compare.

An early example of instructions designed to empower learners to shape their own learning experience dates back to 1965, where Keller [ 6 ] introduces and documents the result of a \go-at-your-own-pace" course. This resulted in an inverted (U-shaped), polarized (highest concentrations for Grades 'A' and 'F') grade distribution at the conclusion of the courses, making clear the di erence between students who can and cannot self-regulate e ectively. Increased learner feedback and awareness could be the nudge some of these students need to remain engaged and pass the course. 2.3

Increasing Learner Efficiency

Guo and Reinecke [ 4 ] studied to what extent students in MOOCs access the full o ering of learning materials. Sampling from four edX MOOCs, they found that, on average, certi cate-earning students do not access, or \ignore," 22% of course materials [ 4 ]. Although instructors and instructional designers may not be too pleased by this nding, it has the potential to make future students more e cient in their learning. If there is certain content that students repeatedly skip without having their grade su er, future students| maybe low on time or extrinsically motivated|can re ne their learning plan based on this information. 2.4

Open Learning Analytics

The Learning Tracker realizes much of the personal-level, student-facing dashboard envisioned in [ 14 ]. Along with the three other views (educator, researcher, and institutional), [ 14 ] proposes a dashboard in which students can see metrics ranging from progress compared to current peers, previous students who took the course, their own past activities, and instructor-de ned benchmarks. Siemens et al. [ 14 ] here also suggest multiple levels of the dashboard, such as options for \drilling down" into more detailed data visualisations.

Siemens [ 12 ] calls for Personal Learner Knowledge Graphs to boost awareness of a student's own current knowledge state in a given topic. This idea then evolved into Personal Learning Graphs [ 13 ], which stress the \importance of individuals owning their own learning representation" [ 13 ]. While the present Learning Tracker widget is not owned by the learner, it empowers MOOC students to assume a more active role in shaping their own learning experience. To our knowledge, these remain undeveloped and only conceptualisations of what dashboards should be. 2.5

Dashboards as Explorable Visual Narratives

To see if learner feedback data visualisations can elicit change in student behavior (similar to our research question), Yousuf and Conlan [ 17 ] implemented a dashboard (VisEN) that intended to emphasize to the student viewers a sort of \visual narrative" in the form of data visualizations. There is no text-based narrative provided for the students; rather, this dashboard, pictured in Figure 3, consists of heavily-annotated data visualizations from which students were expected to draw their own narrative arc. Findings from their three studies, taking place over three years and including 223 students, yielded a very strong Pearson correlation coe cient between dashboard views and learner engagement [ 17 ]. A fundamental di erence between this approach and our Learning Tracker presented here is that VisEN incorporated tactics to directly encourage engagement such as reminders and \bad or poor engagement noti cations". Our Learning Tracker, on the other hand, stops short of any direction-giving or motivation and merely presents the learners with a comparative view of their own behavior and that of successful learners. Furthermore, the Learning Tracker dashboard operates at scale in MOOCs.

3. WIDGET DESIGN

Based on prior works [ 11, 10, 9, 15 ] that have investigated the factors impacting learner success in MOOCs and e ective feedback strategies (such as the \simple design principles" outlined in [ 2 ]) and some subjective judgement on our behalf, we identi ed six indicators that are related to learner success and at the same time readily understandable to learners:

Time on the platform in seconds

Time watching videos in seconds

Fraction of time spent watching videos while on the platform: whereas the previous two give total time commitment measures, this provides feedback on how they allocate their time in the course

Number of course videos watched

Number of graded quiz answers submitted

Timeliness of quiz answer submission: how early students submit answers relative to the deadline, to expose procrastinating behavior

In order to enable learners to directly compare their behavior to successful learners, we require a set of \gold standard" successful learners. In MOOCs that are reruns (our target in this work) we can simply consider all learners that passed one or more of the MOOC's previous editions to make up this set. These successful learners do not exhibit a uniform behavior. However, if we consider the average or median across all these learners for each indicator, we have a relatively robust indicator. For each indicator, the values that fall in the bottom 5% and the top 5% of the data range are omitted.

Having prototyped several di erent visualizations of our indicators (including bar charts, gauges and calendar charts), we settled on the use of a spider chart as shown in Figure 4. Spider charts allow for (i) a concise visualisation of numerous metrics in a small space, (ii) simple legibility|data are shown as single points along straight lines, and (iii) a visual depiction of one's coverage and consistency across all metrics. To allow for a consistent representation in the same graph, all metric values are scaled in a range from 0 to 10, where 0 indicates no activity and 10 the maximum value among the middle 90% of gold standard learners|thus the value of the outer ring increases each week, and the zero point remains constant.

We aim to ensure that any actions learners take in response to the widget are self-conceived. In order to do so, we try to minimize any feelings of external judgment or assessment from the visualisations by making them as \modest" as possible|\simply making things visible that would otherwise remain invisible" [ 2 ]. There are no red \danger zones" or green \in-the-clear zones" on the chart as are found in other learning dashboards such as VisEN [ 17 ] or Coursera (Figure 2). Rather, we present a chart free of not only zones, but also any numbers, similar to the \degraded information" concept in [ 2 ]. All students see is their relative position compared to the set of successful learners on the same plane. It is left up to the learners how to interpret it, what to learn from it, and how to convert this information into actionable knowledge. 4.

EXPERIMENTAL SETUP

We deployed our widget in the TU Delft MOOC Introduction to Drinking Water Treatment running in its second edition on the edX platform between January 12 and March 29, 2016 (11 course weeks in total). The rst iteration of the MOOC ran in 2014, with 10,695 registered learners of whom 281 (2.6%) earned a passing grade.

For this year's edition 10,943 users enrolled before the ofcial start of the course and, in turn, participated in our experiment. Using A/B testing, we presented the Learning Tracker widget to 49.91% (5,462) of the learners. At the start of every course week, the learners were shown on the course page how they compared (up to that point in the course) to our gold standard learners from last year. Alongside the visualisations (concrete examples of which are shown in Figures 4 and 5) we also provided a short explanatory text that included the following statement: These graphs are not meant to be judgements or assessments of your learning in any way; rather, they are a source of feedback for you, the learner, to make you more aware of your study habits and, hopefully, help you change them for the better!

This 11-week course consists of one introduction week, ve weeks of content delivery, and two design assignments that cover the remaining ve weeks. The material published in each content delivery week included an assignment with ve quiz questions. The video-lectures were complemented by a total of 63 practice quiz questions that were not graded. In order to graduate, learners had to earn a nal grade higher than 60. We observe an increase in the percentage of certi cate-earning learners compared to last year's edition of the MOOC: 3.18% (348 out of 10,943 learners). 5.

RESULTS

We now provide an overview of our preliminary ndings. The results are based on all edX log traces up to and including week 9 of the MOOC1. Due to the low number of learners that visited the course material after the course started (3,787 - 34.6% of enrolled) and the high drop-out 1The remaining course weeks are not included in the analysis, as the log traces are not yet available. rate in the rst week of the course (19.26% of enrolled did not return after week one), the data distribution is highly skewed. We analyzed active learners only, de ned by having spent at least ve minutes in the platform.

To explore whether our Learning Tracker widget had any e ect on our learners, we ran a Mann-Whitney U test (normal distributions not assumed) between the test (widget shown) and control (widget not shown) groups. In all analyses that follow we set = 0:05.

We perform the following analyses on the six dimensions shown in the Learning Tracker. We nd signi cant differences between the two groups for the following two dimensions (and no sig. di erences for the remaining four): number of graded quiz answers submitted ; the timeliness of the quiz answer submission.

In Figure 6 we show the progression of both groups through the course with respect to the number of learners that submitted answers to graded quiz questions. Consistently, a larger number of learners in the test group submit their work. By week 9, 34.12% (550/1612) of the active users in the test group submit graded quiz answers compared to 30.77% (485/1576) of learners in the control group. The difference between the groups becomes visible in week 3, a week after the rst Learning Tracker widget was made available to the test group.

In Figure 7 we present the timeliness of the two groups with respect to the weekly quiz deadlines: the test group is better able to self-regulate their behavior, with many learners submitting their work well before the deadline, in contrast to the learners of the control group.

Graded Quiz Submitters By Week Control

Test 600 500 400 s r e rna300 e L #200 100 0 1

5

Week # 2 3 4 6 7 8 9 0.0040 0.0035

Lastly, there are di erences in the percentage of passing learners per group as well: 13:17% among active learners in the test group compared to 11:35% in the control group. According to another Mann-Whitney U test, the di erences between the nal grades attained by the active learners in both groups are statistically signi cant with means of 14:4 for the test group and 12:7 for the control group. In Figure 9 we plot the distributions of the nal course grades; the test group exhibits a consistent, positive shift in grade compared to the control group.

CONCLUSION & FUTURE WORK

We have described in this paper work-in-progress in which 300 0 20 80

100 40 60 Final grade we developed and deployed a Learning Tracker widget in an edX MOOC with more than 10,000 learners. In an A/B test setup, we found our widget to signi cantly increase learner success (in terms of nal grade) and two of the six speci ed measures of learner engagement (speci cally, more timely assignment submissions and more assignment submissions overall). We conclude that a dashboard like ours enables learners to better self-regulate their learning behavior based on a concrete anchor point for comparison (the successful learners of the past).

In future work, we plan to expand our experiments across a number of MOOCs and a number of di erent Learning Tracker designs with di erent levels of granularity and study dimensions to answer the following research questions: Can data visualization feedback elicit positive change in MOOC learners' study habits? How literate are learners of this type of feedback? Are they able to draw their own insights from simple data visualizations?

References

[1] Scott Bateman , Jaime Teevan, and Ryen W. White. The Search Dashboard: How Re ection and Comparison Impact Search Behavior . In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM , 2012 .

[2] Pierre Dillenbourg , Guillaume Zu erey, Hamed Alavi, Patrick Jermann, Son Do-Lenh, and Quentin Bonnard . Classroom Orchestration: The Third Circle of Usability. CSCL2011 Proceedings , pages 510 { 517 , 2011 .

[3] Eva Durall and Begon~a Gros. Learning Analytics as a Metacognitive Tool . In Proceedings of 6th International Conference on Computer Supported Education (CSEDU) , pages 380 { 384 , 2014 .

[4] Philip J. Guo and Katharina Reinecke . Demographic Di erences in how Students Navigate through MOOCs .

In Proceedings of the rst ACM conference on Learning@ scale conference , pages 21 { 30 . ACM, 2014 .

Proceedings of the European MOOC Stakeholders Summit. Lausan: PAU Education , pages 140 { 146 , 2014 .

[5] Je rey Heer and Maneesh Agrawala. Design Considerations for Collaborative Visual Analytics . Information visualization , 7 ( 1 ): 49 { 62 , 2008 .

[6] Fred S Keller. Good-bye, Teacher... Journal of applied behavior analysis , 1 ( 1 ): 79 , 1968 .

[7] Malcolm S Knowles . Self-Directed Learning: A Guide for Learners and Teachers . 1975 .

[8] Daphne Koller , Andrew Ng, Chuong Do, and Zhenghao Chen . Retention and Intention in Massive Open Online Courses . Educause Review , 48 ( 3 ), 2013 .

[9] Eitel JM Laur a , Joshua D Baron , Mallika Devireddy, Venniraiselvi Sundararaju, and Sandeep M Jayaprakash. Mining Academic Data to Improve College Student Retention: An Open Source Perspective . In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge. ACM , 2012 .

[10] Christoph Meinel , Christian Willems , and JR Thomas Staubitz. Re ections on Enrollment Numbers and Success Rates at the Openhpi MOOC Platform . Proceedings of the European MOOC Stakeholder Summit , pages 101 { 106 , 2014 .

[11] Daniel T Seaton , Sergiy Nesterko, Tommy Mullaney, Justin Reich, Andrew Ho, and Isaac Chuang . Characterizing Video Use in the Catalogue of MITx MOOCs .

[12] George Siemens . Personal Learner Knowledge Graph . In ELEARNSPACE , 2014 .

[13] George Siemens . Personal Learning Graphs (PLeG) . In ELEARNSPACE , 2015 .

[14] George Siemens , Dragan Gasevic, Caroline Haythornthwaite, Shane Dawson, S Buckingham Shum , Rebecca Ferguson, Erik Duval, Katrien Verbert, and Ryan Baker . Open Learning Analytics: an Integrated & Modularized Platform . Proposal to design, implement and evaluate an open platform to integrate heterogeneous learning analytics techniques , 2011 .

[15] Kenneth David Strang . Do the Critical Success Factors From Learning Analytics Predict Student Outcomes? Journal of Educational Technology Systems , 44 ( 3 ): 273 { 299 , 2016 .

[16] Katrien Verbert , Erik Duval, Joris Klerkx, Sten Govaerts, and Jose Luis Santos. Learning Analytics Dashboard Applications . American Behavioral Scientist, 2013 .

[17] Bilal Yousuf and Owen Conlan. VisEN: Motivating Learner Engagement Through Explorable Visual Narratives. In Design for Teaching and Learning in a Networked World , pages 367 { 380 . Springer, 2015 .