Encouraging Metacognition & Self-Regulation in MOOCs through Increased Learner Feedback Demonstration Dan Davis∗, Guanliang Chen†, Ioana Jivet, Claudia Hauff and Geert-Jan Houben Delft University of Technology Delft, the Netherlands {d.j.davis, guanliang.chen, i.jivet, c.hauff, g.j.p.m.houben}@tudelft.nl ABSTRACT Learning analytics for learners has the ability to greatly im- prove learners’ self-regulation. Current learner dashboards are mostly providing learners with an isolated view of their learning behavior, while we believe learners will gain more from a comparison of their own behavior with that of suc- cessful peer learners. In this work-in-progress demonstration we describe our design of a Learning Tracker widget that provides MOOC learners with timely and goal-oriented (i.e. towards passing the course) feedback in a manner that en- courages reflection and self-regulation. We also present some preliminary findings which show how exposure to feedback Figure 1: Sample edX dashboard: shows individual can significantly increase student success and engagement. students’ weekly & total assessment scores Keywords Learner Feedback, Learning Analytics, Self-Regulated Learn- ing, Study Planning 1. INTRODUCTION The asynchronous, open nature of MOOCs presents stu- dents with a profound sense of flexibility and freedom in their learning experience compared to the traditional class- room setting. They may study what they want, where they want, and whenever they want. However, along with these Figure 2: Sample Coursera dashboard: shows indi- ostensibly-positive affordances come major challenges. In vidual students’ grade for each quiz, whether or not order to be successful in such a learning environment—with they passed, and the total number of quizzes passed no pressure from teachers/parents, no financial obligations, and no academic credit on the line—students must stay in- credibly disciplined in both the planning and following of flow ecosystem. We believe that MOOC learners can sig- their study habits. Dropout rates of around 95% in the av- nificantly benefit from a timely and goal-oriented feedback erage MOOC [8] are a testimony to the challenge learners of their study habits in MOOCs. Currently, major MOOC face in this environment. platforms provide rather generic learner feedback as seen in The discipline for planning and following a self-imposed Figure 1 and Figure 2, which – while being timely – does schedule does not come naturally to many learners; rather not enable learners to judge their learning behavior in abso- it is a learned skill. And while merely releasing open edu- lute terms: are they on track to succeed in (i.e. pass) this cational resources to the world for consumption is a great course? Are they nearly on track? Are they missing a key start, the next step in the Open Learning movement ought ingredient to being successful? to equip learners with the cognitive toolset they need to ef- We believe that instead of providing a general overview of fectively self-regulate their learning experience. learner behavior, learners will be able to self-regulate better Currently, universities, instructors, and researchers are if we provide them with a comparison of their own learning the chief handlers of educational data generated from MOOCs. behavior against that of previous successful (in the sense Learners do not yet form an important part of this data that they passed the course) students. We have developed a ∗ The author’s research is supported by the Leiden-Delft- first learner widget that reflects this vision, enabling learners Erasmus Centre for Education and Learning. to compare themselves to successful learners and thus em- † powering them to reflect on and adapt their study behavior The author’s research is supported by the Extension School of the Delft University of Technology. in a goal-oriented fashion. 1 Copyright © 2016 for the individual papers by the papers' authors. Copying permitted only for private and academic purposes. This volume is published and copyrighted by its editors. LAL 2016 workshop at LAK '16, April 26, 2016, Edinburgh, Scotland. Not only does this ease the burden of instructors (the 2.3 Increasing Learner Efficiency learners decide how to react to and interpret the information Guo and Reinecke [4] studied to what extent students in shown to them), it also creates a heightened awareness in MOOCs access the full offering of learning materials. Sam- learners that they can keep with them beyond just this one pling from four edX MOOCs, they found that, on average, course and apply in future professional or academic contexts. certificate-earning students do not access, or “ignore,” 22% The following research question guides our line of inquiry of course materials [4]. Although instructors and instruc- into the topic: tional designers may not be too pleased by this finding, it has Can a comparison to previously successful learners serve as the potential to make future students more efficient in their a helpful form of feedback to increase MOOC learners’ en- learning. If there is certain content that students repeatedly gagement and success? skip without having their grade suffer, future students— In this paper, we describe our prototype widget, the design maybe low on time or extrinsically motivated—can refine decisions behind it, the setup we are currently employing in their learning plan based on this information. our experiments, and a preliminary analysis of the results. We find that indeed, our implemented feedback has a sig- 2.4 Open Learning Analytics nificant positive effect on the success of the learners (in terms The Learning Tracker realizes much of the personal-level, of grading) as well as on two out of six evaluated engagement student-facing dashboard envisioned in [14]. Along with the metrics. three other views (educator, researcher, and institutional), [14] proposes a dashboard in which students can see metrics ranging from progress compared to current peers, previous 2. RELATED WORK students who took the course, their own past activities, and instructor-defined benchmarks. Siemens et al. [14] here also 2.1 Search Dashboard suggest multiple levels of the dashboard, such as options for The main inspiration for this research comes from Bate- “drilling down” into more detailed data visualisations. man et al. [1] who, in the context of Web search, created Siemens [12] calls for Personal Learner Knowledge Graphs a “Search Dashboard” that provides an interface for search to boost awareness of a student’s own current knowledge engine users to see and reflect on their search behavior and, state in a given topic. This idea then evolved into Per- furthermore, a comparison of this data against “archetypal sonal Learning Graphs [13], which stress the “importance of expert profiles.” Their approach is very similar to ours in individuals owning their own learning representation” [13]. that they outline searching as something people have come While the present Learning Tracker widget is not owned to depend on in every day life, but rarely do people consider by the learner, it empowers MOOC students to assume a searching as a skill that may be developed and improved. more active role in shaping their own learning experience. The same can be said about learning. Bateman et al. [1] To our knowledge, these remain undeveloped and only con- found that people rarely change their search behavior no ceptualisations of what dashboards should be. matter the situation, but, once exposed to the dashboard, they become more active, aware, and critical of their search- 2.5 Dashboards as Explorable Visual Narra- ing habits—adapting them to be more in line with those of tives the visualized expert searchers. To see if learner feedback data visualisations can elicit change in student behavior (similar to our research ques- 2.2 Feedback to Encourage Metacognition tion), Yousuf and Conlan [17] implemented a dashboard The key processes underpinning our Learning Tracker (VisEN) that intended to emphasize to the student viewers widget are that of (i) feedback prompting and (ii) metacog- a sort of “visual narrative” in the form of data visualiza- nition, which then results in (iii) more effective self-regulated tions. There is no text-based narrative provided for the stu- (or self-directed) [7] learning. Durall and Gros [3] and Ver- dents; rather, this dashboard, pictured in Figure 3, consists bert et al. [16] also outline this process of providing stu- of heavily-annotated data visualizations from which students dents with “self-knowledge” as being key to developing the were expected to draw their own narrative arc. Findings necessary metacognitive skills for self-regulated (or directed) from their three studies, taking place over three years and in- learning. And in order to ease the translation from data to cluding 223 students, yielded a very strong Pearson correla- actionable knowledge, Heer and Agrawala [5] found informa- tion coefficient between dashboard views and learner engage- tion visualization to be an effective sense-making tool due ment [17]. A fundamental difference between this approach to its ability to synthesize complex data in a way for viewers and our Learning Tracker presented here is that VisEN in- to quickly understand and compare. corporated tactics to directly encourage engagement such as An early example of instructions designed to empower reminders and “bad or poor engagement notifications”. Our learners to shape their own learning experience dates back to Learning Tracker, on the other hand, stops short of any 1965, where Keller [6] introduces and documents the result of direction-giving or motivation and merely presents the learn- a “go-at-your-own-pace” course. This resulted in an inverted ers with a comparative view of their own behavior and that (U-shaped), polarized (highest concentrations for Grades ’A’ of successful learners. Furthermore, the Learning Tracker and ’F’) grade distribution at the conclusion of the courses, dashboard operates at scale in MOOCs. making clear the difference between students who can and cannot self-regulate effectively. Increased learner feedback 3. WIDGET DESIGN and awareness could be the nudge some of these students Based on prior works [11, 10, 9, 15] that have investigated need to remain engaged and pass the course. the factors impacting learner success in MOOCs and effec- tive feedback strategies (such as the “simple design prin- ciples” outlined in [2]) and some subjective judgement on 2 among the middle 90% of gold standard learners—thus the value of the outer ring increases each week, and the zero point remains constant. We aim to ensure that any actions learners take in re- sponse to the widget are self-conceived. In order to do so, we try to minimize any feelings of external judgment or assess- ment from the visualisations by making them as “modest” as possible—“simply making things visible that would other- wise remain invisible” [2]. There are no red “danger zones” or green “in-the-clear zones” on the chart as are found in other learning dashboards such as VisEN [17] or Coursera (Figure 2). Rather, we present a chart free of not only zones, but also any numbers, similar to the “degraded information” concept in [2]. All students see is their relative position com- pared to the set of successful learners on the same plane. It is left up to the learners how to interpret it, what to learn from it, and how to convert this information into actionable knowledge. 4. EXPERIMENTAL SETUP We deployed our widget in the TU Delft MOOC Intro- duction to Drinking Water Treatment running in its second edition on the edX platform between January 12 and March 29, 2016 (11 course weeks in total). The first iteration of Figure 3: Selected visualisations from a VisEN stu- the MOOC ran in 2014, with 10,695 registered learners of dent’s visual narrative whom 281 (2.6%) earned a passing grade. For this year’s edition 10,943 users enrolled before the of- our behalf, we identified six indicators that are related to ficial start of the course and, in turn, participated in our learner success and at the same time readily understandable experiment. Using A/B testing, we presented the Learn- to learners: ing Tracker widget to 49.91% (5,462) of the learners. At • Time on the platform in seconds the start of every course week, the learners were shown on • Time watching videos in seconds the course page how they compared (up to that point in the course) to our gold standard learners from last year. • Fraction of time spent watching videos while on the platform: whereas the previous two give total time commitment mea- Alongside the visualisations (concrete examples of which are sures, this provides feedback on how they allocate their time shown in Figures 4 and 5) we also provided a short explana- in the course tory text that included the following statement: • Number of course videos watched These graphs are not meant to be judgements or • Number of graded quiz answers submitted assessments of your learning in any way; rather, • Timeliness of quiz answer submission: how early students sub- they are a source of feedback for you, the learner, mit answers relative to the deadline, to expose procrastinating to make you more aware of your study habits and, behavior hopefully, help you change them for the better! In order to enable learners to directly compare their be- havior to successful learners, we require a set of “gold stan- This 11-week course consists of one introduction week, five dard” successful learners. In MOOCs that are reruns (our weeks of content delivery, and two design assignments that target in this work) we can simply consider all learners that cover the remaining five weeks. The material published in passed one or more of the MOOC’s previous editions to make each content delivery week included an assignment with five up this set. These successful learners do not exhibit a uni- quiz questions. The video-lectures were complemented by a form behavior. However, if we consider the average or me- total of 63 practice quiz questions that were not graded. dian across all these learners for each indicator, we have a In order to graduate, learners had to earn a final grade relatively robust indicator. For each indicator, the values higher than 60. We observe an increase in the percentage of that fall in the bottom 5% and the top 5% of the data range certificate-earning learners compared to last year’s edition are omitted. of the MOOC: 3.18% (348 out of 10,943 learners). Having prototyped several different visualizations of our indicators (including bar charts, gauges and calendar charts), 5. RESULTS we settled on the use of a spider chart as shown in Figure 4. We now provide an overview of our preliminary findings. Spider charts allow for (i) a concise visualisation of numer- The results are based on all edX log traces up to and in- ous metrics in a small space, (ii) simple legibility—data are cluding week 9 of the MOOC1 . Due to the low number of shown as single points along straight lines, and (iii) a visual learners that visited the course material after the course depiction of one’s coverage and consistency across all met- started (3,787 - 34.6% of enrolled) and the high drop-out rics. To allow for a consistent representation in the same graph, all metric values are scaled in a range from 0 to 10, 1 The remaining course weeks are not included in the analy- where 0 indicates no activity and 10 the maximum value sis, as the log traces are not yet available. 3 Figure 4: Three versions of our Learning Tracker widget with data from week 9, one showing a learner who dropped out early in the course (left), one who dropped out in the middle of the course (middle), and one graduate who is highly engaged with the course (right). Figure 5: One learner’s widget for weeks 3 (left), 6 (middle) and 9 (right). These show that this particular learner was a late starter and began engaging with the course some time between weeks 3 and 6. rate in the first week of the course (19.26% of enrolled did In Figure 7 we present the timeliness of the two groups not return after week one), the data distribution is highly with respect to the weekly quiz deadlines: the test group is skewed. We analyzed active learners only, defined by having better able to self-regulate their behavior, with many learn- spent at least five minutes in the platform. ers submitting their work well before the deadline, in con- To explore whether our Learning Tracker widget had trast to the learners of the control group. any effect on our learners, we ran a Mann-Whitney U test (normal distributions not assumed) between the test (wid- get shown) and control (widget not shown) groups. In all 600 Graded Quiz Submitters By Week analyses that follow we set α = 0.05. Control We perform the following analyses on the six dimensions Test 500 shown in the Learning Tracker. We find significant dif- ferences between the two groups for the following two di- 400 mensions (and no sig. differences for the remaining four): # Learners • number of graded quiz answers submitted ; 300 • the timeliness of the quiz answer submission. 200 In Figure 6 we show the progression of both groups through the course with respect to the number of learners that sub- 100 mitted answers to graded quiz questions. Consistently, a larger number of learners in the test group submit their 0 1 2 3 4 5 6 7 8 9 work. By week 9, 34.12% (550/1612) of the active users Week # in the test group submit graded quiz answers compared to 30.77% (485/1576) of learners in the control group. The dif- ference between the groups becomes visible in week 3, a week Figure 6: The total number of learners, by course after the first Learning Tracker widget was made available week, whose #quiz answers submitted > 0. to the test group. 4 0.0045 Timeliness 0.025 Final Grades Control Control 0.0040 Test Test 0.0035 0.020 0.0030 0.015 0.0025 Density Density 0.0020 0.010 0.0015 0.0010 0.005 0.0005 0.0000 0.000 0 50 100 150 200 250 300 0 20 40 60 80 100 Time Ahead of Deadline (hours) Final grade Figure 7: Kernel Density Estimation (Gaussian ker- Figure 9: Kernel Density Estimation (Gaussian ker- nel) plot visualizing how far ahead of the dead- nel) plot visualizing the distribution of final course line learners in each group typically submitted their grades between the two groups. Differences are sig- weekly quiz answers. The left side of the plot is nificant at the α = 0.05 level. indicative of procrastinating behavior, whereas the right side indicates proactivity. Differences are sig- nificant at the α = 0.05 level. we developed and deployed a Learning Tracker widget in an edX MOOC with more than 10,000 learners. In an A/B test setup, we found our widget to significantly increase 0.06 Graded Quiz Submissions Overall learner success (in terms of final grade) and two of the six Control specified measures of learner engagement (specifically, more Test timely assignment submissions and more assignment sub- 0.05 missions overall). We conclude that a dashboard like ours enables learners to better self-regulate their learning behav- 0.04 ior based on a concrete anchor point for comparison (the Density successful learners of the past). 0.03 In future work, we plan to expand our experiments across a number of MOOCs and a number of different Learn- 0.02 ing Tracker designs with different levels of granularity and study dimensions to answer the following research questions: 0.01 • Can data visualization feedback elicit positive change 0.00 in MOOC learners’ study habits? 0 5 10 15 20 25 Graded Quiz Submissions • How literate are learners of this type of feedback? Are they able to draw their own insights from simple data Figure 8: Kernel Density Estimation (Gaussian ker- visualizations? nel) plot visualizing the distribution of the num- ber of submitted graded quiz answers for learners References in each group. Differences are significant at the [1] Scott Bateman, Jaime Teevan, and Ryen W. White. α = 0.05 level. The Search Dashboard: How Reflection and Compar- ison Impact Search Behavior. In Proceedings of the Lastly, there are differences in the percentage of passing SIGCHI Conference on Human Factors in Computing learners per group as well: 13.17% among active learners Systems. ACM, 2012. in the test group compared to 11.35% in the control group. [2] Pierre Dillenbourg, Guillaume Zufferey, Hamed Alavi, According to another Mann-Whitney U test, the differences Patrick Jermann, Son Do-Lenh, and Quentin Bonnard. between the final grades attained by the active learners in Classroom Orchestration: The Third Circle of Usabil- both groups are statistically significant with means of ity. CSCL2011 Proceedings, pages 510–517, 2011. 14.4 for the test group and 12.7 for the control group. In Figure 9 we plot the distributions of the final course grades; [3] Eva Durall and Begoña Gros. Learning Analytics as the test group exhibits a consistent, positive shift in grade a Metacognitive Tool. In Proceedings of 6th Interna- compared to the control group. tional Conference on Computer Supported Education (CSEDU), pages 380–384, 2014. 6. CONCLUSION & FUTURE WORK [4] Philip J. Guo and Katharina Reinecke. Demographic We have described in this paper work-in-progress in which Differences in how Students Navigate through MOOCs. 5 In Proceedings of the first ACM conference on Learn- Proceedings of the European MOOC Stakeholders Sum- ing@ scale conference, pages 21–30. ACM, 2014. mit. Lausan: PAU Education, pages 140–146, 2014. [5] Jeffrey Heer and Maneesh Agrawala. Design Consider- [12] George Siemens. Personal Learner Knowledge Graph. ations for Collaborative Visual Analytics. Information In ELEARNSPACE, 2014. visualization, 7(1):49–62, 2008. [13] George Siemens. Personal Learning Graphs (PLeG). In [6] Fred S Keller. Good-bye, Teacher... Journal of applied ELEARNSPACE, 2015. behavior analysis, 1(1):79, 1968. [7] Malcolm S Knowles. Self-Directed Learning: A Guide [14] George Siemens, Dragan Gasevic, Caroline Haythorn- for Learners and Teachers. 1975. thwaite, Shane Dawson, S Buckingham Shum, Re- becca Ferguson, Erik Duval, Katrien Verbert, and Ryan [8] Daphne Koller, Andrew Ng, Chuong Do, and Zhenghao Baker. Open Learning Analytics: an Integrated & Mod- Chen. Retention and Intention in Massive Open Online ularized Platform. Proposal to design, implement and Courses. Educause Review, 48(3), 2013. evaluate an open platform to integrate heterogeneous learning analytics techniques, 2011. [9] Eitel JM Laurı́a, Joshua D Baron, Mallika De- vireddy, Venniraiselvi Sundararaju, and Sandeep M [15] Kenneth David Strang. Do the Critical Success Factors Jayaprakash. Mining Academic Data to Improve Col- From Learning Analytics Predict Student Outcomes? lege Student Retention: An Open Source Perspective. Journal of Educational Technology Systems, 44(3):273– In Proceedings of the 2nd International Conference on 299, 2016. Learning Analytics and Knowledge. ACM, 2012. [10] Christoph Meinel, Christian Willems, and [16] Katrien Verbert, Erik Duval, Joris Klerkx, Sten Gov- JR Thomas Staubitz. Reflections on Enrollment aerts, and José Luis Santos. Learning Analytics Dash- Numbers and Success Rates at the Openhpi MOOC board Applications. American Behavioral Scientist, Platform. Proceedings of the European MOOC 2013. Stakeholder Summit, pages 101–106, 2014. [17] Bilal Yousuf and Owen Conlan. VisEN: Motivating [11] Daniel T Seaton, Sergiy Nesterko, Tommy Mullaney, Learner Engagement Through Explorable Visual Nar- Justin Reich, Andrew Ho, and Isaac Chuang. Charac- ratives. In Design for Teaching and Learning in a Net- terizing Video Use in the Catalogue of MITx MOOCs. worked World, pages 367–380. Springer, 2015. 6