Dynamic dashboard for educators and students in FutureLearn MOOCs: experiences and insights

Abstract

One of the differentiating aspects of the FutureLearn platform, compared with other MOOC providers such as Coursera and EdX, is the approach to data sharing with partners. This is grounded on the release of a small set of relatively simple source files, which can be downloaded and used as required by end users (e.g. educators, researchers and so on). This approach has both advantages and disadvantages. The major advantage is the simplicity; the most important drawback is the lack of an 'out-of-the-box' set of analytical representations which the end-user can use and digest to obtain immediate insights regarding their online course. In this paper, we discuss these aspects in more detail and document the approach adopted at UNSW Sydney, to use the data as released, and how we produced a set of analytical dashboards for educators and students. The architecture underpinning the dashboards built is explained with a link to a GitHub repository with more detailed information.

Keywords

• MOOCs visualization dashboard learning analytics FutureLearn

-

MOOCs have been around for several years and there are now platform providers hosting courses with a variety of educational designs reaching millions of learners each year [ 29 ]. With the establishment of this way of learning online, the increased availability of MOOC data can offer the opportunity to provide insights to educators and developers into learners’ behaviours, and empower learners to understand their patterns of engagement and performance through learning analytics [ 30 ]. The former allows exploring learning design at scale and has the potential to inform pedagogy. The latter can improve the learning experience and develop crucial metacognitive skills essential for self-directed and lifelong learners. As mentioned in [ 37 ], the field of learning analytics FutureLearn data: what we currently have, what we are learning and how it is demonstrating learning in MOOCs. Workshop at the 7th International Learning Analytics and Knowledge Conference. Simon Fraser University, Vancouver, Canada, 13-17 March 2017, p. 20-35.

Copyright © 2017 for the individual papers by the papers' authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors. has seen a push to move from descriptive analytics to analytics able to inform and direct practice [ 11,41,42 ]. This was also advocated by Gasevic and colleagues as a key area of further research in their review of research in MOOCs [ 16 ] and took centre stage in many presentations at LAK ’17 and in other Workshops about MOOCs presented in this volume [ 40 ].

There appears to be two critical problems hindering the application of learning analytic methods to support and shape pedagogy in MOOCs: 1) constraints of the platforms (e.g. tools and course design) and 2) the availability of data – when it is needed, by different stakeholders to gain a better understanding of learning and take action.

Looking at the wealth of research in MOOCs, a lot is done ‘post-hoc’, or after the course is completed, when the respective platforms release data for exploration. Furthermore, data is often restricted to individual institutions, limited by contractual agreements with platform providers, and this makes it very hard to draw generalizable conclusions. Nevertheless, research has been carried out using Coursera data [ 4,26 ] focusing predominantly on the dashboard offered to partners’ institutions [ 12 ]. In addition, the relative openness of EdX, allowed different teams to develop extensions/plugins to access and use analytics [ 8,25,28,32 ]. Yet, to date, the focus has been primarily on educators and administrators rather than students. 2

The UNSW way of ‘moocing’

UNSW entered the MOOC space with the specific intent to learn at scale and take this back to mainstream education. This meant that the key driver for the development of different MOOCs was research-oriented and experimental in nature. This afforded the use of multiple MOOC platforms, which allowed for different learning designs and social engagement structures to be implemented across the deployed course offerings. This approach has been documented in various outlets [ 5,6,15,24 ]. This shows that learning design took center focus, and with it, the teachers whom were part of the MOOCs were able to push the boundaries of the technology – which has oft been criticised for its inability to deliver beyond the hype of connectivist approaches.

Experimenting with multiple platforms, using very different philosophies has been an essential aspect of the process. To date, UNSW Sydney has delivered ten distinct courses in Coursera; twenty in FutureLearn; a handful in SmartSparrow (mostly closed, on-campus courses focussed on Engineering and Medicine) and a dozen courses in OpenLearning (on-campus courses, namely in computer science).

Another important aspect has been the intentional integration (with other tools via LTI) and differentiation of the offerings, based on learners’ own preferences (elements of personalisation at scale via interactive activities and adaptive tutorials). Examples of this approach are demonstrated in the ‘Learning To Teach Online’ MOOC in Coursera [ 23 ], featuring a system to recommend pathways through the material based on learner’s preferences and the grouping mechanism in the ‘Entrepreneurship’ course in Coursera evolved from successful on-campus blended experiments [ 2 ]. In FutureLearn, adaptivity and personalisation featured in the integration of LTI activities using SmartSparrow (‘Through Engineers’ eyes’) [ 14,18 ], and activities aimed to interact with participants in ‘Personalised Medicine’ and ‘Disability’ courses. All these, within the constraints and affordances of each MOOC platform, provide insights into the experimental approaches to course design taken.

Another element of experimentation was done with the actual development and funding of different courses. Multi-disciplinary teams have been used to support the academic leads (content experts). Driven by the idea of curriculum alignment [ 1 ] and the RASE model for representing elements of course design [ 6 ], the MOOC design process at UNSW focussed on ongoing conversations between educational developers and academic leads to match and fit technology and tools to their pedagogical intentions. Finally, experimentation was done with different levels of de-centralisation to support the planning, design and implementation of the various courses.

Figure 1 provides an overview of the familiar learning analytics cycle [ 7 ], which may be implemented to develop and evaluate a single course (in the middle), but the transition to mainstream is an essential component of the larger cycle situated at institutional level, in which the focus is not only the improvement of a single course, but a more systemic improvement across all educational provision from an institution.

Institutional level, QA/QE

Refine

Collect

Act

Report

Refine Collect

Course level in FL Act Report

Predict

Predict

Given the context, a natural extension of such course experimentation at scale has allowed us to explore certain areas with evidence-based approaches to, these are: 1) the evaluation of projects and programs related to learning and teaching [ 3,22 ]; 2) a desire to systematically implement approaches and systems aiming to enable the use of learning and teaching data to improve pedagogy and 3) bridging the gap between traditional academic development approaches and data-driven approaches with Learning Analytics and Educational data mining [ 38 ].

In this paper, we present the work we completed with the FutureLearn platform, and we make available our work efforts to aspiring and other course development teams.

Visual exploration as an aid to pedagogical design

As mentioned earlier, the use of dashboards to support sense-making from learning and teaching data is not new. There are several examples in which the use of visualisations provide a good starting point for discussing course design and student learning. Our specific focus was to offer tools for educators and educational designers, allowing the visual exploration of data to help understand the way in which learners engage with different elements of the course and provide valuable information to inform future course designs.

To achieve these goals, several streams of work were carried out, resulting in a systematic process based on aspects identified to be used starting points for development: • Collate questions from diverse stakeholders (including educators, developers, learning designers and academic managers • Come up with the definitions of key terms (i.e. – What is engagement? What do we mean with learner experience?) • Explore design variants (Coursera/FL dashboard examples) • Reduce to smaller components/simplify • Re-connect with academics and educational developers • Obtain feedback from learners, by getting them to use the dashboards • Refine and reiterate

At the conceptual level, defining what is intended with terms like engagement and the learning experience is essential to inform the process put in place to measure these constructs. Vigentini and Zhao [ 39 ] provide an overview of the evaluative tools used in MOOCs in their recent meta-analysis of several platforms.

At the practical level, previous research [ 19,31,36 ] identified five key areas which appear to direct the attention of educators and developers in MOOCs: 1) An overview of the course, 2) Who are the participants/learners, 3) How participants interact/engage with the material, 4) How participants interact in the forums, and 5) How participants perform in the course.

In the design of our dashboard around the FutureLearn Data, we took these areas into account as well as principles for dashboard design to showcase information to achieve one or more objectives [ 13 ]. It was important for us to keep the representations simple, avoid overwhelming users with information [ 5 ], thus we employed a minimalist design combined with appropriate structuring of information. 3.1

The purpose of dashboards Intuitively, dashboards are only useful if there are actionable insights. In this sense, data visualizations have often been used as tools to help teachers gain insight about how students engage with the content and resources provided in learning environments, [ 33,34 ], but can also be offered directly to students to help them reflect on their approach to learning [ 9,17 ]. Three areas or domains of representation have proven popular: 1) Learner activity, 2) Learner engagement, 3) Learner experience. Learner activity focuses solely on a descriptive level of what students do in the learning environment. The analysis of student engagement moves further, and attempts to make sense not only of what students do, but also question the effectiveness of activity (usually correlating activity with outcomes, exploring in detail – assessment and the way in which students communicate and interact with each other as well as the platform, or digging into why students behave in the way they do, or attempting to model behaviour by enriching the logs with learning and teaching metadata – for example about the course or learning design). The third level of analysis is an abstraction of what the students’ experience is like, usually emerging from surveys or what students voice about their learning experience (before, during or after taking part in the course).

All of these provide valuable insights in the effectiveness of learning design leading to more effective evaluation and informing redesign [ 14 ][Vulic], but, if done consistently and in a timely manner, can also provide useful information to monitor progress, and allow for much quicker interventions while the course is running. 3.2

Why another dashboard? Over the past year UNSW developed and delivered twenty MOOCs in the FutureLearn platform. This provided an exceptional opportunity to put into action the work done in this space, by using data as the course unfolded. As mentioned earlier, at this point in time, FutureLearn still does not to offer a fully-fledged dashboard to partner institutions, but rather simple to understand accessible files, served up on a daily basis, of which a static report of activity can be generated.

The granularity of the information provided is thus enough to provide some insights, without overwhelming stakeholders. Given the near real-time availability of data, this provides a major opportunity to respond to students’ engagement as a course unfolds. 4

A practical approach to dashboard development

As it was the case for the design and development of MOOCs, the data and evaluation team in the PVC Education portfolio at UNSW Sydney took an experimental stance, creating minimum-viable products and moved from exploration and representation (i.e. supporting sense making of learner activity) to the characterization of patterns related to engagement with elements of the course. In the Coursera dashboard [ 36 ], the focus was on reporting what students did, and this was organised under functional headings (i.e. the use of videos, content, forums/discussion, assessment/learning activities and evaluative activities such as polls and surveys).

As will be presented in the following sections the approach tackled four distinct problems/phases: 1) cater for different systems, different formats, different philosophies; 2) determine appropriate processes for designing dashboards; 3) identify core dashboard ‘building blocks’; and 4) develop and test through a structured development process (Figure 2). There were two fundamental drivers for the development: 1. Studying learning engagement focusing specifically on active participation, measured via proxies of attempts/completion of learning activities and the volume of discussion. Admittedly neither measures learning directly, but the characterisation of the patterns provide insights in the way that participants go about their learning. 2. Beginning to include more detail about the learning experience and what participants said about their motivations, quality and satisfaction when learning through MOOCs.

What we strived to achieve was to put analytics into practice and empower both students and educators with the information and knowledge derived from the data. Each MOOC published in the FutureLearn website is presented in a hierarchical structure with weeks, activities and steps (Fig. X). Steps contain different types of material and can be recognized by the label next to the step title. Typically, in the FL courses designed at UNSW, eight different step types were used: article, discussion, video, exercise, quiz, test, audio and LTI activity.

At the time of writing, FutureLearn provides eight separate data sources prepared as comma separated values (CSV) files. Table 1 describes each file with detail about purpose. The datasets provided in FutureLearn have three main limitations: 1) granularity of user activity (currently limited to the time of the first/last access rather than a full interaction log); 2) minimal contextual information (lack of metadata about the learning context, such as video interaction data), and 3) partial demographic information to understand learners (only about 10% of participants have chosen to share personal details in the platform). These limitations are the byproduct of FutureLearn’s choice to provide easy to access and stable datasets. Nevertheless, these sources provide an excellent starting point to demonstrate the use of analytics in action.

Weeks

Activity

Steps

Progress

Step type

Timeline Although each file can be used in isolation to answer particular types of questions, in order to gain deeper insights, the data required transformations and integration. File Enrolments Step Activity Comments Question Response Team Members Peer Review Assignment Peer Review Reviews Campaigns

The purpose of the file This file provides basic information regarding enrolled learners (and staff). It includes demographic information of learners derived from responses to FutureLearn’s ‘more-about-you’ survey, which captured gender, country, age range, highest education level, employment status, and employment area.

This file stores information regarding step activity from learners, e.g. the time when a step is first visited, and the last time a step is marked as completed. Information about learners’ contributions to the discussion in each step is noted. The file includes the full-text of comments, timestamped according to when the comment was made. Likes associated with comments are also stored.

This file holds information about the quiz activity of learners. It stores learners’ responses, whether correct or not, and when the quiz was answered.

Information about organization staff members including their FutureLearn ID and full names are stored in this file.

This file provides information regarding peer review assignments including when the assignment was first viewed, when it was submitted and the number of reviews associated with the assignment.

This file provides information about the reviewers on an assignment, including when the review was submitted, the reviewer’s ID and feedback for each of the assignment guidelines.

Information about the referral used to advertise a course is stored in this file, following the number of enrolments and active learners for each referral. 4.2

The technology stack Based on our previous experience in building Learning and Teaching dashboards with Coursera MOOC data where we choose Tableau cloud to present insights, [ 35,36 ], we quickly identified key differences in the process required to develop a FutureLearn dashboard. The choice of appropriate tools was driven by previous experience combined with a preference for wanting to experiment with different products and services to realize different perspectives.

In the case of Coursera data, despite being able to present information in a rich and interactive way, the solution adopted was fit for a scenario in which the data was available only at the end of the course. This meant that after the first preparation step was completed, a data analyst could build an entire dashboard based on the Tableau templates by simply adapting the dashboard to the new course by fitting the released dataset and thus authoring a dashboard to the cloud.

However, given that data from FutureLearn is available daily, one of the main challenges was to develop a sustainable, dynamic and near real-time dashboard. This meant the choice of an appropriate technology stack to make the update process regular and seamless. Additionally, in a similar way to the use of dashboard templates for Coursera, keeping the visuals up-to-date, meant a deciding on a series of re-usable building blocks – thus further allowed common questions to be answered.

In Coursera, the technology used focused on the preparation and pre-processing of the data, predominantly leveraging on Python to create a replicable processes and data products ready to be used in Tableau.

In FutureLearn, given the relative simplicity of the dataset provided, what we wanted to focus on was more of the analytical process and therefore shifted toward the use of R scripts in-conjunction with a Shiny Server for publishing of outputs developed across the analysis areas chosen. Fig. 3 provides an overview of the technologies adopted in the two cases. Notably the front-end technologies did not change drastically between the two implementations.

Coursera (sessional) - Csv, json, mysql - Python, sql - Html, angular js, tableau Coursera (on demand) - Docker, csv, postgres - Python, sql - Html, angular js, tableau

FutureLearn - Csv, - Python, R, sql - Html, bootstrap, Rstudio

server, shiny server, d3js

LTI integration A simple architecture was developed around a two-stage model: 1) data extraction and pre-processing and 2) dashboard development. This allowed automation of the download and preparation of data, where more time could be spent on the customization of dashboard views to enhance the end-user experience (note: the dashboards were accessible via the web, via authentication – password / LTI) (see Fig. 5). As noted earlier, the R platform was used as the analytical engine, with a cloud-based Shiny server with an Apache server to serve the web content.

For the R dashboard creation process, Python was used to pull the data daily (a scheduled job), transformations were applied, and outputs stored in cloud-based MySQL database. In the data extraction and pre-processing phase, Python scripts were used to automatically login into the FutureLearn platform and download all available files for each course. Each time new CSV files were downloaded, they are stored, named as per the course title, with previous CSV files archived. After downloading all course files and loading the data into the database, the pre-processing routines (written in R) prepared the data, transforming the source files into views for specific purposes. This step was essential to make the web data requests scalable at run-time by querying

FutureLearn CSV data sources

Data processing scripts

UNSW cloud AWS MySQL RDS AWS Linux with Apache and Shiny

Client-side

HTML Dashboards (LTI connector for auth) pre-computed information sets, keep the hosting server size minimal and also keeping costs down. Although there are several examples of the use of dashboards with educators, there is a lack of examples with on learners. Corrin et al. [ 9,10 ] discussed the issues and difficulties of exposing data to students, but others like Liu and Pardo [ 21 ], have demonstrated how careful selection of specific data representations for students can lead to action. Here we present our current work-in-progress efforts which have focused around two key questions: (1) what data is useful to students? and (2) which user interfaces are likely most effective for the type of data being presented?

Based on previous research and continuing conversation and engagement with students, we propose a framework for student dashboard elements in four areas: • Learning Communities: this includes an awareness of their social context, the nature/structure of interactions with others in this context and learning conversations; • Student Progress: focuses on an awareness of the learning space in which a student learns, and provides a sense of the learning activities required to achieve learning goals. This also means that students can obtain markers (feedback) to motivate and promote self-regulation (i.e. articulate their goals in the context and know where they are compared to others); • Student Performance: focuses on both formative and summative elements of the course providing an overview of strengths and weaknesses as well as clear references to other students’ performance. This feedback is essential to inform and provide evidence for action; and • Student Experience: this is harder to measure and in most cases, comes from surveys about learners’ satisfaction with different aspects of the course. Further, based on the previous work with educators’ dashboards we specifically focused on four elements of design: 1) Layout, 2) Chart types, 3) Chart features and 4) Other visual/functional features.

The code repository overview

The repository of our work-in-progress efforts is shared in GitHub under a AGPL GNU licence. This is intended to foster collaboration and provide benefits for both endusers and others who may want to contribute to developments of the dashboard.

In the repository, there are two separate streams/folders: A dashboard for educators/administrators and a dashboard for students. These are now presented. 5.1

A set of dashboard building blocks

In this section, we provide an overview of the dashboard design and the visual building blocks used. At this point in time, the dashboard design is based on a minimalist html design template which dynamically loads the shiny dashboard applications to an iframe. An alternative to this is to use a native Shiny Dashboard ‘fluid page’, but this would require additional maintenance coding between individual apps. Displaying visuals in the iframe kept the design of the dashboard scaffold separate from the R scripts of each application, and promoted rapid iterative development.

A more extensive description of the various elements of the dashboard can be found in the Git repository [https://github.com/moocunsw/FL-dashboard]. The key questions driving the overall structure are in the table below.

In this section, we provide a description of the dashboard currently used in the FutureLearn course “Enhancing Learning and Teaching in Higher Education”. The tool is a set of Shiny applications wrapped into a simple LTI application. The LTI connector allows users to authenticate to the application, leveraging on the learner ID to present a personalised data view of the data for that use. The following four Shiny apps are available in the GitHub repo: • myClass: gives an overview of the participants’ geographical distribution, gender and ages. • myCommunity: provides a basic representation of the learners’ interactions with others in a network, and a view of the learning discussions in the course (based on popularity (views and likes) as well as general sentiment). • myProgress: displays a detailed account of the learners’ engagement with the content/activities of the course. The visual display also provides a ‘snapshot of the ‘course dna’ [ 27 ] as it allows to see what types of activities are available. • myPerformance: focuses on the quizzes, showing a detailed account of responses. At the time of writing the dashboard is about to be deployed to a live FutureLearn course, so as to collect student feedback across the duration of the course, to information both future designs and as a measure of success of the implementation. 6

Future work

The work presented here shows a clear trajectory of development which UNSW has taken to bring analytics to both educators and students in the MOOC space. The experience provided several opportunities to build internal capacity, but, by sharing the artefacts of this work, we hope to provide a solid starting point for others just getting started. By sharing this work, we also hope that is seen as an opportunity to begin to build a community of practice toward pushing the implementation and envelope of learning analytics practice. Of course, automating the development of an analytics dashboard for the FutureLearn platform for all would be a very desirable goal for both FutureLearn and the partner institutions.

The development process allowed us to explore alternative ways to implement a dashboard that other FutureLearn partner institutions may find useful, with several different visualizations explored, as well as consideration of related literature [ 31 ]. Furthermore, a direct comparison with a similar effort by Leon-Urrutia et al. [ 20 ] demonstrates the viability and effectiveness of the implementation.

Our implementation provides an opportunity to consider possible ways to use the tool with both educators and students. However, considering earlier discussion, we also show a cautious increase of sophistication in the student dashboard, compared with the educator dashboard. The increase of sophistication, however, is highly dependent to what FutureLearn will provide in the future. There is plenty of opportunity for more data to be exposed, for example: in-video behaviours, clickstream information, and user-session based information. All these are important to explore in more detail, particularly in trying to determine and judge on/off task behaviours, retention and resilience as well as relating survey data about goals and motivations to performance and achievement. 7

References

1. John Biggs . 2014 . Constructive alignment in university teaching . HERDSA Review of Higher Education 1 , 1 : 5 - 22 .

2. Martin J. Bliemel . 2013 . Getting Entrepreneurship Education Out of the Classroom and into Students' Heads . Entrepreneurship Research Journal 4 , 2 : 237 - 260 . https://doi.org/10.1515/erj-2013 - 0053

3. Simon Buckingham-Shum , Cassandra Colvin, Linda Corrin, David Gibson, Marcel Lavrencic , and Lorenzo Vigentini . Running a Learning Analytics Centre: Stories & Conversations . In A-LASI 2016 .

4. Muthu Kumar Chandrasekaran , Min-Yen Kan , Bernard C. Y. Tan , and Kiruthika Ragupathi . 2015 . Learning Instructor Intervention from MOOC Forums: Early Results and Issues . arXiv: 1504 .07206 [cs]. Retrieved October 16 , 2016 from http://arxiv.org/abs/1504.07206

5. Mahsa Chitsaz , Lorenzo Vigentini, and Andrew Clayphan . 2016 . Toward the development of a dynamic dashboard for FutureLearn MOOCs: insights and directions . In Proceeding of ASCILITE.

6. Daniel Churchill , Mark King, Beverley Webster , and Bob Fox . 2013 . Integrating learning design, interactivity, and technology . In Electronic dreams. Proceedings 30th ascilite Conference , 139 - 143 .

7. Doug Clow . 2012 . The Learning Analytics Cycle: Closing the Loop Effectively . In Proceedings of the 2Nd International Conference on Learning Analytics and Knowledge (LAK '12) , 134 - 138 . https://doi.org/10.1145/2330601.2330636

8. Ruth Cobos , Silvia Gil, Angel Lareo, and Francisco A. Vargas. 2016 . Open-DLAs: An Open Dashboard for Learning Analytics . In Proceedings of the Third ( 2016 ) ACM Conference on Learning @ Scale (L@S '16) , 265 - 268 . https://doi.org/10.1145/2876034.2893430

9. Linda Corrin and Paula de Barba. 2014 . Exploring students' interpretation of feedback delivered through learning analytics dashboards . In Proceedings of the ascilite 2014 conference , 629 - 633 .

10. Linda Corrin and Paula de Barba. 2015 . How Do Students Interpret Feedback Delivered via Dashboards? In Proceedings of the Fifth International Conference on Learning Analytics And Knowledge (LAK '15) , 430 - 431 . https://doi.org/10.1145/2723576.2723662

11. Linda Corrin , Gregor Kennedy, Paula de Barba, Aneesha Bakharia, Lori Lockyer, Dragan Gasevic, David Williams, Shane Dawson , and Scott Copeland . 2015 . Loop: A learning analytics tool to provide teachers with useful data visualisations . 409 - 413 . Retrieved June 13, 2016 from http://www.2015conference.ascilite.org/wp-content/uploads/2015/11/ascilite-2015 -proceedings.pdf

12. Chuong B. Do , Zhenghao Chen, Relly Brandman, and Daphne Koller . 2013 . Self-Driven Mastery in Massive Open Online Courses . MOOCs FORUM 1 , P: 14 - 16 . https://doi.org/10.1089/mooc. 2013 .0003

13. Stephen Few . 2006 . Information dashboard design .

14. Robin Ford , Lorenzo Vigentini , John Vulic, Mahsa Chitsaz, and Bgangadhara Prusty . 2016 . Through engineers' eyes: A MOOC experiment . 27th Annual Conference of the Australasian Association for Engineering Education : AAEE 2016 : 654 .

15. Bob Fox . 2016 . Flexible Frameworks for Blended Learning in Higher Education . International Institute of Social and Economic Sciences. Retrieved May 31 , 2017 from https://ideas.repec.org/p/sek/itepro/3905704.html

16. Dragan Gasevic , Vitomir Kovanovic, Srecko Joksimovic, and George Siemens . 2014 . Where is research on massive open online courses headed? A data analysis of the MOOC Research Initiative . The International Review of Research in Open and Distributed Learning 15 , 5: 134 - 76 . https://doi.org/10.19173/irrodl.v15i5. 1954

17. Imran Khan and Abelardo Pardo . 2016 . Data2U: Scalable Real Time Student Feedback in Active Learning Environments . In Proceedings of the Sixth International Conference on Learning Analytics & Knowledge (LAK '16) , 249 - 253 . https://doi.org/10.1145/2883851.2883911

18. M. Asif Khawaja , Gangadhara B. Prusty , Robin A. J. Ford , Nadine Marcus , and Carol Russell . 2013 . Can more become less? Effects of an intensive assessment environment on students' learning performance . European Journal of Engineering Education 38 , 6 : 631 - 651 . https://doi.org/10.1080/03043797. 2013 .834295

19. Fatemeh Salehian Kia, Zachary A. Pardos , and Marek Hatala . Learning Dashboard: Bringing Student Background and Performance Online.

20. Manuel Leon Urrutia, Ruth Cobos, Kate Dickens, Su White , and Hugh Davis . 2016 . Visualising the MOOC experience: a dynamic MOOC dashboard built through institutional collaboration . In Proceedings of the European MOOC Stakeholder Summit 2016 .

21. Weizhe Liu , Łukasz Kidziński, and Pierre Dillenbourg . 2016 . Semiautomatic Annotation of MOOC Forum Posts . In State-of-the-Art and Future Directions of Smart Learning, Yanyan Li , Maiga Chang , Milos Kravcik, Elvira Popescu, Ronghuai Huang, Kinshuk and Nian-Shing Chen (eds.) . Springer Singapore, 399 - 408 . https://doi.org/10.1007/ 978 -981-287-868-7_ 48

22. Lori Lockyer , Elizabeth Heathcote, and Shane Dawson . 2013 . Informing Pedagogical Action: Aligning Learning Analytics With Learning Design . American Behavioral Scientist: 0002764213479367 . https://doi.org/10.1177/0002764213479367

23. Simon McIntyre , Karin Watson , and Negin Mirriahi . 2015 . Learning to Teach Online-Evolving approaches to professional development for global reach and impact . In e-Learning Excellence Awards 2015: An Anthology of Case Histories . 128 - 140 .

24. Negin Mirriahi, Dennis Alonzo , Simon McIntyre , Giedre Kligyte , and Bob Fox . 2015 . Blended learning innovations: Leadership and change in one Australian institution . International Journal of Education and Development using Information and Communication Technology; Bridgetown 11 , 1 : 4 - 16 .

25. Zachary A. Pardos and Kevin Kao . 2015 . moocRP: An Open-source Analytics Platform . In Proceedings of the Second ( 2015 ) ACM Conference on Learning @ Scale (L@S '15) , 103 - 110 . https://doi.org/10.1145/2724660.2724683

26. Bill Parod . 2014 . Developing an Analytics Dashboard for Coursera MOOC Discussion Forums . Retrieved October 16 , 2016 from https://www.cni.org/wp-content/uploads/2014/12/Mon_Parod_DevMOOCDashboard.pdf

27. Rob Rubin , Alicia Redmond, Gregory Weber, and Khan Guirez . 2017 . Habits of Highly Successful Professional Learners and the Corresponding Online Curriculum .

28. Javier Santofimia Ruiz, Héctor J. Pijeira Díaz , José A. Ruipérez-Valiente , Pedro J . Muñoz-Merino , and Carlos Delgado Kloos. 2014 . Towards the Development of a Learning Analytics Extension in Open edX . In Proceedings of the Second International Conference on Technological Ecosystems for Enhancing Multiculturality (TEEM '14) , 299 - 306 . https://doi.org/10.1145/2669711.2669914

29. Dhawal Shah . 2016 . By The Numbers: MOOCS in 2016 - Class Central . Class Central's MOOC Report. Retrieved May 21 , 2017 from https://www.class-central.com/report/mooc-stats-2016/

30. George Siemens , Dragan Gasevic, Caroline Haythornthwaite, Shane Dawson, Simon Buckingham Shum, Rebecca Ferguson, Erik Duval, Katrien Verbert, and RSJD Baker . 2011 . Open Learning Analytics: an integrated & modularized platform . Open University Press. Retrieved from http://cmapspublic3.ihmc.us/rid=1KC16KK3Y -1DGTX1Y-H2/KG%20OpenLearningAnalytics .pdf

31. Kristin Stephens-Martinez, Marti A . Hearst , and Armando Fox . 2014 . Monitoring MOOCs: Which Information Sources Do Instructors Value? In Proceedings of the First ACM Conference on Learning @ Scale Conference (L@S '14) , 79 - 88 . https://doi.org/10.1145/2556325.2566246

32. Kalyan Veeramachaneni , Sherif Halawa, Franck Dernoncourt, Una-May O'Reilly , Colin Taylor , and Chuong Do . 2014 . MOOCdb: Developing Standards and Systems to Support MOOC Data Science . arXiv:1406 . 2015 [cs]. Retrieved June 13 , 2016 from http://arxiv.org/abs/1406.2015

33. Katrien Verbert , Sten Govaerts, Erik Duval, Jose Luis Santos, Frans Assche, Gonzalo Parra, and Joris Klerkx . 2014 . Learning Dashboards: An Overview and Future Research Opportunities. Personal Ubiquitous Comput . 18 , 6 : 1499 - 1514 . https://doi.org/10.1007/s00779-013-0751-2

34. Dominique Verpoorten , W. Westera , and M. Specht . 2011 . A first approach to “Learning Dashboards” in formal learning contexts . Retrieved September 3 , 2017 from http://orbi.ulg.ac.be/handle/2268/151988

35. Lorenzo Vigentini and Andrew Clayphan . 2015 . Pacing through MOOCs: course design or teaching effect? In Conference on Educational Data Mining (EDM).

36. Lorenzo Vigentini , Andrew Clayphan, Xia Zhang, and Mahsa Chitsaz . 2017 . Overcoming the MOOC Data Deluge with Learning Analytic Dashboards . In Learning Analytics: Fundaments, Applications, and Trends. Springer, Cham, 171 - 198 . https://doi.org/10.1007/978-3- 319 -52977- 6 _ 6

37. Lorenzo Vigentini , Manuel León Urrutia, and Ben Fields . 2017 . FutureLearn Data: What We Currently Have, What We Are Learning and How It is Demonstrating Learning in MOOCs . In Proceedings of the Seventh International Learning Analytics & Knowledge Conference (LAK '17) , 512 - 513 . https://doi.org/10.1145/3027385.3029433

38. Lorenzo Vigentini , Negin Mirriahi, and Giedre Kligyte . 2016 . From Reflective Practitioner to Active Researcher: Towards a Role for Learning Analytics in Higher Education Scholarship . In Learning, Design, and Technology , Michael J. Spector , Barbara B. Lockee and Marcus D. Childress (eds.). Springer International Publishing, 1 - 29 . https://doi.org/10.1007/978-3- 319 -17727- 4 _ 6 - 1

39. Lorenzo Vigentini and Catherine Zhao . 2016 . Evaluating the “Student” Experience in MOOCs . In Proceedings of the Third ( 2016 ) ACM Conference on Learning @ Scale (L@S '16) , 161 - 164 . https://doi.org/10.1145/2876034.2893469

40. Yuan Wang , Dan Davis, Guanliang Chen, and Luc Paquette . 2017 . Workshop on integrated learning analytics of MOOC post-course development . In LAK , 506 - 507 .

41. Alyssa Friend Wise. 2014 . Designing Pedagogical Interventions to Support Student Use of Learning Analytics . In Proceedings of the Fourth International Conference on Learning Analytics And Knowledge (LAK '14) , 203 - 211 . https://doi.org/10.1145/2567574.2567588

42. Alyssa Friend Wise, Jovita Maria Vytasek, Simone Hausknecht, and Yuting Zhao . 2016 . Developing Learning Analytics Design Knowledge in the “Middle Space”: The Student Tuning Model and Align Design Framework for Learning Analytics Use . Online Learning 20, 2. Retrieved October 16 , 2016 from http://olj.onlinelearningconsortium.org/index.php/olj/article/view/783