There are a number of similarities and differences between FutureLearn MOOCs and those offered by other platforms, such as edX. In this research we compare the results of applying machine learning algorithms to predict course attrition for two case studies using datasets from a selected FutureLearn MOOC and an edX MOOC of comparable structure and themes. For each we have computed a number of attributes in a pre-processing stage from the raw data available in each course. Following this, we applied several machine learning algorithms on the pre-processed data to predict attrition levels for each course. The analysis suggests that the attribute selection varies in each scenario, which also impacts on the behaviour of the predicting algorithms.
The advances in telecommunications in the last decade, together with an increased
accessibility to personal computers and internet-enabled devices, have revolutionised
teaching and learning. This increased accessibility has meant that for more than 35
million students, geographical and economical barriers to learning have been
overcome by accessing Massive Open Online Courses (MOOCs) offered by more than
500 universities. This is a figure which has doubled from 2014 to 2015, and is
expected to continue to increase, given that (according to Class Central [
The richness of the diversity of learning with MOOCs provides unprecedented opportunities for study. In tackling this diversity, it helps to understand the principles and affordances given by the FutureLearn platform compared with another wellrecognised MOOC provider with similar courses which could be used for a comparative study (such as edX).
FutureLearn data: what we currently have, what we are learning and how it is demonstrating learning in MOOCs. Workshop at the 7th International Learning Analytics and Knowledge Conference. Simon Fraser University, Vancouver, Canada, 13-17 March 2017, p. 74-93.
Copyright © 2017 for the individual papers by the papers' authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors.
Against this background, we investigated whether the inherent similarities and differences between these two different MOOCs platforms (FutureLearn and edX) could influence learner behaviour (assuming all other things are equal) and whether there are any observable factors that can provide an early attrition prediction in either case. This is especially valuable as it could be used to inform interventions designed to improve learners’ performance in future courses.
The structure of this paper is as follows: in section 2 (Positioning FutureLearn and edX ) we discuss theoretical underpinnings and describe the practical organisation of one exemplar FutureLearn course, contrasting it against that of a comparable edX course. In section 3 (Learning Analytics) we review related work on learning analytics, which has predominantly been concerned with studying dropout rates and in demonstrating the feasibility of machine learning algorithms for classification and prediction. In section 4, (Context of the present approach), the research questions are specified; whilst the processes followed in addressing them are described in section 5 (Methodology) alongside a detailed description of the courses selected (as the context of our study) and other technical details. The results are shown in section 6 (Analysis and Discussion), and the insights obtained are summarised in section 7, (Conclusions and Future Work), where we also identify further research. 2
The emergence of MOOCs is a consequence of the increased interconnectivity of the
digital age. When Siemens [
In contrast, other courses were designed to adapt the medium, learning materials
and assessments of traditional (instructivist, or cognitive behaviourist [
Noting that there are many similarities as well as important differences between learning-MOOCs and x-MOOCs (summarised in Table 1), it is interesting to compare them in practice by analysing case study courses.
Characteristic Number of learners Method of delivery Communication approach Design primarily supports the…
delivered (year) Related learning theory
… creation of connections between learners, resources and ideas x-MOOCs Should scale to large numbers
Instructivism or behaviourism … relationship between teachers and learners, mediated through task completion
It is of interest to investigate whether the inherent similarities and differences between these models (which in turns translate in a number of affordances provided by MOOC platforms) may influence learner behaviours. In particular, in this research study we compare the results of applying algorithms for predicting course attrition within two case studies. More specifically, we have selected a FutureLearn MOOC and an edX MOOC, and secured the corresponding datasets for their analysis.
FutureLearn courses are organised in weeks. Each week contains a set of activities,
called “steps”, each of which has a learning object belonging to a prescribed category.
Typical examples of these categories are: videos, articles, exercises, discussions,
reflections, quizzes and peer reviews. For each step, learners are able to write
comments, each of these in turn can be visibly “liked” (as in mainstream social media
platforms) and have replies or follow-up comments. This facility allows learners to
build connections amongst the community and with the learning objects presented, as
often these comments allow for their personal reflections and expressions of their own
understanding (or lack thereof). The use of such architecture reflects FutureLearn’s
pedagogical underpinnings inspired in social constructivism and Laurillard’s
conversational framework [
Similarly to FutureLearn courses, edX courses consist of weekly sections, which are composed of one or several “learning sequences”. These learning sequences are 1 A recent innovation is the facility to work in small groups “to come together and reach shared understanding” (https://www.futurelearn.com/about-futurelearn/our-principles). composed mainly of short videos and exercises, often with the addition of extra educational content such as links to web pages or interactive educational resources. All courses have an online discussion forum where students are able to post and review questions and comments to teaching assistants and other students. Despite offering this facility, the fostering of conversations for co-creation of knowledge does not feature prominently in the guiding principles of their pedagogy. Instead, edX courses can be categorized as x-MOOCs and follow an instructivist approach where the assessment is based on the completion of exercises. The data traces that learners create through their participation in the courses not only allow the institutions to award certification (when all assessment has been satisfactorily completed) but as it is recorded, it has the potential to be analysed further to predict whether the learner is likely to be eligible for a certificate at the end of the course.
Differences of pedagogical approaches aside, both FutureLearn and edX capture
data related to learners’ participation in their courses. These data traces left behind by
participating learners allow the institutions to award certificates (when all assessment
has been completed to satisfaction) but also, while they are captured, have the
potential to be analysed to predict whether the learner would be eligible for a certificate at
the end of the course [
In practical terms, the platform data is collated by both MOOC providers and given to the subscribing institutions with a structure that specifically affords the study of behavioural characteristics of the learners in the course (e.g. their graded achievements or the social interactions of the learners). This wealth of data offers great opportunities for collecting learning analytics (discussed in section 3, below), however, there are challenges in aligning the data collected and performing comparison studies not only because of the fundamentally different approaches taken by each MOOC but also important differences in the technical implementations adopted. 3
The term learning analytics is widely understood as “the measurement, collection,
analysis and reporting of data about learners and their contexts, for purposes of
understanding and optimising learning and the environments in which it occurs” [
Despite the recent coinage of this term, applying analytics in learning and
education has a long tradition, as educators have been interested in issues such as attrition
and dropout rates for many years. Attrition is typically understood in its simplest
terms as the rate in which the total number of learners in a course diminishes over
time, or the number of individual learners who drop out against those who were
originally registered. This definition has been long recognised as not being able to capture
the dynamic nature of learning [
However, the simplicity of this definition allows its application in many scenarios,
and this is of special interest in the context of Massive Open Online Courses
(MOOCs). Their arrival has coincided with the increasing application of learning
analytics as a transformative force: the value of analytics has now been extended
beyond the merely administrative and can now be used to inform and transform the
teaching process, with a potentially-significant impact on learning, assessment
processes and scholarly work (as foreseen by Long and Siemens [
The arrival of MOOCs coincided with increased concern with dropout rates in
traditional education. Although MOOCs attract many learners, typically only a very
small proportion actually completes these courses, following what Doug Clow called
a “funnel of participation” [
Despite the limitations of dropout as a stand-alone metric, which has led
researchers to question its value as a measure of success both in MOOCs [
It is worthwhile noting that the importance of accurate predictive models of
attrition or disengagement as studied through MOOC data can also be applied to
face-toface instruction, by making available predictions to teachers so that they can provide
timely feedback or take any other suitable action [
As explained earlier, the main motivation for this research study was to investigate whether the inherent similarities and differences between these two different MOOCs platforms could influence learner behaviour and whether there are any observable factors that can provide an early attrition prediction. The authors are researchers from two institutions, each delivering MOOCs under one of the paradigms, hence enabling a comparative study. The institutions are the Universidad Autónoma de Madrid (UAM) and the University of Southampton (UoS).
The UAM became a member of the edX consortium in 2014 and currently offers
eight MOOCs at edX [
Against the existing background, we formulated the following research questions to conduct this comparative study: 1. Amongst those attributes that are common to both MOOCs, which are the most valuable with regards to the prediction of attrition? 2. Is the most valuable attribute in predicting attrition for the FutureLearn (FL)
MOOC different from the one for the edX MOOC? In pursuing these questions, it was important to use a well-performing machinelearning algorithm (in terms of the accuracy of the prediction) for both MOOCs. In particular, it was important to establish how soon it is possible to make a reasonably accurate prediction of attrition within each MOOC. More specifically, in what week (out of the total length of the course) are the predictions sufficiently accurate to be useful for each of the case study courses. 5 5.1
Firstly, we selected suitable case study courses from those available in FutureLearn and edX, delivered by the collaborating institutions (to ensure student datasets would be readily available). The criteria used in the manual selection of these courses included: they should be of a similar discipline or theme in the broadest possible way (i.e. either from STEM subjects or social sciences), and of a similar duration. In the case of there being more than one matching pair, we would give preference to those for which the duration is the longest. If more than one “run” of these courses had data available, we would select those for which the cohorts were the largest.
After applying the above criteria to the courses available (a total of 22: 8 in edX and 14 in FutureLearn), we selected the following: the 7-week-long edX course in Spanish history titled “The Spain of Don Quixote”2 (Quijote501x) and the 6-weeklong in archaeology titled “Archaeology of Portus: Exploring the Lost Harbour of Ancient Rome”3 (Portus). We refer to these courses in the rest of this paper as the edX MOOC and FL MOOC respectively. Both courses had a certification available to those learners who meet the platform completion criteria, and both have some sort of assessment (exercises and quizzes, respectively). 5.2
For both cases, in a pre-processing stage, we computed the value of a number of attributes from raw data available in each context, such as the number of sessions, the 2 https://www.edx.org/course/la-espana-de-el-quijote-uamxquijote501x-0 3 Archaeology is regarded as being on the intersection of science and humanities (https://www.futurelearn.com/courses/portus/4/steps/76822). However, as the humanities element of the course is history, we felt this discipline is sufficiently close to that in the Quijote 501x course, and that therefore the Portus course would attract learners of not too dissimilar interests and backgrounds. number of days, the number of events, total time spent on exercises/quizzes and the number of social interactions in discussions forums). Following this, we applied machine learning techniques to the pre-processed data to predict attrition levels for each dataset (the algorithms mentioned previously in the “Machine learning” subsection). The prediction is performed looking forward to the week ahead, and it becomes more accurate when there is more information available, as would be expected. However, we have established the point by which an early warning could be provided with a reasonably high degree of accuracy for each case. This was interesting to ascertain as there is a clear trade-off between accuracy and timeliness of the prediction: clearly there is less value of a dropout prediction after the student has already left the course, whereas a timely prediction (even if less accurate) could enable an intervention which might help the student to continue). 5.3
The anonymised datasets for each MOOC were processed using an adaptation (of the
early stages) of Jo et al’s pipeline for expediting learning analytics [
The FutureLearn MOOC selected has been offered four times to date as shown in
Table 2. Rather than aggregating the four datasets, we opted for selecting the offering
(run) with the highest number of learners eligible for certification as this would be the
least imbalanced dataset of those available (however, due to the “funnel of
participation” effect [
Therefore, in the selected FL MOOC dataset there was data from 8935 enrolled learners, from which 3646 learners were actively involved in the course content. Of all the students, only 1843 engaged as social learners (typically posting comments, but also through “likes” as in social media). A total of 2075 completed at least 50% of the learning activities and thus were eligible to receive a certificate.
The course runs for 6 weeks, during which a number of learning activities are presented (videos, articles, exercises, discussions, reflections and quizzes as mentioned earlier). The results of the assessment (in quizzes specifically) are shared with the learner (and recorded) but the actual results do not affect the eligibility for the certificate, as this is based on only completion of activities (as explained above). edX MOOC Dataset.
The edX MOOC selected has been offered three times to date. For consistency, we also selected the offering with the highest number of learners, which was also on its first delivery (February to May 2015): a total of 3530 learners enrolled in the edXMOOC, from which 1718 students were actively involved in the course content. Of all the students, only 423 engaged in some activity or viewed multimedia content over the last week. A total of 164 obtained a grade of more than 60% and thus received a certificate.
The length of the course is seven weeks. In addition to the discussion forum every week there are multimedia resources, both in text files and in video formats, and practical activities without evaluation. Each week ended with an evaluation activity that is a test of 21-23 questions. Each weekly evaluation contributed 14% of final grade for the course.
Similarly to FutureLearn, edX stores all learners’ events. There is one file per day with the events that happened. Each event has a category. The most common ones are: navigation events, video interaction events, assessment interaction events, discussion forum events.
As presented before, each MOOC platform creates different type of learners’ events that are relevant according to the philosophy behind their MOOC approach. As a result, there are a potentially large number of attributes that could be analysed if studying attrition separately for each of these contexts. However, in order to facilitate a meaningful comparison between both approaches, only the intersection of the attributes from the available data was considered. The following is the list of attributes known for both datasets: • number_sessions: total number of sessions in the course. This was important to calculate as neither platform provides such data. In determining a session within the edX MOOC, an inactivity threshold was established, if the elapsed time between two consecutive interactions of the learner exceeds the threshold, these interactions were assumed to have taken place in two separated sessions as the learner was considered not to have been active during this time. For the FL MOOC the start time of a given activity (step) is only recorded the first time the learner accesses the given step, so the calculation of the inactivity threshold was performed slightly differently (taking into account the finishing time of a previously finished activity instead) but applying similar principles. number_comments: total number of social interactions (comments and replies) in the course. total_time: total time invested in the course (inactivity periods aside). More specifically, it is defined as the aggregate of the elapsed time between the access to each problem or exercise and the submission of the corresponding attempted solution (calculated individually in the case of there being several attempts). As before, an inactivity threshold is applied, considering the student not active if the elapsed time between getting the problem and the problem submission exceeds the threshold.
time_problems: total time invested in answering exercises (assessments).
These attributes were calculated for each week and each learner. The aim of the formulation of our predictive models was to detect those learners which are eligible for a certificate. In the case of FutureLearn learners, they need to complete at least 50% of the course activities (regardless of assessment performance), whilst edX learners need to obtain more than 60% marks in the assessments to obtain a certificate (regardless of participation). The dependent attribute in both cases was to detect whether the learner would obtain a certificate. 5.4
In recent years there has been significant increase in development and the availability
of data analytics tools. This includes readily available, complex machine learning
algorithms, toolkits such as WEKA, and domain-specific packages and libraries for
programming languages such as python and R 4 (amongst others inventoried by Slater
et al. [
rithm, similar to AdaBoost, which can be used for multi-class regression problems.
GBM was first proposed by Freund and Schapire [
Weighted k-Nearest Neighbours (kNN) The kNN makes use of simple heuristics
of distance (or similarity sampling) to perform the classification [
Support Vector Machine [
Using these datasets, we implemented four classification models that have been extensively tested and shown to generate good classification results. We used the machine learning algorithms presented in subsection 5.4.
Due to the content of the MOOCs being organized in weeks, we calculated weekly models of each course, in a similar way as Kloft et al. 20. In order to measure the performance of these models we have used the area under the ROC curve (AUC) metric 5. (See Appendix A for the full set of ROC curves for all the models using both datasets.) These measurements helped us to select the machine learning algorithm that is best suited for each context. Then, taking into account these previous selections, we took the approach of finding out the best attributes for each MOOC. 6.1
Firstly, we compare the performance of the four machine learning algorithms mentioned.on the FL MOOC dataset. The two best-performing algorithms (with regards to the AUC metric) are GBM and XGBoost, (although the difference between them is negligible) (see Fig. 1).
However, when considering the performance in computational time, XGBoost is faster than GBM by an order of magnitude (during the training phase) as shown in Fig. 2. Due to this comparative advantage, we consider the XGBoost as the best algorithm amongst those tested on this dataset.
Once a machine learning algorithm was selected, we studied the varying importance of the attributes throughout the duration course for this algorithm (see Fig. 3). This “importance” suggests the predictive value of each attribute at a given week. 5
Note that in this context, execution time per model is not relevant, given that the predictions are not required tocalculated in real time (can be calculated in real time.daily processes, once data is updated). Therefore, a “poor” performance in this metric is much more importantless indicative of the goodness of the model than the accuracy as reported by the AUC metric.
During all the weeks, the most relevant attribute is number_sessions; however, the attribute related with social interactions is the second most relevant one.
W1 GBM
W2
W3 kNN
W4 LogReg
W5
W6 XGBoost 1 number_sessions total_time number_comments time_problems As before, we first compare the performance results of the four mentioned machine learning algorithms (see Fig. 4Error! Reference source not found.). As in the case of the FL MOOC dataset, the best performing algorithm for the edX MOOC dataset is GBM, though in this case the difference is significant. Finally, we studied the importance of the attributes throughout the course for this algorithm (see Fig. 5).
From the start of the course, attributes number_sessions and total_time are the most valuable for the prediction models. However, from the end of fifth week the most reliable attribute is time_problems. We found that in this course, which follows an x-MOOC approach, the attribute related to social interactions (number_comments) did not contribute to the prediction. 6.3
Each of the machine learning algorithms benchmarked provided good results for both scenarios; however their performance varied in terms of accuracy. GBM is the best one for both approaches from the beginning to the end of the courses; and XGBoost is the second best for the FL MOOC throughout the course and for the edX MOOC after the third week. Based on these results, we selected the XGBoost algorithm for the FL MOOC and the GBM for the edX MOOC.
Once the algorithms were selected, we studied the importance of the attributes in both courses. On the one hand, the most relevant attribute along the duration of the FL MOOC was number_sessions and number_comments was the second relevant attribute especially during the first weeks of the course. Results confirm that the progression dedicated in the course is the important issue because the most relevant at1 0.95 0.9 0.85 0.8 0.75 0.7 100 90 80 70 60 50 40 30 20 10 0
W1 GBM
W2
W3 kNN
W4
W5 tribute was the number of session in the course. Moreover, social interactions also have some importance.
W1
W2
W3
W4
W5
W6
W7 number_sessions total_time On the other hand, the most relevant attributes for the edX MOOC, were total_time and time_problems. The total_time attribute was the most relevant until the fifth week and after that the most relevant one was time_problems. These results confirm that in courses such as this, it is important to dedicate time to learning the course content and also to undertake the assessments.
Finally, we were interested in knowing how soon it is possible to have a reasonably accurate prediction of attrition. In the case of the FL MOOC, the baseline accuracy of the predictor that classifies learners that do not complete 50% of the course is 0.91. In the case of the edX MOOC, the baseline accuracy of the predictor that classifies non-certificate earners is 0.90. 7
A study has been carried out to investigate whether the inherent similarities and differences between the affordances provided by MOOC platforms may influence learner behaviours, such as engagement towards certification or dropout. More specifically, machine learning algorithms were applied to prediction attrition in MOOCs that have been delivered in two different platforms. To this end, we selected an edX MOOC and a FutureLearn MOOC of comparable structure and themes, and selected observable factors that can be used as an early predictor for attrition in these cases.
Common attributes to these cases were identified and the most valuable of these (with regards to the attrition prediction) were used. For both datasets we extracted the following comparable attributes: number_sessions: total number of sessions in the course; number_comments: total number of social interactions (comments and replies) in the course; total_time: total time invested in the course and time_problems: total time invested in answering exercises or quizzes (assessments).
Next, we generated several predictive models to detect (for a FL MOOC) that the students could complete at least 50% of the course, and (for an edX MOOC) that they could obtain a grade of more than 60% and hence a certificate. The key attribute in both cases was to predict whether the learner would obtain a certificate.
These predictive models were generated using these four machine learning algorithms: k-nearest neighbours (kNN), gradient boosting machine (GBM), extreme gradient boosting (XGBoost) and boosted logistic regression (logitboost). Due to the content of the MOOCs being organized in weeks, we calculated a model per week of each course.
From those tested, the best machine learning algorithms for both the edX MOOC and the FL MOOC are GBM and XGBoost. However, the relevant attributes were different for each course. In the FL MOOC the most important ones were number_sessions and number_comments, both related with the connectivism paradigm: as expected, learners who engage more in activities facilitating connections with others and with knowledge itself would do better than those who do not. In contrast, for the edX MOOC the most important attributes were total_time and time_problems, which is consistent with the pedagogical design of instructivist courses, where learners devoting time to learning activities gain more from these, and therefore do better than learners who do not.
The predictive models offered a reasonably accurate prediction of attrition within each MOOC approach (over 90% accurate prediction, available approximately halfway through the course delivery).
As future work, more case studies could be added to this study. On the one hand, taking into account more deliveries of the studied courses and, on the other hand, including courses from other disciplines. Finally, we are planning the generation of warning systems that may automatically warn the student at risk of not obtaining a certificate. However, more work in understanding attrition and learner dropout models is required still to be able to measure the impact of such interventions.
This work has been partially funded by the Madrid Regional Government with grant No. S2013/ICE-2715, the Spanish Ministry of Economy and Competitiveness project "Flexor" (TIN2014-52129-R), and by the Web Science Institute Stimulus Fund 2015/16 Project “The MOOC Observatory Dashboard: Management, analysis and visualisation of MOOC data”. The authors are grateful to Darron Tang and Jasmine Chen from the University of Southampton for their work in this project during their summer internships.
Special thanks are due to the Universidad Autónoma de Madrid and the University of Southampton for the support of this inter-institutional collaboration and facilitating the access of the data of the respective MOOCs on which this study is based. A
a) Generalised Boosted regression Models b) k-Nearest Neighbours c) Boosted Logistic Regression d) XGBoost Fig. 6. ROC values for all weeks prediction models varying the algorithm on the FutureLearn MOOC dataset 90 a) Generalised Boosted regression Models b) k-Nearest Neighbours c) Boosted Logistic Regression d) XGBoost Fig. 7. ROC values for all weeks prediction models varying the algorithm on the edX dataset