In recent years, we have experienced the rise of e-learning and the growth of available Massive Online Open Courses (MOOCs). Consequently, an increasing number of universities has dedicated resources to the development and publishing of MOOCs on portals. A common practice for operators of such MOOCs is to re-o er the same course over the years with similar modalities and minor improvements. Such re-runs are still a ected, as most of the MOOCs, by a low percentage of enrolled users who manage to successfully complete the courses. Hence, analyzing similar MOOCs can provide valuable insights to better understand the reasons of users for dropping out and potentially can help MOOCs' administrators to better shape the structure of the courses to keep users engaged. To that end, we analyze two successive o erings of the same MOOC, created by Curtin University and published on the edX platform. We extract features for our prediction experiment to detect dropouts, considering two di erent metrics: (i) the percentage of users active time and (ii) the initial week after users rst interaction with the MOOC. We train a Boosted Decision Tree classi er with the extracted features from the original run of the MOOC and predict dropouts on its re-run. Furthermore, we identify a set of features that likely indicates whether users will drop out in the future or not. Our results indicate that users interacting with particular tools at the very beginning of a MOOC are more likely to successfully complete the course.
Nearly all MOOCs su er from high dropout rates. While the number of users
who enroll is high, the percentage of those that successfully complete the course
is very low (generally lower than 10%) [
Gutl et al. [
Co rin and colleagues [
Kloft and al. [
Amnueypornsakul and colleagues [
In our previous work [
Teusner et al. [
Our dataset consists of the original o ering of a MOOC, referred to as MOOCC1, and the rst of its re-runs, coded as Re-Run1. Both MOOCs were created by Curtin University and were available on edX4. The original o ering MOOCC1
was available online during the second semester of 2015, while the re-run ReRun1 was available online between April and May 2016. Both o ers had no entry prerequisites. However, users needed to have access to YouTube in order to watch the independently created video content. Regular activities, such as polls, questions, and discussion board tasks were integrated into the course content. The course syllabus consisted of a total of four modules, each estimated to require a time commitment of two hours per week. An extra introductory module and a course wrap up module completed the course calendar. To complete each of the four main modules, participants needed to complete an activity and a quiz, with the quizzes being an extension of activities. Therefore, engaging in the activities helped participants to answer the questions in the quizzes. Each quiz accounted for 25% of the nal grade, with a Certi cate of Achievement issued to participants with an overall score of equal or greater than 70%. The team of instructors consisted of an associate professor and a teaching assistant. The two o ers were for the larger part similar to each other regarding contents and activities, with Re-Run1 undergoing only some minor changes. Table 1 reports a summary of the enrollments and completions of the original MOOCs and of its rerun. As the column Enrollments reports, MOOCC1 has a total of 21,948 enrolled users and its re-run, Re-Run1, counts 10,368 enrolled users. Within the enrolled users, we distinguish users that enroll and leave the MOOCs without engaging any further (i.e., Inactive users), from those who have more than the simple enrollment interaction (i.e., Active users). Completers are users that successfully completed the MOOCs, while Dropouts are those who failed to do so. Overall, the dropout rates are never lower than 89%.
Both o ers are structured in a self-paced manner and are organized in two phases. During the rst phase, users can only access the course main page and enroll. At this stage, the course's material is not available, and only a limited number of interactions is possible. This initial phase lasts for roughly 2 months for both MOOCs. At the beginning of the second phase, the course material is uploaded all at once and users can engage at their own pace. Enrollment is possible during the second phase as well. After the o cial end of the MOOCs, users still can register and interact with the MOOC, but, in this case, they can not obtain a certi cate as the course is already o cially over. Due to these settings, we consider only enrollments happening before the o cial end of the MOOC. Furthermore, we also discard interactions happening before the course's o cial start. Such interactions do not represent users learning style because the course material is not available yet. Both MOOCs also included a course forum where users can post and discuss.
We extract a set of features to describe each user of the MOOCs from Curtin University. First, we calculate a set of time-based features that build upon the concept of sessions. A session is a set of chronologically ordered interactions, in which each interaction happens within a certain timespan from the previous and the next one. Particularly, we use a threshold of 30 minutes and calculate sessions for all users. Following this concept of sessions, we de ne the following features: Sessions as the total number of users' sessions; Requests as the total number of interactions per user; Active Time is the total time users interacted with the MOOC (as sum of all sessions' duration); Days indicates the number of days during which users interacted at least once with the MOOC. Furthermore, we compute 4 averaged features; Timespan Clicks is the average timespan between two consecutive clicks in the same session (averaged over all sessions); Session Length as Active Time divided by Sessions ; Session Requests as Requests over Sessions ; Day Requests is Requests divided by Days. Moreover, we exploit the detailed edX logs, to identify the type of event triggered and the particular tool each interaction refered to. Curtin University's MOOCs includes 6 speci c tools5; Course Navigation, Video, Problem, Poll & Survey, Bookmark and Discussion Forum. For each of these tools, a set of events indicates the particular action that took place. The list of events for each tool, together with the session related features we calculated, are reported in Table 2. The Video tool, is the only one that allows distinction between Browser and Mobile (through the edX mobile application) triggered interactions. To categorize these two sources, we create an additional tool, ltering out Mobile interactions from Video, and we name this new tool Video Mobile. We create a feature for each event, counting the number of times each of these was triggered by users' interactions. Secondly, we de ne two di erent approaches and calculate the features according to these. First, we consider a varying percentage of users total active time. Particularly, we consider 5 The complete list of edX's events is available at http://edx.readthedocs.io Main Page Links
About, Faqs, Home, Instructor, Progress, StudyAtCurtin Course Navigation
TabSelected, PreviousTabSelected, NextTabSelected, LinkClicked, OutlineSelected
Video Video Mobile
Problem Poll & Survey
Bookmark Discussion Forum
CaptionHidden, CaptionShown, LanguageMenuHidden, LanguageMenuShown, Loaded, Paused, Played, PositionChanged, SpeedChanged, Stopped, TranscriptHidden, TranscriptShown CaptionHiddenM, CaptionShownM, LanguageMenuHiddenM, LanguageMenuShownM, LoadedM, PausedM, PlayedM, PositionChangedM, SpeedChangedM, StoppedM, TranscriptHiddenM, TranscriptShownM Check, CheckFail, FeedbackHintDisplayed, Graded, HintDisplayed, Rescore, RescoreFail, Reset, ResetFail, Save, SaveFail, SaveSuccess, Show, ShowAnswer PollSubmitted, PollViewResults, SurveySubmitted, SurveyViewResults Accessed, Added, Listed, Removed CommentCreated, ResponseCreated, ResponseVoted, Searched, ThreadCreated,
ThreadVoted all interactions within the rst 1% to 100% of the total active time (per user) and call this setting Scaled Time.
As a second approach, we calculate features considering the rst 7 days after
a users' rst interaction. We name this setting Days. To overcome the class
imbalance problem [
We run classi cation experiments with Boosted Decision Trees for both approaches using two di erent set of users as input. First, we consider the all Enrollments and then only the Active users, as indicated in Table 1. We evaluate our experiments using accuracy, which is calculated as the number of correctly predicted examples divided by the total number of predicted examples. Therefore, this measure can assume values between 0 and 1; a value of 0 indicates that all examples have been wrongly classi ed, while a value of 1 means that all examples have been correctly predicted.
As a nal analysis, we investigate the importance of our features in the classi cation task. Boosted Decision Trees also provide a weight for each used feature. Their weight represents the number of times a feature is used to split the data across every single decision tree. Thus, the higher the weight, the more precise the obtained split. We explore the ranking of the features for both metrics when the input consists only of Active users. 4
0.9 1.0 0.9 Fig. 1. Dropout prediction results on Re-Run1. Each sub gure depicts the accuracy results for the two proposed approaches. Figure 1(a) reports the results for the Scaled Time approach and Figure 1(b) depicts the results for the Days approach, with the xaxes indicating the considered percentage of user active time or the considered number of days after user rst interaction with the MOOC respectively. The y-axes (bounded between 0:4 and 1) of each gure indicate the accuracy, with the baseline being plotted as a solid black line at 0:5. The green lines refer to the experiments in which we consider all users (cf. Enrollments in Table 1), while the ones in red represent experiments in which we analyzed only the active users (cf. Active in Table 1). Considering only active users always yields the highest accuracy, except when the considered percentage of a user's active time is larger than 80%. In this case, the green and red lines switch places, however, the overall performance of the prediction experiments only minimally di ers. It is likely that a lot of users register for the MOOCs at an early stage, or at least more than a week before the material becomes available. At this time, there are few possible interactions, and it is likely that users only come back once the material is made available. Thus, this approach has a lower overall accuracy.
The results of the Scaled Time experiment are plotted in Figure 1(a). Also, with this setting, considering only the active users is the approach that produces the highest accuracy. Again, the accuracy is never lower than 0.7 and gets as high as 0.96 when the whole users' active time is considered. When all enrolled users are taken into account, the accuracy ranges from 0.59 to 0.98, constantly increasing as the percentages of users' active time get higher. Both settings have a similar pro le when these percentages get higher than 80%.
Table 3 lists the best performing features for the two approaches when the active users are considered. The rst column lists the Tool and the second the speci c features that belong to it. The remaining columns report the weights for the features, rst for the Days and then for the Scaled Time experiments. For reasons of space, we report only some values for both approaches. Particularly, we show day 1, 4 and 7 for the First 7 Days approach and 5%, 50% and 100% of users' active time. The weights highlighted bold are the highest for that particular experiment. We see that Progress is always one of the features with the highest weight for both experiments. Interactions of type Progress refer to users accessing a dedicated page to track their scores for single problems and the current overall course grade. Particularly, this page includes a grading chart, which reports the obtained scores on each graded assignment in the form of a bar chart. Moreover, the page also o ers a panoramic overview of the whole set of problems, organized per-section and listed in the order they occur in the MOOC. The weight of this feature increases with the days and time percentage. If we extract the number of interactions of this type for both classes, we obtain a total of 17; 240 for the Completers and of 30; 916 for the Dropouts for Re-Run1. For MOOCC1 we have 121; 228 interactions for the Completers and 54; 768 for the Dropouts. The class average yields 82.88 and 80.98 Progress interactions for the Completers, and 6.88 and 5.40 for the Dropouts of Re-Run1 and MOOCC1 respectively. Besides, if for the Dropouts we include also the users with only the enrollment action, the averages gets as low as 3.04 for Re-Run1 and 2.68 for MOOCC1.
Hence, we nd that together with having a high number of correctly solved problems, constantly monitoring the personal progress strongly indicates whether a user will drop out or not. Similarly, ProblemCheck becomes more signi cant, the more days and higher percentages of interactions are analyzed. This interaction indicates a problem being correctly checked by the system after users submitted an answer. The high scores of this feature come as no surprise, as users are likely to solve problems only after they study and learn from the course's material, that is, at a later stage during the MOOC.
Certain tools and their features never obtain signi cant weights, such as Video Mobile or Forum Discussion. The low weights for Video Mobile indicate that users mostly interact with the MOOCs using a desktop machine rather than the edX mobile application. Poll & Survey and Bookmark are rarely used tools, either due to being poorly advertised or to users not regarding them as particularly useful to complete MOOCs. It also appears that interactions within the Forum Discussion barely relate to Completers or Dropouts. First, it is possible that the course's structure does not require users to engage with the forum. This could be due to unchallenging courses or, more likely, due to the self-paced setting of the MOOCs. Users engage at their own pace and confront the same challenges at di erent times. As a consequence, the role of the forum as a realtime communication channel and as the rst source of help might be limited.
In this work, we experimented with dropout detection on a MOOC re-run o ered on the edX portal by Curtin University. We train a Boosted Decision Tree classier on the initial o ering of a MOOC and predict users that will drop out on its re-run. Our results indicate that the rst week of users' interactions already provide information about whether users will complete the re-run or eventually drop out. Furthermore, we evaluate the importance of each of our proposed features for the classi cation. We discover that the frequency users check their progress and correctly solve problems within a short period after rst interacting with a MOOC, are related to users' probability of completing the MOOC. We also note that certain tools are barely used by users and, therefore, do not carry any valuable information for the prediction task. Moreover, we nd that the bene ts of social tools, as in our case the discussion forums, appear to be related to the way MOOCs are organized (i.e., limited bene ts for self-paced MOOCs) and on the e orts they require for users to engage with them.
Analyses at tool level can represent a valuable next step. Abstracting from the particular event that took place, might help to di erentiate more precisely most and less used tools, and to con rm our current ndings. Using di erent approaches that focus on the initial users' interactions, will help learn more about users' behaviors. Analogously to interaction and click-pattern mining approaches from other domains [14{17], we plan on identifying interaction types of users by clustering users of MOOCs according to their click- and interaction patterns to improve dropout detection. 6
This work is in part supported by the Graz University of Technology, Curtin University, and the MOOC Maker Project (http://www.moocmaker.org/, Reference: 561533-EPP-1-2015-1-ES-EPPKA2-CBHE-JP). The authors particularly thank Curtin University for providing the analyzed datasets.