The wide collection of data on digital platforms has enabled researchers to conduct many analyses around students' behaviors. One specific relevant context is the one of Massive Open Online Courses (MOOCs), where dropout is very frequent and other related behaviors such as deadline-driven strategies could be analyzed. This work aims to analyze learners' behaviors, such as deadline-driven behaviors and help seeking, and carry out predictive models to forecast how early learners will deliver their tasks and dropout. Results show that deadline-driven behaviors were frequent (about one third of learners delivered the tasks in the last 48 hours) and they had a moderate correlation with dropout. Moreover, predictive models served to predict the number of days learners delivered their tasks in advance with RMSEs between 1.1 and 1.9 days, depending on the task. As for help seeking, this was frequent in videos but not with the forum. In addition, when analyzing dropout, results showed accurate results (AUC up to 0.94) although variables about videos and formative exercises did not perform so well by themselves, without previous summative grades.
Massive Open Online Courses (MOOCs) allow the collection of data about their learners, which can
be very useful to understand learners’ behaviors. One typical research line is prediction in MOOCs
[
Some other works have tried to include more complex variables that capture learners’ behaviors.
Among them, Moreno-Marcos et al. [
Among those possible behaviors, the moment when learners deliver their task may also have an
impact. In this direction, some works have analyzed procrastination, and e.g., Yao et al. reported
connections between past and future activities [
Another interesting behavior is help seeking. This behavior has been widely analyzed in Intelligent Tutoring Systems [15] where the systems usually ofer hints, but it is interesting to analyze how learners do their assessments in the MOOC. For example, research has shown that many learners use the strategy of opening the assessments first to then look for the questions in the videos [ 16]. Similarly, it would be relevant to analyze whether or not learners look for information in the forums while taking the exams and the relationship of these behaviors with dropout.
Considering this context, the objective of this work is to contribute with an analysis of deadline-driven behaviors and help seeking, and how to predict both deadline-driven behaviors and dropout using multiple features, including help seeking.
The remainder of the paper is as follows. Section 2 presents the methodology of the paper, including details about the MOOC design, the independent and dependent variables included in this study, and the analytical methods and metrics. Section 4 details the results of this paper, including the results derived from a descriptive and a predictive analysis. Finally, Section 4 provides the conclusions of this paper and discusses the main limitations and possible future research directions.
The study was carried out using data from a MOOC with five weekly modules on programming hosted by Universidad Carlos III de Madrid. This course was instructor paced and consisted of five weekly multiple-choice tests (15% of the grade each) and two peer-review programming assignments (25% of the grade), which were done in weeks 3 and 5. The passing rate of the MOOC was 60% and it was not necessary to complete all the assignments to pass the course. The total duration of the course was 8 weeks, considering that each module was opened for two weeks (except the first one, which was opened for three weeks). Regarding the possible activities, the course included multiple videos and formative questions to prepare for the assessments, and a forum to ask questions. Considering the interactions with these materials, several variables were computed for this analysis. The full list of them is presented in Table 1.
For the predictive models, the variables to predict are the number of days in advance learners submit the tasks (deadline-driven behavior, dif_ex ) and prevail (used in this study to represent dropout), and the rest serve as predictor variables. Models are evaluated using Decision Trees (DT), Random Forest (RF), k-Nearest Neighbors (kNN), and Support Vector Machines (SVM). For the deadline-driven behavior analysis, Root Mean Square Error (RMSE) is used, and for the dropout one, Area Under the Curve (AUC) is computed to evaluate the results. For the first case, RMSE is used as the dependent variable is continuous and this metric might be better than others for this type of research, according to the literature [17]. Moreover, AUC is used for dropout as this metric is generally appropriate for student behavior classification problems [17].
This section is divided into two subsections to present a descriptive analysis of deadline-driven behaviors and a predictive analysis of deadline-driven behaviors and dropout.
The first analysis consisted of determining how early learners usually delivered their tasks. To do that, a histogram with the distribution of the number of days in advance that learners delivered the MOOC assignments was calculated. This histogram is shown in Figure 1. From this figure, it can be inferred that roughly 22.8% (4487 students) delivered their tasks within the last 24 hours, and 10.23% between the last 24 and 48 hours.
While this is not necessarily negative since many learners may organize well to do it in the last days, it is also interesting to delve into how delivering early may afect dropout. In order to do that, the number of learners who drop out the course the following week when they deliver the task late is analyzed. Table 2 indicates the number of learners who deliver each task within the last 24h and between the last 24-48h, and the number of them who drop out at next exam.
In the first exam, results show that not many learners (considering the total) delivered the tasks in the last moment, probably due to their enthusiasm to start the course. However, as time passes, the percentage of students showing deadline-driven behaviors increased, which could highlight lack of time, happiness with the course or an increased/reduced task dificulty, which is a situation that has been previously observed in the literature [18]. It is interesting that the number of learners who showed deadline-driven behaviors was higher in Week 3. A possible reason is that the first peer-review activity was due that week and learners had to organize well to cover both the exam and the peer-review lab activity. Moreover, it is interesting that almost 50% of the learners who submitted their tasks in the last 24h in the first two tests dropped out the course by the next examination, and this behavior has a moderate correlation with prevail (0.41, i.e., learners who submitted their tasks with more days in advance were less likely to drop out). In addition, 40% of the learners who submitted between 24-48h in advance also dropped out. These figures decreased in the next exams and only 36% of the learners who delivered in the last 24h in Exam 4 dropped out the course in the last exam.
Finally, regarding help seeking, Figure 2 shows the number of interactions showing help-seeking behaviors when learners used videos or the forum while taking the exams. Results show that the level of forum help-seeking is almost negligible, although it was more frequent for the case of videos. This shows that videos could be a valuable source of information that learners might use to clarify their concepts while taking the MOOC assessments. Nevertheless, a limitation of this figure is that learners might also use other sources (e.g., materials outside the platform) to seek help during the task, and this is not tracked. Further analysis should be done around this.
As for the predictive models, weekly models to predict the average days of anticipation in submissions (deadline-driven behavior) and drop out were developed using the algorithms mentioned in Section 3.
For the prediction of deadline-driver behaviors, four situations were analyzed and five models were developed for each week and algorithm. These situations (S) are: • S1. The average number of days of anticipation in submissions (considering all five weeks) is predicted using weekly cumulative data (i.e., variables in week 3 include interactions in weeks 1, 2 and 3 to predict the dependent variable referred to the average of weeks 1-5).
S3 S2 S2 • S2. The weekly number of days of anticipation in the submissions is predicted using weekly cumulative data (i.e., variables in week 3 include interactions of weeks 1, 2 and 3 to predict the dependent variable referred to week 3). • S3. The average number of days of anticipation in submissions (considering all five weeks) is predicted using weekly independent data (i.e., variables in week 3 include interactions in week 3 to predict the dependent variable referred to the average of weeks 1-5). • S4. The weekly number of days of anticipation in the submissions is predicted using weekly independent data (i.e., variables in week 3 include interactions in week 3 to predict the dependent variable referred to week 3).
Considering these situations, results of the predictive models are presented in Table 3. Results show that it is possible to detect the number of days of anticipation in tasks with an RMSE below 2 days in most of the cases. Considering situation 1, it is possible to detect the overall number of days of anticipation with an RMSE of 1.3 in all the weeks (except the first one, which cannot be predicted as there is no previous data) and the best results are obtained using RF. When comparing situations 1 and 3, which predict the same dependent variables, results are generally better when using cumulative data, although the best overall value in week 5 is with RF and situation 3. However, when predicting specific values in specific exams, most recent data provided better results in most of the cases. In this case, the reported RMSEs are between 1.1 and 1.9 days depending on the task, and while the best result in week 2 is achieved with cumulative data (situation 2) and kNN, values obtained in situation 4 stand out in the rest of the cases. Regarding the algorithm, RF stands out in weeks 4 and 5, while DT achieves the best results in week 3. Thus, RF stands out in most of the cases considering the diferent cases.
Regarding the dropout analysis, several models were also defined depending on several situations (S) to analyze the predictive power using multiple sets of variables. Particularly, the situations defined for this case (all of them using cumulative mode) are: • S1. This model includes all the variables in Table 1 (prevail is used as dependent variable and the rest as independent variables). • S2. This model includes variables related to interactions with videos to predict dropout (prevail variable). The independent variables are uninterrupted_x, watched_x, and hsv_x. • S3. This model includes variables related to interactions with formative activities (form_x and n_c_f_e) to predict dropout ((prevail). • S4. This model includes variables related to interactions with videos and exam grades to predict dropout (prevail variable). The independent variables are uninterrupted_x, watched_x, hsv_x, and ex_x. • S5. This model includes variables related to interactions with formative activities and exam grades (form_x, n_c_f_e and ex_x) to predict dropout ((prevail).
Considering these situations, several models have been implemented and results are presented in
Table 3. From this table, it can be seen that it is possible to obtain an AUC up to 0.94 using all variables
(and above 0.8 from week 3). However, variables on video interactions (including help seeking using
1
2
3
4
5
0.69 0.50 0.50 0.58 0.54 0.61 0.52 0.50 0.59 0.54 0.59 0.56 0.50 0.59 0.54 0.50 0.50 0.50 0.50 0.50
0.75 0.60 0.53 0.71 0.74 0.73 0.60 0.53 0.73 0.73 0.66 0.62 0.67 0.72 0.69 0.60 0.50 0.50 0.57 0.73
0.81 0.61 0.70 0.80 0.81 0.88 0.64 0.69 0.84 0.85 0.74 0.63 0.70 0.84 0.86 0.69 0.60 0.50 0.63 0.87
0.88 0.71 0.72 0.88 0.88 0.92 0.72 0.73 0.91 0.90 0.82 0.71 0.61 0.89 0.92 0.80 0.71 0.81 0.73 0.93
0.94 0.70 0.75 0.74 0.92 0.94 0.72 0.76 0.92 0.93 0.86 0.83 0.79 0.92 0.94 0.86 0.72 0.84 0.74 0.94
videos) and formative assessments were not enough (AUCs up to 0.83 and 0.84, respectively), although
they increased up to 0.92 and 0.94 when adding previous summative grades, respectively. This result
indicates that previous performance is a key factor for predictive models, as reported in other works
(e.g., [
This work has analyzed the efect of deadline-driven behaviors and help seeking and the predictive models of deadline-driven behaviors and dropout. Deadline-driven behavior was found to be a common behavior in MOOCs as about one third of the learners delivered their tasks in the last two days. In addition, it was found that these behavior have an impact on dropout and e.g., almost 50% of the learners who submit their tasks in the last 24h in the first two test drop out the course by the next exam. Furthermore, predictive models were developed to forecast the number of days in advance learners submitted their tasks. Results showed that it is possible to achieve accurate predictions with an RMSE between 1.1 and 1.9 days depending on the task, and an overall RMSE of around 1.3 days when predicting the average value of all tasks. As for the the case of help seeking, help was mainly sought through videos, and there were not many identified cases of help seeking using the course forum. With regard to drop out prediction, accurate models were obtained, particularly when using previous summative grades. Particularly, an AUC up to 0.94 was achieved using all variables and above 0.8 from week 3. Nevertheless, variables related to interactions with videos and formative exercises provided accurate results only at the end of the course, which limit their impact when they are not combined with other variables related to student performance.
Despite the aforementioned findings, there are some limitations that are worth mentioning. First, results have been obtained using a single course, which limits the generalization of the findings. In addition, deadline-driven behaviors might also be impacted by the number of days available for the task, and more contexts are needed to address that. Furthermore, help seeking variables are limited as they only focus on the activity within the platform and learners may seek help from other sources. In addition, other possible models for the detection of help seeking might be used. Moreover, the definition of dropout has been established considering the completion of all exams, but other definitions could have been used and that could afect the results.
AS future work, it would be relevant to analyze more contexts to discover how results may generalize to other scenarios. Moreover, further analyses could be done to analyze the reasons behind submitting the tasks near the deadlines and/or dropout. In addition, it would be relevant to track gather more data about how learners take the exam and what additional sources they might use (including e.g., the use of Generative Artificial Intelligence). Finally, it would be relevant to carry out pilots in live courses to evaluate the applicability of these findings and how instructors can benefit of them to make an impact on their learners.
This work was supported by Universidad Carlos III de Madrid (UC3M) through the Grants for the Research Activity of Young Doctors of the UC3M’s Own Research and Transfer Program (ASESORIA project). Moreover, it was supported by FEDER / Ministerio de Ciencia, Innovación y Universidades - Agencia Estatal de Investigación through the grant PID2023-146692OB-C31 (GENIE Learn project) funded by MICIU/AEI/10.13039/501100011033 and by ERDF/UE, by the UNESCO Chair of “Scalable Digital Education for All” at UC3M and by the grant RED2022-134284-T funded by MICIU/AEI/10.13039/501100011033.
The authors have not employed any Generative AI tools. [13] A. Sheshadri, N. Gitinabard, C. F. Lynch, T. Barnes, S. Heckman, Predicting student performance based on online study habits: A study of blended courses, in: Proc. 11th International Conference on Educational Data Mining, Bufalo, NY, USA, 2018, pp. 1–7. [14] A. H. Nabizadeh, J. Jorge, S. Gama, D. Gonçalves, How do students behave in a gamified course?—A ten-year study, IEEE Access 9 (2021) 81008–81031. [15] V. Aleven, I. Roll, B. M. McLaren, K. R. Koedinger, Help helps, but only so much: Research on help seeking with intelligent tutoring systems, International Journal of Artificial Intelligence in Education 26 (2016) 205–223. [16] J. Maldonado-Mahauad, M. Pérez-Sanagustín, P. M. Moreno-Marcos, C. Alario-Hoyos, P. J. MuñozMerino, C. Delgado-Kloos, Predicting learners’ success in a self-paced MOOC through sequence patterns of self-regulated learning, in: Lifelong Technology-Enhanced Learning: 13th European Conference on Technology Enhanced Learning, EC-TEL 2018, Leeds, UK, September 3-5, 2018, Proceedings 13, Springer, 2018, pp. 355–369. [17] R. Pelánek, Metrics for Evaluation of Student Models., Journal of Educational Data Mining 7 (2015) 1–19. [18] S. Shah, A. Mumtaz, A. Chughtai, Subjective Happiness and Academic Procrastination Among Medical Students: The Dilemma of Unhappy and Lazy Pupils., PRAS Open 1 (2017) 1–17.