Classical evaluation models were based on measuring the grades of a test or set of tests performing along a specific course. The method had drawbacks, and continuous evaluation is nowadays a preferred method. Continuous evaluation performs a continuous measure of the progress the student fulfils to achieve learning outcomes. However, and in practice, continuous evaluation systems usually consist - or at least, include - a set of assessments along the course which are graded and computed. In this article, we demonstrate the relevance of the first test grade in terms of withdrawal. We make use of a quite uncommon method in learning analytics, such as survival curves. The method is commonly used in other fields, and in particular in medicine, to detect differences among populations under study. Practical implementation will be carried out by analyzing a dataset containing higher education online university courses. Results show that the students failing the first grade show not only higher withdrawal rates, but also disengage earlier from the course. This evidence should reinforce the need to design actions targeted to this group, but also to use an initial test as a measure of engagement.
Dropout in general, and withdrawal in particular, is one of the core problems of higher education institutions. Dropout means inefficient use of resources, and at the same time, implies frustration for students who leave either the system as a whole or a specific subject in particular.
This fact was one of the topics the Bologna process would aim to resolve, in a general framework of trans-European coordination [1]. As a specific factor, continuous evaluation was encouraged. Instead of the classical approach, where the students had a reduced set of tests to evaluate performance, competencies were considered the key, and continuous evaluation was encouraged.
From a practical point of view, this continuous evaluation would require “the evaluation of a subject through daily classwork, course-related projects, and/or practical work(s) instead of a final examination system.” [2]. In practice, however, tests are still being performed and competence acquisition is often measured through grades gathered in a set of tests.
In this scenario, we hypothesize that continuous evaluation is conditioned in some way by the first of the evaluative assessments. Although its relative weight can be minor compared to the whole set of activities, a low grade can have a discouraging effect on the student. To validate our hypothesis, we raise the following research question: RQ: To what extent does failing the first evaluative assessment condition withdrawal decision of online students?
As it can be seen we look for an answer that goes beyond a pure affirmative question, indicating that there can be an influence. We aim to compare withdrawal depending on this first test, quantifying the specific impact.
To provide additional significance, we will analyze an assorted set of courses from an open database provided by the Open University. It includes data about a set of courses that have not been specifically designed for our research, including 22 editions of 7 different courses with over 30.000 total enrollments. Results will show that the students failing this first test show not only a higher withdrawal ratio but also tend to abandon the course earlier. 2
The theoretical models behind dropout were established around 1975. Works by Tinto [3] establish the first model on the topic. Tinto’s model was known as the student integration model. It included both academic factors related to the student herself and factors related to the institution. As a whole, the model considered a set of interactions that conditioned the decision to drop out.
After this initial model, we can find different works that rely on this theory. [4] introduced the ‘student attrition model’ which relies on the concept of behavioural intention, where dropout is conditioned by a mixture of factors, which include academic, social-psychological, environmental, and socialization factors.
None of these theories considers specifically online studies. [5] performs an analysis, based on the previous theories, and conforms the ‘composite persistence model’. In this model, academic performance and dropout are finally a combination of student characteristics, student skills, external factors, and internal factors. The three models abovementioned are the most cited references for the study of dropout but are not unique [6]. We can also cite models by Kember [7] and Lee and Choi [8].
The core of these models is centred on university dropout, which could be defined as ‘leaving the university study in which they have enrolled before they have obtained a formal degree’[9]. However, this phenomenon can also be analyzed at a micro-level. In particular, considering the fact of a student leaving a course she is enrolled in. In this case, the term withdrawal is preferred, although compilation works show that the formal definition is unclear. 78% of the recent studies do not provide a clear definition of the term [6]. There are also no specific theoretical models for withdrawal.
For the sake of our research, we will consider it as “voluntary or involuntary removal from a course before completion”, a consistent definition with references in the literature [10], [11]. It is noticeable that the concept includes not only the decision to abandon the course but also considers the time, as withdrawal is carried out before the end of the course.
Withdrawal analyses are normally set up on specific course analyses. Besides, we can find a mixture of quantitative and qualitative analysis. Early works related to withdrawal in online environments detected that the pressure of work, technical problems, and lack of time were withdrawal determinants[12]. More recently, focuses on family and organizational support, and course satisfaction and relevance.
Despite the relevance of time in withdrawal, time analysis is uncommon. Most studies are limited to a classification problem, aimed to determine variables the influence whether a student withdraws or not. Among those references to studies considering the relevance of time, we can cite [13], [14] which are focused on university dropout. Focused on a specific course, we can find a MOOC case example[15]. It must be pointed out that most studies focus on the institutional level, and not specifically on withdrawal at the course level.
Among those techniques to approach the problem, we can find correlation analysis, classifiers – both Bayesian and different decision trees -, variance analysis, logistic regression, support vector machines, neural networks, or machine learning techniques. These techniques are found both at university and course level and also in traditional university courses and MOOCs [16], [17]. MOOCs are one of the fields where withdrawal has been more analysed due to its higher rates[16].
Despite survival analysis is commoly used in other disciplines[18], references to survival analysis techniques in e-Learning problems are not so common. The basics behind the technique are described in [19]. The interest on it is more than justified, due to its focus on time – which is particularly relevant when analysing withdrawal – but also for providing better results that classical approaches in terms of prediction[20].
[20] suggests that more research should be performed using this approach. Among those works focusing on time, we can cite [13], [21]. Results in [21] indicate that the beginning of the course is a critical moment that concentrates a high number of withdrawals. [13] performs a survival analysis over time from a university-level perspective, with results showing that grade point average at first semester, gender, and location are relevant for determining university dropout.
Whichever method, the relevance of early activity is considered in different studies. Early activity in general is considered a predictor of final course performance[22]–[24]. Assignment grades in particular constitute a strong predictor of the final performance in MOOC courses [25].
In this scenario, we analyze the specific impact of early grades in evaluative assessments on the withdrawal decision of the student.
From a methodological perspective, two critical factors arise. First, the technique to be used. Second, a specific database to work with. Regarding the method, and due to the relevance of time, we will map our study as a survival analysis problem, as described in Section 3.1. Regarding the data, we will make use of a publically available database created by the Open University. Details of this database are included in Section 3.2.. 3.1
Survival analysis is ‘a collection of statistical procedures for data analysis where the outcome variable of interest is time until an event occurs’ [19]. The method is commonly used in other disciplines such as medicine, where survival time or time to relapse is under consideration. A really interesting view of the technique with a practical approach can be seen in a series of articles[19], [26]–[28].
References to survival analysis are scarce in the field of education in general and withdrawal in particular. As indicated in Section 2, we can cite a couple of analyses of university dropout [13], [14] and another focused on MOOC courses[15].
Two specific aspects are needed to perform survival analysis, which are the event under consideration, and the time to event. The event under consideration will be the fact of withdrawing, while the time to event will be the number of days the student remains enrolled in the course.
As different courses will be analyzed we will consider as t=0 the initial day of the course. Times above t=0 will be interpreted as the number of days after the course starts. Negative values reflect a withdrawal after enrolling but before the course effectively starts.
As specific tools we will make use of Kaplan-Meier curves to visualize and analyze the relevance of the variable under analysis, looking for statistical significance and clear interpretation[19]. Statistical validation will be performed considering the null hypothesis that different groups generated based on the grade of the first assessment share the same hazard functions. Log-rank test (in particular, Peto’s) will be used for being more robust, and also as it provides more weight to earlier events [29]. As a limitation, Kaplan-Meier does not allow quantifying hazard. Hazard can be computed by using a simple non-parametric method, such as Nelson-Aalen[19].
To quantify the impact, and considering we are not segregating populations based on multiple parameters, we can use a non-parametric method. In particular, we will use the Nelson-Aalen method to estimate cumulative hazard. Although non-cumulative hazard at a specific time can also be computed, cumulative estimation is preferred for being more stable. 3.2
The search for a dataset that allows for analysis linked to our RQ has lead us to consider the public dataset offered by the Open University[30]. At the highest level, this dataset provides information about 22 editions – namely presentations in the OU nomenclature – of 7 different courses. All courses present at least two editions. A total of 32,593 students are enrolled in these courses – modules in the OU nomenclature -.
This database includes both personal and academic data of the students under consideration. For the sake of our purpose, specific information to determine withdrawal – and in particular, withdrawal date - is included. Regarding assessments, the dataset also includes the whole set of evaluative activities linked to every course, with its weight and grades obtained by the students in the different editions of the course.
Table 1 includes information about the first evaluative assessment of the different courses under analysis, as well as its weight. We also provide the total course duration.
Module
With this information, we can map the problem as a survival analysis problem as indicated in Section 3.1. It is also noticeable that the data used is suitable for performing Kaplan-Meier analysis. The OU has an active policy to manage dropout. Withdrawal time is always recorded. Due to this fact, independence of censoring and survival is guaranteed. 4 4.1
Before quantifying the potential impact, a detailed analysis must be performed to determine whether passing students and failing students of the first test show statistically significant differences in withdrawal patterns. Figure 1 shows the results of the Kaplan-Meier estimates comparing both groups:
Considering that courses have different withdrawal ratios, we have analyzed also curves on a per-course basis. Results are shown in figure 2, where curves are shown including confidence intervals:
Statistical comparison for all groups results in relevant differences (p<0.005) in all cases, indicating both groups show different patterns regarding withdrawal. Table 3 reflects final withdrawal ratios for the generated groups and different courses, including the increase factor to facilitate comparison.
While survival curves provide group comparison in terms of survival probability, we can still get deeper into analyzing withdrawal hazard. Kaplan-Meier estimates do not provide this information, and we have to make use of specific methods to compute it. In particular, and considering that populations are segmented based on a single covariate, and we have no assumption about distribution, we can use a non-parametric estimator. In particular, we use Nelson-Aalen.
Nelson-Aalen is used to compute the cumulative hazard risk, understood as the probability of a student withdrawing from the course within a small interval of time, assuming she has survived up until the beginning of that interval. Cumulative hazard is preferred to point-wise estimations for being more stable.
In our case study, plots over time for individual courses have been plotted in Figure
As it can be seen, those students who fail the first test, show higher withdrawal rates in the long term, but also a relevant increase in early withdrawal.
The research carried out contributes in two specific ways. First, it demonstrates the utility and potential application of survival analysis applications in e-learning. Second, it provides insight into the relevance of the first assessment in withdrawal.
Focusing on the RQ stated, results in section 4 clearly show that there are relevant differences in withdrawal among those students who fail the first test and those who pass it. This difference is reflected in Figures 2 and 3 for individual courses.
Regardless of the course, global withdrawal ratios are higher for those students failing the first test. As Figure 1 shows, the mean difference in survival at the end of the course is 2.57 higher for those students who pass the first test. When looking at individual courses, the increase in withdrawal at course end can be 4.22 times higher as reflected in Table 3. Besides this difference in final withdrawal, Figure 3 shows also an interesting insight. Much of the above difference is based on a much higher early withdrawal. Before going deeper into this fact, it must be pointed out that these tests are made in an early period when comparing to course duration. Data in Table 1 show that – except for course GGG - the test is performed around the first month of the course, in a course lasting for around 9 months. Also, some of the assessments do not even compute for global course grade and are under 20% in all cases.
With this data in mind, our results would indicate that the first assessment of a course has an interesting predictive power, regardless of its weight and even time. From a learning analytics perspective, the group of students who fail the first test are suitable for targeted interventions aimed to retain them in the course. This result is aligned with the literature reflecting the impact of early activity in a broader sense [22]–[24].
From a methodological perspective, the method exposed fits the suggestion to explore survival analysis in e-learning scenarios[20] with a specific application to withdrawal analysis. It is at least noticeable that a method that is common in medical research shows really few references in e-learning. The authors are open to collaborate in research lines following this idea, and in particular, linked to the analysis of the impact of different factors on withdrawal. [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20]