A plethora of data on students' activities and interactions is acquired through the use of virtual learning environments. The analysis of gathered data and the prediction of withdrawal and academic success has been an ongoing point of discussion in the field of Learning Analytics. The main aim of this paper is an early prediction of students' academic performance in terms of the final grade, while also targeting the early identification of students at risk of failing the course. We introduce new handcrafted features encompassing interaction patterns of students' virtual learning environment usage, and highlight their predictive powers in monthly predictions. The experiment showed that a combination of the proposed features improved the Root Mean Square Error by up to 7.2 grade points compared to a model without the proposed features. Improvement of up to 0.52 was also observed in the coeficient of determination. Proposed model's accuracy of predicting students at risk of failure reached 74% at the end of April.
Virtual learning environments (VLEs) have become all-important tools for educational
institutions in the recent past [
The analysis of data collected by VLEs has received much research attention in the last
decade, for its potential to impact and improve the teaching-learning process and address
significant challenges in education, including identifying students at risk of failing or withdrawing
from their respective courses. Many research endeavors have applied machine learning-based
approaches, which confirmed that data from VLE activity logs present good predictors for
forecasting students’ academic performance [
In summary, the key contributions of this work are threefold: (1) We present an insightful prediction of student academic performance in terms of final mark forecast at a fine-grained level, i.e., month-wise granularity, focusing on the early identification of students at risk of failing the course. (2) By introducing new handcrafted features encompassing interaction patterns of students’ VLE usage, we highlight their predictive powers, and provide an additional understanding of the characteristics of at-risk students, which builds on findings from previous work. (3) The proposed features, on average, improve the Root Mean Square Error (RMSE) of predicted grades by 3.6 in all observed months, with improvement of up to 7.5 grade points in particular months. Compared to the models without the proposed features, our model improved the explained proportion of observed variation in grades, with a mean R2 improvement of 0.245.
The rest of the paper is organized as follows. Section 2 presents a literature review of related work. The methodology is outlined in Section 3. Section 4 reports the results of the experiment. In Section 5, the results are discussed and compared to approaches from related work. Section 6 concludes the paper with summarized remarks and proposed future research directions.
Predicting students at risk of failing a course with a classification approach has been explored
by [
Classification studies focused on student success (or failure) recognition at diferent times
in the semester report high accuracy of their predictions. Accuracy is defined as the number
of correct predictions divided by the total number of predictions, while precision refers to the
number of true positives divided by the total number of positive predictions. On the OULA
dataset, Kuzilek et al. [
This study uses an experimental design. The experiment was conducted in Python 3.8.5 in a Jupyter notebook 3.8.5 environment, using Pandas 1.2.4, Numpy 1.19.2, Sklearn 0.23.2, and Matplotlib 3.3.2. Statistical analyses were performed using Jamovi 1.2.27.
The OULA dataset [
Study Prediction
Features
Time Interval Results
Data preprocessing included several strategies and techniques, with the main goal of selecting data for analysis and computing features from the data. First, tables with the course, assessments, and student data were filtered to the values connected to the DDD course. For each assessment, we joined the data (grades and submission dates) from the table with student assessments. Missing values were converted to zero. From the scores and assessment weights, first, the TMA was calculated, following the formula ∑︀ =1 ℎ × , with being the number of assessments. The final exam and TMA scores were used to calculate the final score. The resulting dataset was then joined with the students’ demographics. At this point, 1,803 students were observed from DDD course. Due to the high unregistration(i.e. student withdrawal) rate, only students who remained registered during the semester and submitted at least one assignment or took the final exam were kept. As we were not interested in student performance on a coursegranular level, students were removed who had previously already attended the course. The ifnal dataset included data for 777 unique students. Categorical data was encoded as numerical by converting it into dummy variables. The earliest date was extracted of a student’s interaction with a VLE. VLE interaction dates were transformed into DateTime format, and information about the date, weekday, and month of each VLE activity were added to the dataframe. The VLE activity logs were then grouped separately in months and weekdays for each student. The sum of clicks on each weekday and each month were calculated, as well as weekday (Mon–Fri) activity percentage, compared to weekend (Sat, Sun) activity. The overall sum of clicks and clicks on a particular activity type were calculated for the whole semester and each month, including the activities in September before the course started. We were interested in observing patterns of inactivity in the VLE. Thus, we included features for the number of days when a student was active in a VLE from the start of the semester, the number of inactive days since the start of the semester, as well as the highest number of consecutive active and inactive days during the observed time period, which were calculated as the longest intervals of consecutive days students interacted (or had not interacted) with the VLE. Eight copies of the dataset were created, where the metrics were recalculated for observations on the last day of each month between September and April. The final datasets included 24 to 68 features, with the number varying due to the available assessment grades and the number of previous months. Features that have, to the best of our knowledge, not been observed in related work on the OULA dataset, are presented in Table 2. Additionally, normalization of all inputs with MinMaxScaler was conducted for prediction, which scaled and translated each feature so that it was in the given range on the training set (i.e., between 0 and 1), enduring features with a higher value (e.g., number of clicks) do not, inherently, have higher importance than features with lower values (e.g., the first day of interaction).
Our dataset includes 467 (60%) male and 310 (40%) female students. Most students (570 or 73%) were in the age bracket of 0–35 years, 202 (26%) were aged between 35 and 55, and only five were older than 55. Most students (395 or 51%) achieved A level or equivalent education before taking the course; another 259 (33%) had education lower than A level, 114 (15%) had higher education qualifications, 3 had no formal qualification, and 6 had post-graduate qualifications. Based on the university’s performance standards thresholds, students were divided into seven brackets (bare fail: 0–14, fail: 15–29, bare fail: 30–39, pass 4: 40–54, pass 3: 55–69, pass 2: 70–84, pass 1: 85–100). Out of 777 students, 124 failed the class. By exploring students’ activity, measured with the sum of clicks in VLE, and their success in passing the course, we found out the following. The hypothesis for normal distribution of the number of clicks in the VLE was rejected with the Kolmogorov-Smirnov (KS) test (p<.001, D=.199). The Man-Whitney U (MW-U) test showed a significant diference (U=12763, p<.001) between the number of VLE clicks of students who passed and those who failed. Successful students interacted more with the VLE (N=653, M=1338, SD=1357), while students who failed the course interacted less (N=124, M=432, SD=345). The hypothesis for normal distribution of the date of the first interaction with VLE in the groups of successful and unsuccessful students was rejected with the KS test (p<.001, D=.24). The MW-U test also showed a significant diference (U=28865, p<.001) in successful (N=653, M=-20.3, SD=6.5) and unsuccessful students (N=124, M=-17.5, SD=7.1) based on the date of their first interaction with the VLE, with 0 being the first day of the semester. Further insight into VLE interaction is presented in Figure 1, displaying the sum of students’ daily clicks in correspondence with course events. Correlation between students’ interaction with the VLE and the beginning of the semester is visible, as well as their increased interaction before the final exam in May. Students’ average activity in the VLE during the week was highest on Thursdays and Fridays, while the least activity is observed on Tuesdays. The trend of increased clicks on Thursdays and Fridays and the lowest activity on Tuesdays was similar in groups of successful and unsuccessful students. On average, students interacted with the VLE more over the weekend than on Monday, Tuesday, or Wednesday. The mean share of weekday clicks through the semester was 68.5%, meaning almost one-third of interactions in the VLE were conducted at the weekends. Observing all VLE activities in the semester by activity type, students most commonly clicked on forums (M=427), homepage (M=303), subpages (M=170), and outcontent (M=145).
We observed diferences in academically successful and unsuccessful students through the pro- posed metrics. The mean values of the percentage of VLE clicks over weekdays, average daily VLE clicks before the semester, the number of active days, and the number of consecutively active days, were re higher for successful students. On the contrary, the mean values of the overall number of inactive days and the overall number of consecutively inactive days were higher in the group of unsuccessful students. Comparing the ratio of weekday clicks, the mean number was higher for successful students (M=.688) than unsuccessful (M=.665), suggesting they were more active during the weekdays than at the weekend. The hypothesis for the normal distribution of the weekday clicks ratio in groups of successful and unsuccessful students was confirmed with the KS test (p=.187, D=.0401). The assumption of variances’ homogeneity was violated using Levene’s test (F=19.7, p<.001). Thus, Welch’s T-test was used, and no significant diference (t(111)=-1.86, p=.065) was observed between the weekday clicks ratio in the groups of successful and unsuccessful students. Further, the average daily pre-semester clicks were observed. The hypothesis for the normal distribution of average pre-semester clicks in groups of successful and unsuccessful students was rejected with the KS test (p<.001, D=.229). The MW-U test showed a significant diference (U=30219, p<.001) in daily VLE clicks presemester between the observed groups. Before the semester, successful students had interacted daily more with the VLE (N=653, M=4, SD=5.7) than unsuccessful students (N=124, M=2.3, SD=3.1). The mean daily number of pre-semester clicks for successful students (M=4) was almost double compared to unsuccessful students (M=2.3). Additionally, the KS test rejected the hypothesis for normal distribution of the overall number of active days in groups of successful and unsuccessful students (p<.001, D=.0995). The MW-U test showed a significant diference (U=10376, p<.001) between the overall number of active days of students who passed and those who failed. Successful students interacted with the VLE on a more frequent daily basis (N=653, M=101, SD=50.9), while students who failed the course (N=124, M=42.3, SD=24.8) interacted with the VLE less often in terms of daily accesses. The KS test rejected the hypothesis for normal distribution of the overall number of consecutive active days in groups of successful and unsuccessful students (p<.001, D=.0261). The MW-U test showed a significant diference (U=-9.34, p<.001) between the overall number of consecutive active days of successful and unsuccessful students. Successful students interacted consecutively more with the VLE (N=653, M=11.1, SD=14.7), while students who failed the course (N=124, M=5, SD=3.5) interacted with the VLE in larger gaps. Oppositely, successful students had shorter consecutive inactive periods of VLE usage (N=653, M=22.2, SD=11.9), while unsuccessful students had longer (N=124, M=98.6, SD=66.1). The KS test rejected the hypothesis for normal distribution of the overall number of consecutive inactive days in the two groups of students (p<.001, D=.0209). The MW-U test showed that the diference was significant (U=12.8, p<.001)
Linear regression was used for prediction. Due to the small sample size, 5-fold cross-validation with scikit KFold was used to improve the estimate of the model performance. RMSE and R2 ︁√
1 nsamples
=0 Σnsamples −1 ( − ˆ )2, where were obtained for each period,calculated as RMSE = is the predicted value of the2 -th sample, and ˆ is the corresponding true value, and unadjusted ︀∑ corresponding tru e ==v11a(( l u−−e ^¯f)o)2r,twothaelrsea mˆ pilsest.he predicted value of the -th sample and is the 2(, ˆ) = 1 − ∑︀
To compare the impact of the proposed metrics on the performance, three other models were created – MB containing grades and VLE metrics, used most commonly in related work; MC containing demographics, VLE metrics, and grades; MD containing demographics and grades. A detailed overview of the features included in each model is presented in Table 3. Prediction models were compared monthly, as times indicated in Figure 1, with mean RMSE and R2 calculated from all folds for each monthly prediction. The evaluations were conducted after the end of each month on days 0, 31, 61, 92, 123, 151, 182, and 212. Accuracy in predicting failure was an important point of interest. It was calculated manually with the same formula as recall
TP
as proposed in [
The predictions of MA are visualized in Figure 2. We can observe a tendency in the improvement of predictions through time as more data were introduced to the model. A detailed overview of RMSE and R2 for data for each time period compared to the scores of other models are represented in Table 4. In April, after gathering all TMA, the model was capable of explaining 86% of the variability in final grades‘ predictions, with an RMSE of 8.0, which, in most cases, was within one grade bracket. MA was relatively successful in predicting the variability in data, even before the start of the semester (RMSE=14.2). In the first months, we observed the highest prediction errors in students whose actual grade fell below the passing threshold of 40 points. Compared to other models, MA covered more variance in predicted final grades in all the observed months. The mean RMSE was also lower in all observed months in MA compared to other models. The accuracy of grade predictions for students failing the test was relatively low, which is visible from Figure 2, where the grades around the 40-point threshold were the least accurately predicted, as well as from Table 4. In April, a month before the final exam, 74% of the failing students were detected. MA MB MC MD
Sep
The correlation between features was observed with Spearman’s correlation coeficient for MA. As expected, there was a moderate to a strong positive correlation between the final result and the assessment grades (correlation coeficient of TMAs was 0.44, 0.46, 0.65, 0.76, 0.79, and 0.77 for assessments, respectively). The ratio of weekday clicks had a very weak correlation with the final score (0.05), and the mean number of daily clicks before the start of the semester had a weak correlation (0.29). A moderate negative correlation was observed between the longest number of consequently inactive days and the final score. The negative correlation increased through the months, with the highest correlations observed in March (-0.46), April (-0.49), and May (-0.53). A weak positive correlation was observed between the longest number of consequently active days and the final score, again increasing through the months, with the highest correlations observed in March (0.32), April (0.32), and May (0.35). The number of inactive days in each month also correlated negatively with the final score, though the correlation was weak to moderate, increasing slowly from October (-0.33) to April (-0.49) and May (-0.51). Oppositely, a positive correlation was observed with the number of active days observed each month, again increasing from October (0.33) to April (0.49) and May (0.51). The exact contribution of proposed metrics is visible in the diferences of the RMSE and R 2 scores in all the observed months between MA and MB, where the only diference between them was in the introduction of proposed metrics in MA.
Diferent models were deployed to predict the final grades of the students based on their behavior, demographic characteristics, and VLE engagement patterns. The model with the proposed metrics, created only with data from VLE interactions, performed best in the case of the observed course in terms of RMSE and R2, while the accuracy of predicted failure was higher than comparing models until January. While the accuracy of failure prediction in MB, MC, and MD was very low in the first months, it improved visibly after February, reaching up to 3% higher accuracy than MA. Comparing R2 between the models, we can observe the predictions of MA approximated the real data points better than the other three models up to January. From February to April, MD was comparable to MA, with the latter having 0.02 to 0.05 lower R2. Comparing RMSE, MD reached the most comparable values from January to April, with 0.4 to 1.3 higher RMSE values compared to MA.
Though not directly comparable, the classification approach to predicting at-risk students in
related work reached a high accuracy; 84% accuracy at 20% of the course (day 50, mid November)
and up to 94% accuracy at the end of the course [
We observed increasing negative correlations between monthly inactive days and the final score, which can be explained by the lower weight of the first TMAs, as, until the end of November, students can achieve up to 25% of their final TMA grade with three out of six finished assessments. In later months, the weights of TMAs increase, and with them, the importance of interaction with material available in the VLE. Similarly, an increasing negative correlation between the longest number of consequently inactive days and the final score can be interpreted that the inactivity in the first months of the semester can be improved during the semester. However, due to the rising weight of TMA’s, inactivity in the last months of the semester contributes greatly to a student’s lower final score. We can interpret regular students’ activity (measured with consequent active days) in VLE as the prerequisite for a positive grade, as it does not actively contribute directly to their final grade. However, the lack of consequent activity has noticeable negative consequences. Correlations between the six new proposed metrics and the ifnal score in MA suggest that metrics measuring the daily clicks pre-semester, the number of active and inactive days, and the longest number of consequently active and inactive days reach weak to moderate correlation with the student’s final score, while the metric measuring the percentage of weekday clicks achieved very weak correlation, and was overall less suitable for prediction. The findings suggest that it is meaningful for learning analytic researchers to take into account the possibilities of calculating new, custom features from the available raw data.
This study aimed to predict students’ final grades by converting the problem into monthly formats and observing the performance evaluation based on diferent features presented to the model. The primary motivation of this paper was to draw attention to the exploration of previously unused metrics on an existing dataset, and observation of their impact on prediction. The metrics which contributed to this discussion on the OULA dataset are the mean daily number of pre-semester VLE clicks, the number of active and inactive days, and the number of consecutive active and inactive days. Main limitations of this work are: (1) Narrow focus and validation on one course, with a chance that the proposed features prove less useful on large-scale or generalized data; (2) Used linear model is simple, and can be optimized further to improve performance, especially around the passing threshold; (3) Limited practical implications, as withdrawn students and students retaking the course were not included. In future work, the introduction of various methods for identifying the use of the proposed features in diferent courses will be explored, as well as testing the proposed features with classification models.
The authors acknowledge the financial support from the Slovenian Research Agency (Research Core Funding No. P2-0057).