In online education, students are expected to be independent learners who can self-regulate and re ect on their activities during the learning process. However, not all students have self-regulated learning (SRL) skills, and students with weak SRL skills tend to underperform in distance learning environments. The aim of our pilot study was to promote selfregulated learning in online education by triggering tailored SRL interventions automatically. As a rst step toward, we constructed a quantitative research design where 58 students participated in 1) learning about introductory descriptive statistical concepts and 2) interacting with a self-paced online learning software throughout the experiment. We used the participants' action log les as a dataset to extract generalizable features, including pretest grade, quiz grade, reading time, and posttest grade. Then, we trained a random forest regressor model to predict student outcome (posttest). The correlation between actual and predicted posttest score was r = .576, indicating promise for accurately predicting and intervening. In the next phase of this work, we will apply SHAP (SHapley Additive exPlanations) to personalize SRL interventions by recommending each student to review the single topic that most negatively contributes to predicted posttest grade.
Students with di erent online learning skills, academic
performance, and levels of technology experience may go through
hardships to become autonomous learners who academically
succeed in ever-growing online learning courses in
universities. Previous studies demonstrated that successful
acaCopyright ©2021 for this paper by its authors. Use permitted under
Creative Commons License Attribution 4.0 International (CC BY 4.0)
demic achievement is strongly connected to the level of
students' self regulative abilities, in which students take
initiative in the learning process [
Self-regulated learning (SRL) skills are techniques for
achieving academic success by regulating a student's own actions
and decisions during the learning process [
However, distance learning poses threats to student
success, especially students without SRL skills, since they are
supposed to learn and complete assignments on their own.
Having SRL skills in e-learning environments is particularly
important for students to achieve academic goals, but SRL
ability is not an inherent skill that every student possesses [
Our study responds to the need for SRL support by
promoting self-regulated learning in an online learning environment
via interventions automatically customized for each student.
Many works demonstrated that aspects of student
performance and experiences can be predicted by using students'
action log les [
We take a step forward by using predicted student outcomes to trigger personalized SRL interventions in online courses. In our study, SRL interventions consist of suggesting that students engage in speci c SRL behaviors, such as reviewing particular readings or quizzes that contribute to a lower predicted posttest grade. We emphasize triggering tailored interventions automatically for each student since it can help a particular student at the right time with interventions chosen by a machine learning model. The machine learning model in this case is designed to predict students' outcomes (speci cally, posttest grade), while interventions are based on model explainability methods intended to discover SRLrelated reasons why the model made a particular prediction. Our study involves three conditions: the model training condition, the treatment condition, and the placebo condition. However, in this paper, we primarily discuss an investigation with the model training condition, in which we collect the data from students' action log les to train the machine learning model that will ultimately trigger SRL interventions in the treatment condition. We propose a method of 1) predicting student outcome (posttest grade) in online learning using machine learning techniques, and 2) triggering tailored SRL interventions for students by implementing SHAP (SHapley Additive exPlanations) analysis for the predicted student outcome. We focus on the methodological step #1 in this paper, but discuss ongoing work toward step #2. We evaluate aspects of this methodology in a study where students learned about introductory descriptive statistics concepts and interact with a self-paced online learning environment.
In our pilot study, we attempted to answer the following research questions:
RQ1) How much do generalizable learning features (e.g., quiz/test grades) extracted from e-learning platforms predict student outcomes in machine learning models? RQ2) Is it possible to suggest each student to review speci c topics from the learning module by triggering SRL interventions at the right moment?
Previous studies have demonstrated the importance of
selfregulated learning (SRL) in academic contexts [
In order to analyze and measure students' self-regulatory
behaviors in distance learning, researchers have used students'
action log les in various ways. For instance,
Maldonadomahauad et al. [
Our study's primary objective is to apply SHAP analysis to
extend current modeling methods so that they can support
SRL in online education. In this regard, Mu et al. [
Our overall study design consists of an experiment with three conditions: the model training condition, the treatment condition, and the placebo condition. In this paper, we primarily focus on the model training condition where we collected data for machine learning model training and implemented SHAP analysis. The other two conditions (treatment condition and placebo condition) are currently collecting data with interventions from the model described here. Participants in the experimental condition group will receive tailored SRL interventions based on machine learning predictions, whereas students assigned to the placebo group will get SRL interventions almost identical to the ones from the experimental condition group except not based on machine learning predictions. In all conditions, including the model training condition we study in this paper, participants engaged in learning about introductory statistical concepts by using custom web-based online learning software (Figure 1). Figure 2 below illustrates an overview of our research procedure.
At the start of the study session, students completed a brief survey regarding their demographics and prior academic history. Following the survey, participants took a 10-minute pretest and used the self-guided learning session for up to 1 hour. The self-paced online learning environment included 12 di erent illustrated statistical readings, and one quiz to go with each reading (see partial screenshot in Figure 1). Throughout the learning session, students were not required to complete all the modules, and were allowed to complete the components more than once if they wanted to. After 30 minutes elapsed during learning session, each student received a simple baseline SRL intervention in which they were told which topics they had not yet viewed, or a list of topics in order from least- to most-viewed if they had viewed them all. Hence, in the model training condition, the intervention was not based on machine learning. The message prompt and the corresponding list of topics would, in theory, help to make aware patterns in the student's learning behavior up to that point in time, which could lead to reecting more deeply about their current learning trajectory. Based on this information, we expected the student would be able to make more informed decisions on how to regulate their learning by adapting their future learning behaviors. This would allow for more systematic and controllable decision-making processes to determine which topics to visit or review, and make salient what areas they may feel to have not su ciently studied. This intervention repeated every 10 minutes thereafter until the 1-hour learning session was over or the student chose to end it before the hour. Subsequently, they completed a 10-minute posttest with questions modeled after the pretest questions (but not identical).
We collected data from 58 university students who participated in our pilot study. Students were required to have completed either zero or one college-level statistics course, but not more, to avoid inappropriately matching introductory material to expert students. Students' event log les were extracted from the online learning system, which recorded their learning activities in real-time including information needed to provide interventions. These log les contained activities that were recorded during the students' interactions with every stage of the web-based online learning software.
Following data collection, we extracted each student's various features, including some attributes related to SRL behaviors, in order to use them as predictors for training the machine learning model. Students' interactions were distributed across various log les for each possible learning activity, quiz and test scores, time spent reading, and other les. For feature extraction, we merged all students' feature outputs into a single table which we later used for machine learning data analysis. Extracted features consisted of:
Pretest grade (mean of multiple-choice question correctness)
Time spent reading each 12 descriptive statistics readings Number of times the student reviewed statistical readings/quizzes Number of events where the student clicked the button for going back to the Main Topics Menu Whether a student looked at other windows/tabs (i.e., the learning environment lost focus)
Note that there were 12 versions of each quiz grade and reading time feature, each extracted from one of the readings or quizzes. Features following the learning session were outcomes, rather than predictors; in particular, we focus on posttest grade in this paper.
From the extracted feature data, consisting of 58 instances (1 per student), we discovered some of the students did not seem to try their best to participate in our study. Five students did not attempt to take any of the quizzes from the learning session and their posttest scores were lower than their pretest scores. Nevertheless, we did not exclude these observations from our dataset since these students may re ect future treatment condition students and real-world classroom students.
We performed exploratory data analysis on the feature dataset and will discuss our ndings in the Results (Section 4).
Initially, we trained our model using a decision tree regressor
to predict student performance (posttest score) using quiz
grades, reading times, and pretest score (in Python using
Scikit-learn [
Using posttest scores predicted by the Random Forest Regressor model, we implemented SHAP analysis to interpret the model prediction. We calculated SHAP values using the tree explainer to explain model predictions and will use these explanations (in the next steps of the project) to trigger individualized SRL interventions to meet each student's need. We sorted calculated SHAP values in ascending order to determine speci c features that contribute to getting a lower posttest score.
In this section, we present ndings on each stage of our research process in detail.
We calculated Pearson correlations as a rst step of data analysis in order to measure the strength of a linear relationship between posttest grade (target variable) and other potential predictor variables. We expected that clear relationships would be needed for the machine learning model to succeed given the small dataset size. Among the predictor variables, pretest grade had the highest correlation (r = .530), indicate at least one sizable|if unsurprising| relationship in the data. From this we discovered that students' initial knowledge (as evident from pretest score) was closely related to posttest grade.
However, quiz grade features and reading time features had promising positive correlation coe cients (up to r = .395). Though most features these were not statistically signi cantly related to posttest score given the large number of predictors and small dataset size, trends indicated that these were promising indicators for the success of machine learning methods. Since our goal was to suggest speci c reading/quiz topic that students should review, we included 12 quiz grades and 12 reading time features for our predictors, along with pretest. Some features related to SRL had correlation coe cients trending in the expected direction, but in order to make our machine learning model simple, we did not use them as predictors. Moreover, we wanted to use highly generalizable feature types that could be easily extracted from diverse online learning platforms. Within the chosen predictors, we checked whether pretest grade and posttest grade were normally distributed. We plotted frequency histograms for pretest grade and posttest grade features. Corresponding histograms were relatively bell-shaped and symmetric about the mean values, so we concluded that pretest and posttest grade features are normally distributed. This is essential to avoiding ceiling or oor e ects for analysis of learning.
Initially, we extracted 11 additional features from students'
log les, but we used only pretest grade, quiz grade, and
reading time as features for predicting posttest grade. Since
we had a small sample size with 58 observations, we had
to reduce the number of predictors to make the model
simple. However, in future work with more participants we
plan to use more predictors, such as features related to
speci c SRL constructs extracted via coherence analysis [
In Figure 3, the left grey boxes represent the overall study process that students went through in our experiment. On the right, the gure shows the composition of a student's recorded action log le as a whole, which is composed of demographic survey, pretest, descriptive statistics surveys, reading times, log, browser tab focus, and posttest les. The diagram shows from which log les the predictors and a target variable were extracted.
Since students were not asked to complete all the reading/quiz components from the learning session, there were many participants who did skip several readings or quizzes. In these cases, we assigned -1 for corresponding reading times and quiz grade features to di erentiate the cases. Table 1 below shows a statistics summary of extracted features, including minimum, maximum, and possible values that the features can take. DEMOGRAPHIC
SURVEY PRETEST app : app
SELF-PACED ONLINE LEARNING
POSTTEST Pretest grade
Posttest grade Quiz Grade
Reading Time :
We trained the random forest regressor model using pretest grade, quiz grade, and reading time features to predict posttest grade. We used 5-fold cross-validation for validation, as mentioned earlier, and measured R2, root mean squared error (RMSE), and Pearson's r for model evaluation metrics. During the model training, we set the maximum depth of the trees to be 4 and xed the random seed in order to produce a consistent outcome. We used 4 as a maximum depth of the tree because the results became notably more stable in initial tests (with a partially collected dataset) by reducing the depth of the trees.
After training, we evaluated the model and obtained the following results: 1) mean R2 value within 5-fold crossvalidation was .262, 2) mean RMSE was 15.17 (on a 0{100 posttest grade scale), and 3) mean Pearson's r was .576. From these observations, it is noticeable that our mean R2 value was far from perfect averaged across folds. However, given the small data size we have, the trained model works relatively well. Furthermore, the accuracy was stable across folds: across the 5 folds, the standard deviation of R2 was .067 and the standard deviation of RMSE was 2.20.
The objective of our pilot study was to promote self-regulated learning in online education by triggering individualized SRL interventions using machine learning techniques and SHAP values. To accomplish this, we began with extracting learning relevant features from 58 students' action log les which recorded students' learning traces during the online learning process. Using three types of predictors (pretest grade, quiz grade, and reading time), we trained a random forest regressor model to predict student outcome (posttest grade). Using the predicted student outcome, we applied SHAP technique to trigger personalized SRL interventions by recommending each student to review the single most learning session that contributes to getting a lower student outcome. As a result, we developed a exible way of triggering individualized SRL interventions in a digital learning environment.
Our presented method is noteworthy in several aspects. Firstly,
our proposed technique can be used as a means to help
students who lack SRL skills, technology experience, or have
weak academic performance to develop SRL skills in e-learning
environments. Especially with the unexpected outbreak of
the global COVID-19 pandemic, a large number of students
are using online learning platforms as a way of learning and
assessing learning outcomes. The need for methods to help
students learn e ectively online is thus increasing rapidly;
we hope our proposed method can contribute to the
technology development of helping students in online education.
Secondly, our approach is original in that we integrated
complex machine learning techniques (i.e., a complex, non-linear
regression model) with SHAP as a recent machine learning
interpretability method to decide how to intervene.
Previously, this idea had only been explored on archival data [
The presented method can be useful in some respects. Students can receive personalized SRL interventions, which can help them to improve their outcomes and develop SRL skills in online learning. In particular, we can support speci c groups of students who need the most help in online education. For instance, students with no prior experience using e-learning platforms tend to be less experienced with monitoring their learning activities. This group of students can bene t from the implementation of our method of triggering tailored SRL interventions since they would be exposed to suggestions that encourage them to engage in speci c SRL activities.
Our method is highly exible since we required only 58 students as a training sample (which is su cient here, but could be expanded) and 3 types of predictors extracted from event log les for training the model and administering SRL interventions. In particular, the three predictor types are generalizable features (pretest grade, quiz grade, and reading time) which can be easily extracted from many online learning platforms. Moreover, most e-learning platforms store students' action in log les, so we expect that it is feasible to introduce our SRL intervention mechanism into online learning platforms. Thus, we expect this method to be applicable in a variety of computer-based learning contexts. Our method will also allow researchers to explore the causal nature of educational interventions driven by machine learning models. For example, are students who spend more time on a speci c topic doing well because of time spent on that topic (as implied by a causal interpretation of the model), or do students who do well also happen to spend time on that particular topic because of some unobserved trait? Our explorations of interventions based on the features will allow us to manipulate the inputs of models and explore the nature of these connections.
Even though we had a fairly stable model accuracy within the sample size (N = 58), model accuracy might be improved substantially with more data. The current training data size limits the feasibility of extracting a large number of specialized features that might only apply to a small fraction of students. Moreover, there may be complex interactions between students' learning behaviors on di erent topics that require additional data to uncover. Likewise, if our model is applied into online courses where students take the course for no credit, then the expected e ectiveness of our method might be weak since students may not be motivated to study in the same way that students taking actual courses for credit. Further data is needed to explore these e ects.
Collecting additional data would also a ord the opportunity to explore new types of features that could address some of the unexplained variance in our model. For example, there are many features unrelated to SRL, such as prior experience level, perceptions of statistics, and others that might improve model accuracy. We hope to address these gaps in the model in future work. Notably, the model explanation method used here will allow us to still provide interventions based on SRL features even with other features included. Improved model accuracy might be most helpful when determining when to provide interventions and to whom, unlike our current approach in which every student receives an intervention at predetermined points in time.
Notwithstanding the aforementioned limitations, our pilot study made an e ort to throw light on the matter of students struggling within online courses through integrating machine learning and SHAP to promote self-regulated learning in digital learning. In the near future, we hope students in the treatment condition produce better learning behaviors and academic outcomes when our proposed method is applied.
This research was supported by a grant from the Technology Innovation in Educational Research and Design (TIER-ED) initiative at the University of Illinois Urbana{Champaign.