We explore how different elements of student persistence on computer programming problems may be related to learning outcomes and inform us about which elements may distinguish between productive and unproductive persistence. We collected data from an introductory computer science course at a large midwestern university in the U.S. hosted on an open-source, problem-driven learning system. We defined a set of features quantifying various aspect of persistence during problem solving and used a predictive modeling approach to predict student scores on subsequent and related quiz questions. We focused on careful feature engineering and model interpretation to shed light on the intricacies of both productive and unproductive persistence. Feature importance was analyzed using SHapley Additive exPlanations (SHAP) values. We found that the most impactful features were persisting until solving the problem, rapid guessing, and taking a break, while those with the strongest correlation between their values and their impact on prediction were the number of submissions, total time, and (again) taking a break. This suggests that the former are important features for accurate prediction, while the latter are indicative of the differences between productive persistence and wheel spinning in a computer science context.
Research on student modeling has identified various behaviors and
patterns related to learning outcomes and student success. One
construct has both a history of research outside of Educational Data
Mining (EDM) and is receiving renewed attention in the EDM
community. Known by the diverse names of grit [
Given the opposing academic outlook of this dichotomy,
understanding what differentiates productive from unproductive
persistence is of critical importance. The latter has been termed
wheel spinning in the literature and has been defined as "a student
who spends too much time struggling to learn a topic without
achieving mastery" [
In this paper, we add to the existing literature by exploring how different elements of persistence on computer programming problems may contribute to learning outcomes. We defined a set of features quantifying various aspects of persistence during problem solving and used predictive modeling approaches to predict student scores on subsequent and related quiz questions. We focus on careful feature engineering and model interpretation to shed light on the intricacies of both productive and unproductive persistence. By investigating these constructs within a computer science course, our study also aims to better understand their application in this context.
The EDM community’s interest in persistence was sparked by [
Subsequent studies have devised variations in criteria for
differentiating between productive persistence and wheel spinning
[
The goal of the most recent studies has been to identify wheel
spinning in ITSs as early as possible. [
The studies mentioned thus far have focused almost exclusively on ITSs, which are most commonly used to teach math. Detecting and studying persistence on computer programming problems requires first understanding how data from these tasks has been analyzed in past studies.
There is growing interest in leveraging data analytic methods to
study students’ action logs produced during programming activities
[
Studies have also developed features to evaluate how similar or
different two computer programs are. [
As our goal was to focus on behaviors related to how students approach solving a problem, rather than investigating the content of the submitted solution, we used an approach in line with the first category to investigate elements of student persistence in a series of computer programming problems. This allowed us to focus specifically on the productive and unproductive behaviors of persistent students.
We collected data from an introductory computer science course at
a large midwestern university in the U.S. hosted on PrairieLearn,
an open-source, web-based problem-driven learning system [
All homework assignments were programming problems with checkstyle, compiler, and problem-specific tests that students’ code had to pass to receive full credit. Students had one day to successfully complete each homework problem. They were allowed to submit solution attempts as often as required until they successfully passed all the tests. After each submission, the system ran tests to check the correctness of the solution and provided feedback indicating mistakes. First, the system tested whether the solution had any checkstyle and compiler errors. If such error existed, the system showed feedback about these errors and stopped. If there were no checkstyle or compiler errors, the system further used several problem-specific tests to examine whether the solution fulfilled the requirement. For example, given some random input, would the solution generate the correct output? If not, the system would return feedback about the problem-specific test error. Otherwise, the solution was regarded as correct.
We aggregated our dataset at the student-problem level using a series of features specifically related to persistence. While persistence can be studied at various grain sizes, we chose this level due to our interest in how students tackle difficulties within a particular programming problem. Similarly, we only kept instances that demonstrated struggling, as defined in section 3.2.1, since these were the cases that could elicit persistence from students. Quizzes were conducted weekly as part of regular class activity to assess learning and consisted of both multiple-choice questions and programming tasks. Quizzes were made available at the end of the week and were designed to provide early assessment related to the content of the homework problems assigned earlier that week. We aligned the content of each homework problem to corresponding multiple-choice quiz questions to directly investigate the relationship between persistence in specific homework problems and outcome on related assessment questions. Once we had these alignments, we calculated for each student-problem instance the total number of points obtained on the relevant quiz questions and the maximum possible points. Using these values, we then calculated the point percentage as the indicator of learning. Only quiz questions that students attempted were considered for these calculations. After these changes and calculations, our aggregated dataset consisted of 7,673 instances of student-problem pairs, submitted by a total of 710 students.
The resulting distribution of the score outcome variable had a strong negative skew, with most instances accumulated at higher scores, as shown in Figure 1. This is because students often managed to obtain a perfect score on their aligned quiz questions. Students were typically given two chances to select the right answer, the second time for half credit.
Given our goal to study how specific behaviors might be related to
persistence, our feature engineering efforts focused on developing
features based on an underlying rationale about their relationship to
productive or unproductive persistence. Following the Carnegie
Foundation for the Advancement of Teaching’s definition of
productive persistence—“tenacity plus the use of good strategies”
[
We defined students as struggling if they worked on a programming problem for a long time or if they submitted a high number of solutions to a problem. We considered that students could only show persistence in the context of problems for which they struggled.
This operationalization of struggling depends on identifying both a time and attempt threshold, each specifically calculated for that homework problem. Thus, once we calculated the thresholds for each problem, we created two binary struggling threshold features: beyond time threshold and beyond attempt threshold. We only kept instances of students that satisfied at least one of these two criteria. We also created two numerical features that measured a student’s deviation from each of these thresholds. Because the thresholds were already calculated at the problem level, standardizing these deviation features would result in perfectly collinear features, so we did not standardize them.
For the time threshold, we used the minimum value between the 75th quantile of students’ total time on each problem and 15 minutes. We combined the 75th quantile and 15 minutes to determine the time threshold based on several reasons. First, given that the course is only an introductory CS course, it is reasonable that one fourth of students struggled with difficult programming problems. Second, the proportion of students who struggled with unchallenging problems would be smaller. Using an absolute threshold would be better for these cases. Third, we used 15 minutes as the absolute threshold because 57.56% of problems had a 75th quantile of total time smaller than 15 minutes. It seems reasonable to regard close to half of problems as unchallenging. Given that the number of attempts is an important indicator of persistence, many attempts on a problem might also be indicative of struggling, even when the total time spent on the problem falls under the time threshold. Analogous to deciding the time threshold, we used the minimum value between the 75% quantile of the number of attempts on a problem and 9 attempts to determine the attempt threshold. If the 75th quantile of the number of attempts on a problem was smaller than 9 attempts, the later became the attempt threshold. We used 9 attempts as the absolute threshold because 56.06% of problems had a 75th quantile of the number of attempts no more than 9 attempts. This number was close to 57.56%, the proportion of problems with a 75th quantile of total time smaller than 15 minutes. 3.2.2 Solved
This is directly related to wheel spinning as defined by [
This is a typical measure used in the persistence literature [
As with the number of submissions, the time that students spend on a challenging problem might indicate the amount of persistence being demonstrated. We again reasoned that more time (and thus more wheel spinning) may be predictive of more struggling and lower scores on the quiz questions.
Our platform only allowed us to measure the time between submissions, so we had no way of knowing with certainty how much time was spent working on a problem. If the time difference between a student’s two consecutive submissions was beyond 15 minutes, we regarded this student as being away from this problem during that interval (see the feature taking a break below for the choice of 15 minutes as a threshold). In these cases, we replaced this time difference with the student’s mean time difference between other consecutive submissions on this problem so that we could estimate the student’s total time on the problem more accurately.
We defined taking a break as a struggling student being away from the problem at least once. When the time between two consecutive submissions on the same problem went beyond 15 minutes, we regarded the student as away from the task. As discussed above, 15 minutes might be sufficient for solving unchallenging problems if students did not struggle. Moreover, 81.57% of pairs of consecutive submissions had a time difference less than 15 minutes. This proportion only increased slightly to 83.77% when increasing this threshold from 15 minutes to 1 hour. Thus, it is reasonable to use 15 minutes as the threshold for being away from the problem. Note that if a student attempted other homework problems between two consecutive submissions on the same problem, we regarded this student as interleaving rather than taking a break.
Our rationale for measuring break taking is based on the idea that a
wheel-spinning state may be overcome by time away from task.
Some of the cognitive benefits of breaks have been documented [
Interleaved practice, as opposed to blocked practice, refers to a
learning technique that mixes up the order of topics, lessons, or
problems presented. Studies have shown that this practice usually
improves learning outcomes [
Quick, consequent submissions may indicate guessing or uncritical
attempts to fix problems without much reflection. This behavior has
been associated with students trying to game the system [
Shorter time between submissions may indicate more unproductive
attempts to push through to an answer without stopping to
think/work carefully or take breaks [
To test the importance of our various features, we created a random forest model using a shuffled 70/30 validation/testing split grouped by student with 5,396 and 2,277 instances respectively. We conducted 500 iterations of Bayesian hyperparameter optimization on the validation set using 10-fold cross validation grouped by student. This hyperparameter tuning was set to optimize the R2 score.
We originally tested a wide array of models, including various
linear, tree-based, and ensemble algorithms, and we further tuned
some of the most promising ones. We found that variations of
gradient boosting models performed best. However, we chose to
focus on random forest for our feature interpretation for two
reasons: (1) the performance gained by using the best models over
random forest was negligible, and (2) random forest models have
been shown to be useful for predictions related to persistence in
other EDM research [
Once we had constructed our final model, we re-trained it on the
entire dataset in preparation for feature interpretation. For the task
of interpreting feature importance, we analyzed SHapley Additive
exPlanations (SHAP) values. SHAP is a game-theoretic approach
that calculates the effect that each value in the feature matrix has
on that instance’s prediction, relative to the mean prediction [
Our tuned random forest model attained an average cross-validated R2 of 0.133 and an average RMSE of 0.129 on the validation set. On the held-out testing set, the resulting R2 was 0.145 and the RMSE was 0.130. Our persistence features accounted for roughly 14% of the variation in related quiz scores.
A preliminary analysis of our model uncovered certain important patterns. For one, our least impactful features were all binary measures—such as whether interleaving, rapid guessing, or breaktaking were observed—whereas our top features were the standardized measures of those binary features. Figure 2 shows the entire set of feature rankings based on mean absolute SHAP values. A detailed exploration of these features revealed what appears to be an opposing impact between some binary features and their standardized counterparts. For example, the feature solved has a negative correlation between its values and its SHAP values (r = -0.217, p < 0.0001), whereas its standardized version, solved_std, has a positive correlation (r = 0.33, p < 0.0001). Measuring this correlation between feature and SHAP values allows us to better understand how the model is using the feature. Higher correlation, and thus a stronger linear relationship, suggests a more straight-forward interpretation for the feature’s role in the model. While the impact of solved is very small in the overall model (ranked 17th, mean absolute SHAP = 0.00003), solved_std is our top feature in terms of overall impact on the predicted score (mean absolute SHAP = 0.02129). We found this same inverted relationship between many other impactful standardized features and their original, binary, far less impactful counterparts. Because we standardized features at the problem level, the correlation between each unstandardized and corresponding standardized feature is never quite perfect, but some do come close. Random forest models typically do not suffer from collinear features the way more traditional statistical regression methods do. This is largely because of the way features are randomly sampled for each tree. Even when both collinear features are part of the feature subset, a decision tree will typically ignore one in favor of the other. We suspect that much of our model’s preference for the standardized features over unstandardized ones is the added problem-level information they contain, which could be interpreted as information regarding the difficulty of the problem. However, while the predictive power of a random forest is not affected by collinear features, model interpretability suffers, as we found through our preliminary analysis. Given our goal of better understanding the different aspects of persistence and their relationships, we decided to remove the original non-standardized features. We also removed time_threshold_deviation and attempt_threshold_deviation, which were very highly correlated with total_time_std and num_submissions_std respectively. We then re-trained and re-tested our model.
After removing these features, we found that our model’s average cross-validated R2 on the validation set increased slightly, from 0.133 to 0.134, while RMSE remained constant. On the held-out testing set, its R2 also increased, from 0.145 to 0.147, while RMSE remained constant. We then re-trained our model on the entire dataset in preparation for our in-depth feature analysis.
Our analysis using SHAP values found that the solved_std and rapid_guessing_std features had the biggest effect, accounting for an average impact of 0.0215 and 0.0172 on the predicted score respectively. The third most important feature, taking_break_std, had an average impact less than half as strong at 0.0076. Together, these three features account for 75% of all features’ total impact on the predicted score. Figure 3 shows the feature rankings based on mean absolute SHAP values, while Table 1 allows for comparison with other methods such as Gini-impurity-based importance and permutation importance. Rankings based on these three different approaches yielded almost identical results with only minor variations, strengthening the reliability of our findings. Besides ranking the features by impact on the predicted score, SHAP values allow us to explore the nature of that impact more deeply, as well as the interactions between features. Figure 4 is a beeswarm plot of SHAP values by feature with color indicating the value of each individual instance. To further aid our interpretation, we also explored which features had the highest absolute correlation between their values and their corresponding SHAP values. In essence, this correlation is a measure of just how linear each feature’s effect is on the predicted score. We calculated Pearson’s r for all features (see Table 1) and found that all p values were below 0.0001, except for sd_time_diff. Throughout this analysis, we point out when a feature’s correlation is indicative of a linear relationship. 4.2.2 Solved We can see (Figure 4) that the bulk of solved_std is composed of high values (red color), indicating that most students managed to solve most homework problems. The long positive skew suggests that small, positive variations in this feature could potentially push the predicted quiz score up by about 0.1. The few lower values in this feature (blue color) are found on the left side of the plot, suggesting that not solving the problem tended to pull the predicted score down. Indeed, we found a moderate positive linear relationship between solved_std and its SHAP values (r = 0.34), further confirming our initial analysis.
This confirms our hypothesis. It suggests that solving a challenging problem (productive persistence) may be related to a better understanding of the underlying concepts, whereas not solving the problem (wheel spinning) suggests a lack of understanding.
Our models’ second most impactful feature, rapid_guessing_std, is
in many ways the opposite. Most students did not engage in rapid
guessing. Those who did, particularly on homework problems
where few others did—identified by high rapid_guessing_std, or
red color in the beeswarm plot (Figure 4)—were more generally
affected negatively in their predicted score based on this feature.
This effect can more clearly be seen when plotting the SHAP values
for the feature against the values of the feature itself (Figure 5).
This view allows us to get a better sense of how most instances with
a higher rapid_guessing_std value impact the predicted score
negatively. This aligns with our hypothesis: rapid guessing, with its
potential implications of wheel spinning [
Our model’s third most impactful feature, taking_break_std, has a
very clear pattern that is easily observable in Figure 4. Lower
feature values generally lead to a positive impact on predicted
score, whereas taking a break is more likely to have a negative
impact on score. We found a negative linear relationship between
taking_break_std and its SHAP value (Figure 6), with Pearson’s r
of -0.608. The distribution of SHAP values for this feature indicates
a potential negative impact about three times as large as the positive
one.
This result is the opposite of what we hypothesized. Since taking
breaks during a difficult task has been shown to improve cognition
[
One possible explanation is that students who took a break did, in fact, perform better than they would have otherwise. Since our method does not directly test causation, our model may be using this feature as a proxy for students who struggled more than others. Another possibility is that this feature is not solely capturing intentional break-taking, but also interruptions to students’ work, which may serve as distractions—certainly not an ideal learning situation. We did not calculate how many times students took a break, only if there was at least one 15-minute gap between submissions when struggling. Finally, because homework problems were due at midnight on the day they became available, students may simply not have had sufficient time for effective break taking. Without additional information about learning context or calculating additional features, we have no way of knowing which of these explanations, if any, are the most likely.
For beyond_time_threshold_std, we can see in Figure 4 that lower values generally lead to increases in the predicted score and vice versa. This is indicative of the underlying attribute this feature attempts to capture—going beyond the time threshold yields smaller (generally negative) SHAP values, whereas not going beyond the time threshold yields larger (generally positive) values, the exact value being heavily affected by how much other students crossed the threshold on the same problem. For students who take longer than the norm, this generally has a negative effect on their score. The relationship here is moderately linear with an r of -0.349. The fifth top feature that we identified, beyond_attempt_threshold_std, does not have such a clear pattern. The SHAP values seem to be widely spread irrespective of the feature’s values. The feature’s distribution is bimodal, as is the case with most of the features that standardize a binary variable, and we did find a small distinction in the SHAP values between the two modes (Figure 7). While the mean for each mode is essentially zero, higher instances of beyond_attempt_threshold_std, which correspond with student-problem instances that went beyond that problem’s attempt threshold, have a moderate negative correlation with their SHAP values (r = -0.409, p < 0.0001) and lower instances, on the other hand, have a positive correlation about equally as strong (r = 0.382, p < 0.0001). This suggests that the impact of this feature on predicted score is highly dependent on how much one’s status on the underlying binary variable (beyond_attempt_threshold) varies from the norm for that given homework problem.
We found that num_submissions_std, our model’s sixth top feature in terms of impact, has the strongest correlation between its feature values and SHAP values (r = -0.833). This fits with our hypothesis. The more attempts that students submit, the more likely they are to be struggling, and the less likely they are to perform well when tested on the same skills during their weekly quiz.
We found that our three time-related features—not including beyond_time_threshold_std, which is of a very different nature since its non-standardized version is a binary feature—had some of the weakest predictive power in our model. total_time_std had a still moderate mean absolute SHAP at 0.00257 and a very strong correlation between its feature and SHAP values with r = -0.773. avg_time_diff_std and sd_time_diff_std, by comparison, had a much lower mean absolute SHAP (respectively 0.00099 and 0.00098) and no correlation.
The strong, negative correlation between total_time_std and its SHAP values mean that the model is interpreting longer time on a problem as being related to lower learning outcomes, or at the very least as a student struggling enough with a problem to lead to a lower score on the weekly quiz. This latter possibility is in line with our hypothesis and with what we found for beyond_time_threshold_std. Interestingly, this pattern is far more pronounced for instances that went beyond the time threshold (red points in Figure 8), whereas the relationship is seemingly reversed for cases where students did not go beyond the time threshold (blue/purple points in Figure 8). As for the two features that specifically look at time between submissions (avg_time_diff_std and sd_time_diff_std), their weakness both in predictive impact and correlation with SHAP values suggest at face value that this factor has little value at predicting learning success (or lack thereof) when students struggle with a problem. These features’ impact may also have been affected by the high correlation between them (r = 0.76). Similar information may have also been captured by a combination of beyond_time_threshold_std and beyond_attempt_threshold_std.
Finally, our model’s least impactful feature, interleaving_std, had
by far the lowest mean absolute SHAP value (0.00017) and a low
correlation between its features and its SHAP values (r = 0.126).
We originally hypothesized that this feature would play a bigger
role in predicting students’ scores, considering that the practice of
interleaving when struggling is generally considered a good
learning practice [
Our study suffers from limitations primarily related to the alignedquiz-question scores we calculated for each student-problem instance. For one, the score distribution was heavily skewed due to the abundance of almost perfect quiz scores. Additionally, while the PrairieLearn platform allowed us to use the course’s quizzes without requiring students to take an additional posttest, the scores did not take into account students’ prior knowledge and skills. This made it difficult to measure the impact of students’ productive vs. unproductive persistence directly.
These factors likely led to our model’s limited predictive performance (R2 = 0.147 on the held-out test set). While we believe that our final model’s performance was sufficient for our purposes of interpreting the relationship between elements of persistence and learning outcomes, it should be possible to create a more accurate model without severely sacrificing interpretability.
The most impactful features were those related to solving the problem, rapid guessing, and taking a break. Those with the most straightforward linear effect were the number of submissions, total time, and (again) taking a break. All three of the latter had a strong negative correlation between their feature values and their impact on prediction. In other words, more attempts, taking a longer time, and taking a break are all correlated with lower scores on related quiz questions. Solving the problem—our most impactful feature— had a moderate positive correlation, highlighting the positive nature of the relationship between successfully completing homework problems and score on subsequent related quiz questions. This all suggests that solving the problem and rapid guessing are important features for accurate prediction, while the number of submissions and total time are indicative of the differences between productive persistence and wheel spinning in a computer science context. Taking a break fits into both of these categories. Perhaps most important, we were able to identify features that are directly related to learning strategies. Our findings suggest that students should avoid rapidly submitting subsequent programming attempts without actively trying to address problems in their code (rapid guessing). Taking a break may also be unproductive behavior, though this finding may be an artifact of the specific context in which students were able to submit homework in this course, as well as the particular way in which we calculated this feature. As for interleaving, its predictive strength in our model was low, but its effects nevertheless suggest that a future investigation should study whether it can be an effective practice when struggling on a problem.
In order to address the limitations of our study, we suggest that future research focus on devising a more robust measure of learning that takes into account students’ individual starting points. Additionally, for the CS context of this study, a valid measure of programming proficiency that considers the problem-solving process would be superior to the quiz scores we used as proxy.
We would like to acknowledge NSF grant #DRL-1942962 for making this work possible.