Online homework systems are common in university courses. While scienti c ndings about learning could have bearing on how instructors design these systems, there is little guidance available for instructors on the problem of extrapolating scienti c results in various contexts to make design decisions in speci c settings. This paper leverages the value of online environments to conduct randomized experiments that directly test principles in a real-world introductory programming course. We investigate the relative bene t of giving students explanations of the correct solution to a problem and giving them an additional problem. We nd suggestive evidence that students do better on subsequent problems in the same exercise when given an explanation, but they do better on a post-test two weeks later when given an additional practice problem. These results can inform instructors' decisions in designing online homework.
Many university courses use online homework systems to give students practice material. These systems enable students to conveniently practice their skills and instructors to automatically grade homework and gather data on student performance. They typically consist of problems with either written or multiple-choice responses for students to complete.
Past research has investigated the e ects of these supports
on student learning. There is experimental evidence, for
example, that explanations and practice problems can help
student learning under certain circumstances [
The ICAP framework [
Repeat Chars v2
First Fifty
Nested For and Range v2
How Many - Nested For Loops
ready have some knowledge about a subject, providing
additional explanations instead of other knowledge-reinforcing
activities can be detrimental to learning [
Similarly, the e ects of solving problems on learning can be
varied and complex under di erent circumstances. For
instance, there is a large body of research on the design of
intelligent tutoring systems which automatically determine
which practice problems to show learners and in what order
to improve their understanding most e ectively [
These studies have shown that even in a controlled laboratory environment, the e ects of these intuitively-helpful interventions vary signi cantly. Instructors seeking to apply these ndings to their courses face the problem of translating ndings in laboratory experiments to design decisions about what kind of support to provide in online problems and other educational environments.
Many counterintuitive e ects have been found in prior education research, so it is essential to empirically study the effects of interventions before recommending them to instructors. In this paper, we extend upon past literature about the role of reading explanations and solving problems in learning and provide empirical evidence on how these forms of student support a ect learning in a real-world setting. Ultimately, we hypothesize that students will perform better on subsequent tests of their understanding when they are shown an additional problem compared to when they are shown an explanation. This hypothesis is motivated by the active-passive boundary from the ICAP framework. 0.2 s t pm0.1 e t t a #0.0 n i e cn−0.1 e r e f f id−0.2 m r e -t−0.3 t r o h S−0.4
Long-Term Effect of Explanation 0.2 s t p tem0.1 t a # in 0.0 e c n e r e ff−0.1 i d m r e -t−0.2 g n o L
The context for the experiment on explanations and additional practice problems was the Programming Course Resource System (PCRS) online homework system for the introductory computer programming course at the University of Toronto. This course spans one twelve-week semester and students are given for-credit online homework exercises each week. The problem we deployed the supports on is shown in Figure 1. It asks students to analyze the runtime of a for loop in Python.
A total of 648 students completed the homework in week 10 of the course. 478 of these students also completed the optional follow-up exercise in week 12. There were 5 problems in each week. This choice of weeks ensures that there is considerable delay between the initial intervention and subsequent measurement, enabling us to measure long-term learning.
In the experiment, after students attempted a homework problem in week 10 of the course, we used a factorial design to independently vary two factors: whether an explanation was provided and whether an additional problem was provided. The experiment was performed in the context of a multiple-choice problem pertaining to run-time analysis, shown in Figure 1. Students were given course credit for completing the main problem, but did not have any direct incentive to read the explanation or attempt the additional problem.
To measure the impact on learning over a longer time frame, we designed a post-test with problems that were either identical to or variants of the problems asked in week 10. We gave these problems to students two weeks after the experiment (week 12). Some follow-up problems were identical to the corresponding week 10 problems and others had features of the problem changed, such as having a loop executing 50 times rather than 30. Students were not at ceiling performance in the post-test, suggesting that they did not remember the exact answers by week 12, so these were non-trivial measures of learning. Figure 3 shows the names of problems in week 10 and the corresponding problems in the week 12 follow-up activity. All of these problems were focused on analyzing the runtime of Python programs.
We experimentally varied two variables in a factorial experiment. Each time the student submitted an answer to the rst problem of the exercise, they were randomly assigned to a condition for the Explanation factor and the Additional Problem factor.
The Explanation factor had three levels: absent (none), short, or long. The short explanation states, \The third answer is correct because the code inside the for loop takes constant steps regardless of len(s) and it will be executed len(s) times." The long explanation states \Suppose s = `cat'. Then, double = double + ch * 2 will be executed 3 times because the for loop iterate through each character of s (i.e. `c', `a' and `t'). Now, suppose s = `google'. Then double = double + ch * 2 will be executed 6 times. As you can see, if len(s) doubles, the number of steps also doubles. So, the third answer is correct." The Additional Problem factor had two levels: absent (none) or present (one additional problem that was very similar to the problem students had attempted in asking them to trace through a for loop and determine its time complexity). A screenshot of this problem is shown in Figure 2.1 To measure how well a student performs on a problem, we used the number of attempts until the rst correct answer. This is simply the number of submissions made before the 1These factors were varied in the context of a larger experiment with more factors that will not be described in this paper in the interest of space. We used weighted randomization in favour of not showing students additional activities to avoid overwhelming them with too many activities
student selects all of the correct options and none of the incorrect options on the multiple-choice problem. For example, if a student gets the problem correct on their rst try, their number of attempts is 1. If they get the rst attempt wrong but the second attempt right, their number of attempts is 2.
To measure short-term improvement, we took the di erence between the number of attempts on problem 1 of the week 10 exercise and the average number of attempts for the remaining four problems in that exercise. This number can be negative if students did worse on the remaining problems than they did on the problem we deployed the supports on, and the higher the number, the greater the improvement. To measure improvement on the delayed exercise, we took the di erence between the number of attempts on problem 1 of the week 10 exercise and the number of attempts on the exact same problem when presented in the week 12 followup.
In this section, we rst show a lack of evidence for an improvement in performance between the problem we added student support to in week 10 and the same problem given in a follow-up exercise in week 12. Next, we analyze the e ects of the Explanation and Additional Problem factors. Our results did not reach the signi cance threshold of p < 0:05, and as such they should be interpreted with caution. We present suggestive evidence that the explanations were helpful on the same homework exercise (t(490)=-1.24, p=0.215), but not on the follow-up test two weeks later (t(364)=0.300, p=0.764). Finally, we show the reverse trend with the Additional Problem: it was not helpful in the same homework exercise (t(646)=0.158, p=0.874) but might have been in the follow-up exercise two weeks later (t(476)=-1.602, p=0.110). We interpret how these results can inform instructors' design choices and address possible limitations of the work. 0.4 s t p tem0.3 t a in#0.2 e c en 0.1 r e f f id 0.0 m r e t -−0.1 g n o L −0.2
Mean = -0.079 n = 367 none
Mean = 0.252
n = 111 additional problem
Students took only slightly fewer attempts to get the problem correct in week 12 compared to week 10. While they took 2.07 attempts on average to get the answer right in week 10, they took 2.02 attempts to get the answer right on week 12, even though they had completed the same problem just two weeks before. We found little evidence that students improved between solving a problem in week 10 and solving the same problem in week 12 (t(1136)=-0.529, p=0.597). This suggests that students might not have remembered the answer to the problem even when they already solved it two weeks earlier, meaning testing them on the same problem in week 12 appears to be a non-trivial measure of their understanding of the same concepts.
We found no statistically signi cant di erence between students who were given explanations and those who were not. However, the results suggest that when students were given explanations, they took slightly fewer attempts to get the right answer in subsequent problems than those who did not, regardless of whether they saw a short (t(490)=-1.24, p=0.215) or long explanation (t(491)=-1.29, p=0.195), as shown in Figure 4. However, the e ect of seeing explanations was much smaller in the long term, as the sample means were similar in all three conditions. This is shown in Figure 5 and suggests that an explanation in a homework context might be useful only during that homework session. This could have happened because the problems tested a procedural skill, namely runtime analysis. While reading an explanation gives students a clear formula they can apply in subsequent runtime analysis, they might forget that formula when they stop working on their homework and lose the bene t of the explanation.
Similarly to the Explanation factor, we did not nd statistically signi cant evidence for a di erence in means for the Additional Problem factor. However, we still found suggestive evidence that giving an additional problem has an e ect in the long term. We did not nd evidence for a di erence between the performance of students who were shown an additional problem and those who were not on subsequent problems in the same homework exercise, but students who received the additional problem took fewer attempts in the post-test (t(476)=-1.602, p=0.110). These results are shown in Figures 6 and 7 respectively.2 This di erence might suggest that the value of the additional practice problem was primarily as a memory aid. Doing more problems could have helped students remember the skill they learned better when writing the post-test. If this knowledge is already in their minds when they are doing the exercise, it makes sense that they did not bene t immediately from more practice. However, they might remember more when writing the post-test, which would explain the improvement in performance there.
A notable limitation of this work was the lack of statistical sign cance. However, the results are consistent with each other and align with ideas from the ICAP framework. As such, they suggest a trend that could inform future research. In the interest of open and replicable science, it can be valuable to publish suggestive and negative results that do not meet the threshold for statistical signi cance. Real-world data is often messy and suggestive results can reveal crucial new directions for analysis.
Another limitation of this work is that the problems in the week 12 follow-up were not all identical to those in the week 10 homework. They tested the same concepts and some were exact copies, but others were slight modi cations of problems on the original homework. Therefore, the observed results might be due to the supports having di erent degrees of relevance to the problems in week 10 and week 12 rather than the duration between support and post-test. We have mitigated this by using the di erences between number of attempts on the problem we applied the explanations and additional problem to and the relevant subsequent problems as dependent variables, so if one intervention improves students' score on problem 1 both in week 10 and week 12, that would be re ected in that the changes to both scores cancel out when the di erence is computed.
One might also raise the concern that we had di erent sample sizes in di erent conditions. More students were assigned to the \none" condition than other conditions for both the Explanation and Additional Problem factors. We intentionally weighted the randomization in this way to minimize the burden on students from having too many additional activities, a strategy used in randomized clinical trials in the medical eld.
Considering that the e ects of reading explanations and solving problems might vary widely with context, such as the week of a course in which supports were given, it is unclear how broadly the trends we identify in our data apply. While it appears possible that giving students more practice 2After this analysis, we noticed that the control and experimental groups had di erent variances, which violates the assumption of the standard t-test. We then ran Welch's ttest and found a p-value of 0.07. (t(476)=-1.813, p=0.0712) problems helps them develop lasting procedural knowledge of how to analyze the runtime of an algorithm, it is not clear that we can conclude the same about di erent tasks in computer science education like learning the syntax of a programming language or how to design an algorithm. Finally, the week 12 post-test was optional, so dropout is a concern. While 648 students completed the exercise in week 10, only 478 completed the follow-up post-test in week 12. Therefore, the conclusions we draw about the e ects of educational supports in the long term might re ect only the population of students who choose to complete the posttest. Though this was the majority of students, the reported e ect could be di erent if, for instance, the students who are unlikely to do optional homework problems in week 12 are also unlikely to attempt an optional problem given to them in their week 10 homework exercise.
Our experiment investigated the e ects of explanations and additional problems on performance both on a post-test and subsequent problems on the same test. We found intriguing but not de nitive insight into the e ects of explanations. The mean number of attempts for students who saw either a short or long explanation was lower than those who saw no explanation, but this di erence was not statistically signi cant (t(490)=-1.24, p=0.215). It is possible that the explanations we showed students simply did not have an e ect on their learning in either the short or long term. It could be that the explanations used in this experiment did not benet students as much as they could have and e ort should be directed to designing better explanations. Alternatively, it is possible that the explanations helped students somewhat on the remaining problems in the homework exercise. If this result were replicated in a larger study, it would be interesting because it could guide instructors in deciding how to e ectively incorporate instructional explanation into their courses.
In exploring the e ect of additional problems, we found that the mean number of attempts on the equivalent post-test problem was lower for students who were shown an additional problem than those who were not. This di erence was not statistically signi cant, though we found stronger evidence for it than we found for explanations (t(476)=1.602, p=0.110). Like with the explanations, it is possible that the additional problem we gave students was truly not e ective and future work should focus on how to design more e ective practice problems. However, if the long-term improvement as a result of the additional problem is replicated in subsequent large-scale experiments, it could provide guidance for instructors in deciding how to incorporate practice problems into their courses e ectively.
If the results reported above re ect a real e ect, they suggest that explanations are helpful in the short term, but not in the long term. Conversely, additional problems are helpful in the long term, but not in the short term. This aligns with what one might expect based on the ICAP framework, as solving a problem quali es as deeper engagement with the learning material than reading an explanation. Instructors likely care more about whether their students retain information in the long term rather than whether they understand concepts immediately, so focusing on the long-term learning measure makes sense.
The possible di erence between the e ects of these interventions is interesting and motivates further research into how immediate and delayed e ects of reading explanations and solving problems might di er. This might help guide instructors in thinking about the trade-o s involved in deciding when to give explanations to students and when to give them more problems.
Future work should investigate how generally this pattern holds. The part of the course we deployed these supports on focused on the procedural skill of reading an algorithm and analyzing its time complexity. Would giving an additional problem still be e ective in teaching a di erent concept in the course, such as the di erence between for and while loops? Perhaps additional problems are more helpful in developing procedural knowledge while good explanations might be more e ective in building propositional knowledge. By running similar experiments at di erent points in the introductory computer science course, we hope to learn more about which types of student support are helpful in developing di erent skills.
Additionally, we are interested in investigating whether the e ects of these interventions di er across subgroups of students. One reason why we might not see a large average e ect is that the e ectiveness of di erent forms of support could vary signi cantly across students. Perhaps, for example, students who take a programming course out of intrinsic interest are more likely to bene t from an additional practice problem than those who take it to satisfy a breadth requirement. By analyzing this experimental data jointly with contextual variables derived from surveys and data mining, we hope to provide a richer picture of which forms of support work for which students and how instructors can tailor interventions more precisely to individual students' needs.