In this paper, we attempt to estimate how much learning happens in required practice activities (homework) relative to elective practice activities (studying). This analysis is done in the context of a large enrollment (N = 601) introductory programming course that made heavy use of autograding randomizing question (item) generators. Because these item generators (and other problems) were used as homework, on practice exams, and as part of exams, a given student may have encountered the same generator multiple times during the class, providing snapshots of the evolution of the student's ability to complete that problem correctly. We use a post hoc model of “this-item-correct” prediction to estimate individual student knowledge on each attempt of a given question. Across vfie exams, correctness tracing attributes 57-65% of the learning that occurs to the homework period and the remainder to elective practice (the study period).
A well-designed course provides students with many
opportunities to learn (e.g., readings, direct instruction, activities
with peers, homework). While summative assessment allows
us to estimate how much learning has occurred, it doesn’t
shed light on where the learning happened. If we could
attribute learning to the activities in which it occurred, this
would allow teachers to increase their use of efective
activities and deprecate ineefctive ones. Our goal as educators is
Copyright '2021 for this paper by its authors. Use permitted under
Creative Commons License Attribution 4.0 International (CC BY 4.0)
to engage students, to assist both them and us in diagnosing
their progress, and to provide formative experiences during
their learning careers [
In most courses, the bulk of the students’ time is spent
outside of course meetings, either completing homework or
performing elective practice (studying). It has been shown
that well-formed homework has a positive impact on
student performance and motivation [
We were curious to explore the degree to which we can
attribute student learning between two kinds of formative
practice activities: required homework and elective practice
performed prior to a summative assessment. Additionally,
as our course utilizes multiple types of questions, we were
curious to know if student experiences diefred between types.
To do so after the completion of the course, we use a post hoc
knowledge estimation method developed by Chen et al [
We apply CT to student submission data from an on-campus introductory programming course. The course used randomly selected questions from question pools and random item generators for exam creation, with many of the questions appearing previously on homework (and optional practice exams) as a studying motivator for students. We use data from student homework, practice exams, and these proctored exams to build a cohesive snapshot of student experience with the same questions in multiple contexts. Specicfially, we share our experience investigating students’ learning in this fashion to address the following questions: RQ1: How much learning happens during required practice activities (homework) relative to elective practice (studying)? RQ2: Does student learning difer based on the type of the questions (e.g., multiple-choice vs. short answer) asked? The rest of our paper is organized as follows. Section 2 describes related work on student learning and knowledge tracing. Section 3 discusses the course from which we collected data and the handling of that data. In Section 4, we explain the assumptions behind CT and detail our use of the method. We follow with our results from the modeling in Section 5 and with interpretation and limitations in Section 6. We conclude in Section 7. 2.1
How students learn is an area of signicfiant study. We are
specicfially interested here in how formative assessment (e.g.
homework) helps students learn. Historically, formative
assessment is claimed to benetfi student learning, although
there is little consensus on what exactly makes good
formative assessment [
Mandatory homework is found to be benecfiial in existing
research, but in large part due to feedback. Gutarts and
Bains found that homework that provides feedback appears
to enhance student performance [
The benetfis of studying are less clearly defined. Chew
suggests the benefit of study can be improved by teaching
students how to study and that expecting students to know
how without designing assignments and material to aid their
studying may be a mistake on the part of some
instructors [
There is a wealth of work on diefrent methods of tracing
student knowledge and modeling student learning and
student behavior. Many of these stem from Corbett and
Anderson’s original knowledge tracing paper [
Pardos and Hefernan modeled individualized learning in
Bayesian knowledge tracing (BKT) [
Deep learning methods have also been applied to
knowledge tracing in deep knowledge tracing (DKT) [
Our data was collected in a large enrollment, introductory programming course for non-CS majors in Fall 2019. The course had 601 total students, with 246 women and 355 men. The majority of students who took the course were freshmen (67%) and sophomores (21%). The course predominantly taught Python programming with some coverage of basic Excel and HTML/web concepts. 3.1
The course was organized as a iflpped class that covered one major topic each week. Students were expected to complete readings in an interactive textbook and an assignment consisting of true/false and multiple-choice questions prior to 1 2 3 4 5 6 7 8 lecture. The weekly 90-minute lecture used peer instruction to reinforce concepts, and the weekly 80-minute lab consisted of practice activities students could complete individually or in pairs, supervised by course sta.f Finally, each topic culminated with a weekly homework assignment that consisted of a mix of short answer (e.g., “What is the value of the variable x after the following piece of code executes?”, “Write a statement that removes the 4th element of a list called ’animals’.”) and small programming (i.e., no more than a small function) questions.
Due to the size of the course, almost all of the homework
activities were auto-graded. The course used the open-source
assessment platform (PrairieLearn) [
Many of the homework questions were item generators that
could produce many possible questions of similar dicfiulty
on the same topic [
The course’s primary mean of summative assessment was
through vfie proctored exams. All the exams had a
50minute xfied time limit, except for the nfial exam (E4) which
allowed for 3 hours. All but the rfist exam were worth a
signicfiant portion ( ≥ 10%) of the course grade. These
exams were conducted in a proctored computer lab with
student scheduled exam times within a three-day window [
Exams featured all four kinds of questions seen on homework (T/F, MC, short, programming), except for E0 which did not have programming questions. Each exam consisted of 20–30 question slots (41 on the nfial). Each slot drew randomly from a pool of questions on a given topic with similar dificulties. Most questions permitted students to attempt them multiple times with a score penalty for each subsequent incorrect attempt until chances to earn credit were exhausted.
Because of the course’s heavy use of item generators and to motivate students to take homework seriously, a significant fraction of the exams were drawn from the course’s pre-lecture and homework assignments. In general, 85–90% of the pools on the exam were drawn from questions previously on homework, and exam-only “hidden” questions were written with similar form and content to previous homework questions. Prior to each exam, students were provided access to a practice exam generator that was similar to the actual exam generator, but without the hidden questions. Reused programming questions are largely recall exercises, as most do not feature random generation. Short answer questions are transfer tasks as they are all parameterized and no two instances of the question should possess the same exact parameters and the same expected student answer. In spite of the exams including a large fraction of previously seen material, we don’t believe that rote memorization was a useful strategy for these exams due to their heavy use of randomization and question pools combined with a large number of questions (20–30) on the exam. True/false and multiple-choice slots on the exam generally drew from pools of 20 to 100 questions, while short answer and programming question slots had pool sizes of 5 to 12. In addition, short answer item generators typically produce at least dozens of meaningfully diefrent variants. 3.2
The decision for exams to mostly use the same questions as homework assignments and practice generators created an interesting context for attributing student learning. Specifically, we could analyze student performance on homework assignments, practice exams, and actual exams to observe how students’ ability to answer these questions improved as they engaged with course material. We pulled all student submissions from PrairieLearn for the entire semester, keeping only submissions for any questions that appeared on both homework and exams.
We cleaned this data set by removing students who had not completed all of the exams, retaining 584 of the 601 students. In total, we retained 1,064,547 individual submissions across homework, optional practice, and exams. Each submission’s score ranges from 0 (incorrect) to 1 (full credit), with scores in-between indicating partial credit.
We subdivide our analysis of the course by exam, focusing on the three week window preceding each of the vfie exams. As shown in Figure 2, each exam is comprehensive, including material that was present on previous exams. For this analysis, we focus solely on the content introduced since the previous exam to see how practice during the homework and study periods contribute to learning for the material’s first summative assessment.
Each student submission is assigned to one of three periods: homework, study, and exam (Figure 3): Question pool composition for each exam hidden (not on homework) new (on homework since previous exam) old (on homework & on previous exam) 4 28 new E0
23 hidden 78 new 24 old E1
21 hidden 61 new 72 old E2 21 33 new 115 old E3
44 hidden 46 new 191 old E4 • The homework period includes all the submissions to homework on or before the homework due date. Submissions in this period represents required practice; while students are allowed as many submissions as they need to get full credit, there is a deadline to receive that credit. • The study period includes all submissions on practice exam generators as well as any submissions on homework after the homework deadline. The homework system remains open and students can repeat problems and complete any problems not previously completed (only 90% of questions are needed to achieve a full homework score). Submissions in this period are elective practice, bearing no credit directly. • The exam period includes the submissions on the actual exam.
The above periods are coarsely denfied to capture the difference between the time spent on required practice with homework assignment and any additional practice following the homework deadline. For our context, problems being completed by students on practice exams as well as after a homework deadline are both elective activities and are suitable to be counted together.
For our analysis, we also tag each student’s first attempt on each question on homework, so that we can estimate the student’s ability to solve that question gained before attempting the question the first time (e.g., from readings, lecture, or solving other problems). A breakdown of the number of submissions during each period is provided in Figure 4. The decrease in submissions throughout the semester in the homework and studying buckets is a result of homework shifting toward fewer, more dicfiult problems as the semester progresses.
Exam Period
To analyze the evolution of student knowledge from homework to exam time, we track student learning at the granularity of individual item generators. This is clearly a signi-fi cant approximation to reality for two reasons: 1) because of pools (of true/false and multiple-choice questions) and parameter randomization (for short answer questions) there is some variation between instances of a given item generator, and 2) there are relationships between item generators (e.g., practice on a programming question relating to loops would likely improve students ability to complete a short answer question related to loops and vice-versa).
Nevertheless, for our purposes, we believe this approach is viable. The items of each item generator were considered suficiently similar by the instructor to be fungible with respect to the exams. Furthermore, the method is robust to whether or not learning occurs between subsequent attempts on the same problem or from students attempting a problem, trying new problems, and returning again to an older problem. If the student learns signicfiantly by completing many other homework problems between two attempts at a given problem during the homework period, we can still correctly attribute the learning to having taken place during the homework period. As such, we made no attempt at topic modeling in this work. 4.1
In general, knowledge tracing (KT) techniques were developed as predictors of student performance or estimators of the latent knowledge state of students. KT is used either to estimate a student’s likelihood of getting the next attempt correct based on previous attempts, adjusting after each success and failure as the student engages with an assessment, or to track changes in students’ latent knowledge over time. Much of the dificulty of KT techniques results from attempting to instantaneously obtain a signal of student knowledge as students are engaging with learning opportunities. In our case, we already have all the data from the course as the course has ended and do not need an instantaneous, updating measure of student knowledge. Instead, we desire to perform a post hoc analysis of students’ submissions to estimate how their learning changed over an entire course’s worth of data. Our chosen method, CT, measures students’ knowledge as demonstrated by an increase in the likelihood that they would get given items correct more frequently over time.
Submissions per bucket by exam subset 178481 77048 47987
131641 40014
120436 92533 30412 175000 150000 sn125000 o i s s i bm100000 u s f o tn 75000 u o C 50000 25000 0
5871161033 67181 55526 14378 10859 14588 10936 15700 22984 7151 6948
where x1, . . . , xn is the result of a series of submissions which are either 1 (correct) or 0 (incorrect), and the method tries to nfid a series of predictions p1, . . . , pn that optimizes the loss function, under the constraints that: (1) p1, . . . , pn are between 0 and 1, as they represent an estimate of the instantaneous probability that the student would get each attempt correct and (2) p1, . . . , pn are monotonically non-decreasing, which is based on the assumptions that the attempts are made over a short enough time period that forgetting is insignicfiant and additional practice would not hurt a student’s ability to answer these questions. Since the homework, practice, and exam attempts occurred over a three-week window, during which there were a lot of related practice, we believe these assumptions are reasonable. Rather than having a model with explicit parameters as found in BKT, the method calculates the probabilities p1, . . . , pn by optimizing them directly for the target loss function. Chen et al have shown that minimizing root-mean-square error (RMSE) and maximizing log-likelihood would yield the same optimal solution under constraints specified in Equation 1.
We chose to use CT over BKT or DKT as it nicely tfi our use case. The CT method is able to nfiely locate and predict the “jumps” in a students’ likelihood of getting a question correct when analyzing the data in a post hoc fashion, which may be too precise a transition for usual predictive knowledge tracing. For our purposes, a high accuracy, post hoc model was ideal for analyzing changing student knowledge as a historical trend from our course’s data.
One important weakness of CT, however, is that it is prone to underestimate student knowledge on an incorrect first attempt because the optimizer sets the probability of correctness to be zero so as to minimize error on that attempt. Similarly, the probability on a correct nfial attempt will always be estimated as 1.0, which may be an overestimate. This potentially could be remedied by adding additional constraints to the method (e.g., limiting the rate of increase), but we did not attempt such constraints with this work. 4.2
To clarify our use of CT, we present a walk-through of how the method models our data for one individual and two questions from Exam 3, selected randomly from students whose behavior allows for representative variety in CT’s estimates. We refer to the student as Harlow, which is a name that was not present in the actual class. On Exam 3, two of the questions that were randomly selected for Harlow to complete were the programming question progLargestLessThanValue and the short answer question valueOfListReordering.
Harlow had notably diferent experiences with these questions; Figure 5 plots the correctness of Harlow’s individual submissions as dots that are color coded based on the period in which the submission occurred. With progLargestLessThanValue, Harlow made two attempts on homework to get the question correct once, got it correct once on a practice exam with a single attempt, and tried it twice on Exam 3 without getting a correct answer. With valueOfListReordering, Harlow had 9 attempts on homework with 6 correct submissions, 4 encounters across two practice exams for 2 correct submissions total, and a correct answer as the only attempt on Exam 3.
Figure 5 also shows the result of running CT as a line indicating the instantaneous estimate of Harlow’s likelihood of getting the question correct. In both cases, Harlow got the rifst attempt wrong, so the model assign’s Harlow’s likelihood of getting the question correct as 0%, so as to minimize the error relative to the actual outcome. While Harlow is iflpping between correct and incorrect attempts, the model computes a likelihood of correctness for each attempt e g d e l w on1.0 tk1.0 n e d u t0.8 .s0.8 e i. t c rr00..66 e o c m e itt00..44 x e n e th00..22 g n i t t g0.0 e fo0.0 ce 0.01 n a h C
Harlow's Knowledge Over Time progLargestLessThanValue valueOfListReordering Average "This-Item-Correct" Chance Across Periods (All Exams) that minimizes the error for those correct and incorrect attempts, constrained to be non-decreasing. Because Harlow’s last three attempts at valueOfListReordering were all correct, the model decides that Harlow has mastered the question with a 100% likelihood of getting the question correct. We ran CT for each student on each question independently. From each trace, we extract six estimates of the student’s likelihood of getting a question right: their rfist and last attempts in the homework period (First, End Homework), their rfist and last attempts in the study period ( Start Studying, End Studying), and their rfist and last attempts on the exam (First Exam, End Exam). Any student without a submission in that period (i.e., students who did not study or students who did not get that question on their exam) has their previous submission to that point in the timeline used in compliance with CT’s assumption that students do not forget. We then average these likelihoods across all students and all questions for a given exam period. This allows us to explore the changing student knowledge as an average for all the students in a course across the diferent learning opportunities presented by homework, studying, and assessment.
The results of running CT are shown in Figure 6. From the slopes of the lines, it can be seen that CT estimates that more learning is occurring (i.e., the change in student likelihood of correct attempts is larger) during the homework period than the study period. The plot suggests that the course material tends to get more dicfiult as the semester progresses, with the initial and nfial likelihood of correctness both decreasing as we move from Exam 0 to 3. Furthermore, the lines for Exams 0 through 3 show almost identical trends.
To address RQ2, we disaggregated the exam data sets by question type to see whether there was any notable diefrence between types. For this analysis, we omitted Exam 0, as Exam 0 did not feature programming questions. Figure 8 shows the per-question type CT results. The only notable finding is that diferent questions start at diferent levels of initial student knowledge and end with diferent amounts of knowledge, which changes the starting and ending points in Figure 8. Because of this, diferent questions drop of faster than others in terms of how much is learned during the practice period. Generally, students have less to learn with true/false and multiple-choice questions through the practice period than they do on programming and short answer questions, although all question types experience a learning drop-of through to the exam. 6.1
Change in average student knowledge (CT) by period Average "This-Item-Correct" Chance By Period and Question Type (All Exams)
CT attributes more learning to the mandatory homework period in this particular course. This is represented as the largest increase in student knowledge from their rfist homework submission to the last. That gives us some condfience that a course with signicfiant homework opportunities does provide students with productive chances to learn as opposed to just inundating students with “busy work.” Interestingly, CT also indicates performance on the exam is better than at the end of the study period. There are a few possible explanations for this. The most likely explanation is that, given the higher stakes of the exam, students are trying harder, resulting in a higher correct rate that is being observed by the model. In addition, some of the score improvements observed on the exam could be attributed to the last pre-exam practice attempt if, for example, the student got the question wrong, but learned from seeing the correct answer. This also might be just be an artifact of CT, as any students that have incorrect and correct attempts to a given question on the exam will have learning attributed to them. Finally, actual learning might be occurring during the exam. The amount of “learning” attributed to the exam period is, however, fairly negligible.
Importantly, one should not attempt to generalize about the learning potential of homework relative to elective practice for all courses from these results. We expect that courses that assign less homework might observe less learning during the homework period and students might compensate by studying more, thereby making more of the learning occur during that optional studying. It could also be the case that there are diminishing returns on each attempt on a specicfi question, which the rfist attempt providing the most learning benetfi, then the second, decreasing further with each attempt from homework through the study period. It is reassuring, though, to see that this course’s homework and study opportunities (i.e., the practice exam generators made available to students) both appear to contribute significantly to student learning. 0.8 0.6 0.4 0.2 1.0 0.8 0.6 0.4 0.2
FEinrsdtHomewSotrakrt StudyinEgndStudyingFirst Exam EndExam
Exam 4 Programming
Short Answer When we disaggregate the analysis by question type, the general shape and progression of results is the same for every question type compared to the source exam. Diefrent questions start with lower amounts of student knowledge, but this appears to mostly be a function of the dificulty of the problem’s type: programming and short answer questions, which require more actual coding on the students’ parts, tended to start and end lower.
The lack of diefrent behavior when we disaggregate by question type is more interesting than it may initially appear. This means that the “shape” of student learning does not difer signicfiantly with the question type. Given this, it appears that homework and additional studying have the same impact on student results regardless of the kind of question. This does mean there are diminishing returns on easier question types over the period compared to harder ones, but not a decfiiency in how homework and practice helps on question types where students still have learning they can do. 6.3
There are some obvious limitations to the current work. First, our nfidings about the relative learning during the homework and study periods cannot be assumed to generalize to other course contexts. Courses with diefrent homework, study materials, and exam structures will likely have diferent breakdowns of learning in each phase.
Second, CT is a fairly coarse measure of learning. Scores as a performance indicator are not alone proof of student learning. Additionally, CT’s potential for underestimating likelihood of correctness of rfist attempts (by strictly optimizing for RMSE) could make the model overestimate the learning that is occurring in the rfist few attempts, which is likely occurring in the homework period. We do not have condfience that these measures of learning are particularly precise. While we omit it from the paper, we also ran a regression model to estimate the learning in the same periods of the course. The regression generally showed the same trends as CT, giving us more condfience in CT’s results. Finally, these methods do not disambiguate from learning that happens during the homework and studying periods and learning that occurs specicfially from homework and elective practice problems. There are notable reasons to believe that students are learning signicfiantly from reading the textbook, engaging in active learning exercises, and, perhaps, even from listening to the lecturer speak. The learning that occurs during these activities is attributed to the period in which it occurs, rather than to the specicfi task.
In this work, we explored the degree to which we can attribute student learning between required homework and elective study performed prior to a summative assessment. To analyze learning, we used a post hoc method of “this-itemcorrect” likelihood (correctness tracing) to estimate student knowledge. We found that (required) homework and (elective) studying both contributed signicfiantly to student learning, with homework contributing slightly more. Further, despite using multiple question types, we found the most notable diference between question types is where student knowledge starts and not the shape of their learning improvements.
We think that our results show that frequent, exam-relevant homework and highly-accessible means for study (e.g., practice exam generators) are both efective means of facilitating student learning and believe that these nfidings could generalize to other contexts. The magnitude of learning from each component may diefr, but courses with similar homework and studying opportunities will hopefully see similar learning gains during each period.
There remain areas for future work. Considering data, we only use students’ submissions to questions that also appear on homework. Some ability to include other learning events, such as reading a textbook, would give a clearer picture of students’ learning process. Additionally, some topic-level labeling might allow us to include questions unique to exams in our data and analysis.
With respect to CT’s model, we made no attempt to compensate for the method’s tendency to underestimate on initial incorrect attempts. Future work could investigate constraining this behavior by limiting the allowable slope. Further, there is room to adapt the model to using a richer source of information than students’ correctness on submissions — for example, by tfiting a similar optimization on students’ knowledge as estimated by methods such as Item Response Theory (IRT). 8.