This paper uses a case-based approach to investigate the impact of data-driven positive feedback on students' behaviour when integrated into a block-based programming environment. We embedded data-driven feature detectors to provide students with immediate positive feedback on completed objectives during programming. We deployed the system in one programming homework in a non-majors CS class. We conducted an expert analysis to determine when data-driven detectors were correct or incorrect, and investigated the impact of the system on student behavior on the homework, speci cally in terms of time they spent in the system. Our results highlight when data-driven positive feedback helps students, what happens when it goes wrong, and how this impacted students' programming behavior. Results from these case studies can shed light on the design of future data-driven systems to provide novices with the positive feedback that can help them persist while learning to program.
Block-based programming environments are intended to
provide novices with the ability to engage in motivating
openended and creative programming with features that limit
syntax errors but allow for simpli ed programming for
interactive media [
While prior work has evaluated the quality of automated
feedback, or its impact on students' performance or
learning, not much is known about the impact of this feedback
on students' programming behaviour when it fails to
provide reliable feedback. This needs more investigation to
understand what measures can be taken or what support
should be provided to students to mitigate adverse e ects
of data-driven automated feedback. Our prior study showed
that data-driven positive feedback increased students'
engagement with the programming task, and improved their
programming performance [
Several block-based programming environments were designed
to reduce the di culties students face while learning a new
programming language in various ways. For example,
Alice [
To provide novice students with individualized tutoring
support, researchers have integrated intelligent features into
block-based programming environments. These intelligent
features dynamically adapt teaching support to mitigate
personalized needs [
The impact and e ectiveness of tutoring supports and
intelligent features integrated into novice programming
environments have been explored by researchers from various
perspectives. Zhi et al. [
We built the data-driven positive feedback system (DDPF)
in iSnap [
We built a system to provide positive feedback while
students program a speci c exercise in the block-based
programming environment, using a data-driven feature detector
algorithm, described in [
We designed an interface, based on our prior work [
We deployed our system in an introductory computing course for non majors in Spring 2020 in a class of 27 students, which took place in a public research university in the United States. In this course, students used iSnap (Section 3.1) to solve their in-class programming assignments and homeworks. We integrated our DDPF system into the programming environment for one homework called Squiral, described in Section 4.1. The data we collected consists of code snapshots for every edit in student code with corresponding timestamps. We also logged all the objective feature states of all students' code snapshots with the time when every objective was completed or broken. Afterwards, we manually checked the sequential code snapshots for each student and documented the following: Early, Late, Incorrect and Just-In-Time Objective Detection: We investigated the code snapshots for each student, and ltered out these snapshots where the system detected the completion of an objective. Two researchers evaluated whether each detection was early (i.e. the objective was detected before the student completed the objective), or late (i.e. the student completed the objective earlier than when it was detected by the system), incorrect (i.e. an objective was detected that was never completed), or just-in-time (i.e. the objective was detected at the step when the student just completed it).
Agreement between Researchers and Automatic Objective Detection: To measure the agreement between researchers and data-driven objective detection, we marked each step of a student's code with: true positive TP, where both the researchers and the system detect the completion of an objective at the same step, or true negative TN, where both the researchers and the system detect that an objective was broken at the same step, or false positive FP, where the system detects the completion of an objective; however it was not detected by the researchers, or false negative FN, where the system detected an objective as broken at a step where the researchers detect no broken objective. Idle and Active Time : We measured the total active and idle time spent by each student on the system while solving the programming homework. We also measured, for each student, the active and idle time spent before each objective was detected by the system and the total active and idle time spent before the last change to any objective was detected by the system. We chose a time gap of greater than 3 minutes [75 th percentile based on the frequency distribution of di erent amount of time gaps] to be considered as idle time. Also, a time gap of greater than 10 minutes [95th percentile] was considered to be the start of a new session and thus, was not added towards either active time or idle time.
The Squiral assignment is a programming homework that asks students to create a procedure to draw a spiraling squarelike shape. One possible solution and its corresponding output is depicted in Figure 1. Using correct student solutions collected from prior semesters, four objectives were identi ed [described in section 3.3] and were provided to the students as sub-goals to achieve while solving the problem. The speci c features required to complete each objective are shown in Table 1.
The goal of this study is to explore the impact of automated data-driven adaptive feedback on students' programming behavior. To achieve the goal, we aimed to answer the following research questions: RQ1: How it Helps.
How did data-driven feedback help students complete the assignment? RQ2: What happens when it Goes Wrong. How did the objective detectors impact student behavior, especially with regard to di erences between researcher and algorithmic objective completion detection? RQ3: How Students Respond.
How did data-driven feedback impact students' active and idle time while solving a programming problem?
When we observed each student's solution at the end of their attempt, we found out that 25 out of 27 students who were provided feedback had working solutions of the Squiral assignment at the end of their attempt, although, according to researchers, their solutions were not always perfect from a logical perspective [For example, using `size x size' in the nested loop instead of `rotations x 4', where `size' = 10 and `rotations' = 25, will produce the same Squiral. But, it is not logically correct, since the purpose of the nested loop is to draw 4 sides of the Squiral in each rotation.]. Among the 27 students, two students (Jade and Lime) attempted to solve Squiral using the system both with and without feedback. To get speci c insight on how the system helped the students to reach a correct solution, we examined code logs of these two students who each attempted to solve Squiral without Data-Driven Feedback, and failed to complete the assignment. Later, those two students each attempted to solve Squiral with Data-Driven Feedback and succeeded. In this section, we present the case studies of these two students to demonstrate how the feedback system helped the students to ll the gaps in their code and led them to working solutions.
Student, Lime, without Data-Driven Positive Feedback: Student Lime, when attempting to solve Squiral without any hints or feedback [Figure 2] given, used a custom block with a parameter. The student used `move' and `turn' statements within a loop in the custom block. However, there were three gaps in the code that the student could not gure out. First, a nested loop was required to iterate for `rotation count x 4' times. Second, the move statement used `length x 2' as its parameter whereas only `length' would be su cient. Finally, the variable used in the `move' statement must be incremented at each iteration. The student spent 18 minutes and 18 seconds before giving up, being unable to gure out these issues.
Student Lime with Data-Driven Positive Feedback: Later, student Lime attempted the homework again after receiving notice that the DDPF system was made available. When Lime received feedback, they gured out the 3 issues and reached a correct solution [Figure 3]. The second objective suggests that there is a correct number of rotations that is needed to be used within the custom block. With this feedback, Lime used `4 x Rotations' in the `repeat' block instead of using `15' and completed the second objective. The third objective suggests the use of a variable in the `move' statement. Lime used an initialized variable `length' in the `move' statement instead of `length x 2' and the objective was marked green. Finally, Lime incremented `length' within the loop and all objectives were completed and they reached a correct solution. With data-driven adaptive feedback, Lime spent 29 minutes 51 seconds before reaching the correct solution. Recall that Lime gave up with an incorrect solution after around 18 minutes when no feedback was given.
Student Jade without Data-Driven Positive Feedback: Student Jade initially attempted to solve Squiral without data-driven positive feedback, spent 16 minutes and 55 seconds before giving up with an incorrect solution. Jade's code [Figure 4] contains `repeat', `move', and `turn' statements on the stage. Jade created a custom block and only used the block to initialize a variable, `length', that was also a parameter to the block. The components to complete the objectives were partially there in Jade's code but it su ered from organizational issues. Also, Jade couldn't gure out that the `move' statement should use a variable instead of a constant and the same variable needs to be incremented at each iteration. The number of repetitions in the repeat block was also incorrect.
Student Jade with Data-Driven Positive Feedback: Like Lime, Jade attempted the homework again when the DDPF system was provided. When given feedback, Jade rst created a custom block and used it on the stage which got the rst objective marked green. The second objective hints on using a loop that repeats for correct number of rotations within the custom block. This time Jade implemented the loop within the block and got the second objective correct. Within the loop, Jade used `move', `turn', and `change' statements and reached the correct solution [Figure 5] with all objectives marked green. With adaptive feedback, it took Jade 14 minutes 29 seconds to reach a correct solution. Whereas without feedback, Jade gave up with an incorrect solution after spending over 16 minutes on the problem.
The 2 case studies of Lime and Jade presented in this section demonstrated that the feedback system was able to help students in lling up the gaps in their code to reach a correct solution. In one case, this achievement came at the cost of a higher active time and in the other case the student reached a correct solution in less time when feedback was provided. Also, we observed most of the students had a working solution [capable of drawing a Squiral] at the end of their attempts, although not logically 100% correct according to researchers. Moreover, almost all (25 out of 27) the students explored syntactic constructs required to complete the objectives, (e.g. move, turn, iteration, variables), which potentially indicates that they closely followed the objectives to accomplish the assignment.
To answer RQ2, we manually walked through the sequential code snapshots of each student when they attempted to solve Squiral and generated case studies where the system went wrong and observed students' problem solving approach. Below we present three case studies demonstrating our system's potential impact on student behavior when the system could not provide reliable feedback. We selected one case where the student completed an objective but the system could not detect it (FN case) and two cases where the system detected completed objectives when the objectives were not completed (according to researchers), which led students to an incorrect solution or made them stop early.
Student Azure started solving Squiral by creating a custom block and got the rst objective correct. Azure used 2 parameters, `size' and `length', to denote the number of rotations and length of the rst side of the innermost loop. They created a loop with a `repeat' block with the correct number of rotations (`size x 4') and got the second objective correct. As Azure used the `length' parameter in the `move' statement within the loop, they got the third objective correct. Then Azure added a `turn' statement and incremented the `length' variable. At this point, the fourth objective was completed, according to researchers. However, the objective was undetected by the system [FN case], because Azure used a `turn' statement that was di erent from those used in the previous students' solutions [that were used to extract and detect the objectives]. According to researchers, Azure's solution was 100% correct at this point [Figure 6]. It took this student only 2 minutes 24 seconds to reach the correct solution.
However, the fourth objective was not detected by the system. Azure kept working on their code. Azure made several changes to their code which led them to an incorrect solution. Finally, Azure ended up submitting a solution that was also 100% correct according to researchers, but was slightly di erent from their initial solution. In the submitted solution, Azure removed the `length' variable from the parameter list of the custom block. The fourth objective was still undetected. While doing these changes, the student spent 12 minutes 5 seconds more in the system which is almost 5 times the amount of time the student spent to get a correct solution in the rst place.
We observed a similar situation in case of student Blue who worked for a total of 1 hour 43 minutes 16 seconds. Blue reached a correct solution at 1 hour 7 minutes 11 seconds according to researchers. But one objective (fourth objective) was undetected. Blue kept working for another 36 minutes 5 sec (almost 50% of the time taken to reach the correct solution at rst attempt).
Student Cyan created a custom block and used it on the stage and got the rst objective correct. Cyan used two parameters in the custom block and used one of them in a `move' statement within a nested `repeat' block that got him the second and third objective correct. However, Cyan implemented another nested loop and added a `change' statement within that loop in the stage instead of adding them to the custom block. The system detected the objective and marked the fourth objective green. At this point, the student Cyan had all objectives correct [FP cases] but the code [Figure 7] was unable to draw a Squiral.
Later, Cyan removed the `change' statement from the stage that caused the fourth objective to be broken. However, removing the custom block from the stage was causing other objectives to be broken. Cyan then moved the `change' statement to the custom block and corrected the rotation count in the `repeat' statement. At this point, the solution was correct and similar to Figure 1a. The incorrectly detected objectives led Cyan to a non-working solution. In this case, Cyan had to ignore the detectors and do extra work to reach a correct solution.
We observed a similar situation in the case of three other students. One of them got four objectives correct and the code was able to draw a Squiral, but according to researchers the code had one programmatic problem. The code was drawing three sides of the Squiral using an inner loop and one side manually. The problem detected by researchers remained undetected in the system and the student ended up submitting a partially correct solution. For the other two students, four objectives got detected when there were syntactical problems present in their programs, as for student Cyan. The students realized that completing the objectives did not necessarily mean that their code could draw a Squiral. Although FP cases led the students to incomplete code with four checked objectives, incorrect output eventually compelled each student to modify their codes and to reach a 100% correct solution at the end.
dents to Stop Early at a Partially Correct
We found 6 additional cases where the students got 4 objectives correct, but the code was partially correct (FP Case) according to researchers. However, their programs were able to draw a Squiral as required. In such cases, the students nished the attempt early and submitted the partially correct solution. We present the case study of Student Indigo here.
Student Indigo's solution [Figure 8] had objectives 1, 3, and 4 correctly completed according to researchers and the objectives were detected by the objective detection system as well. Indigo created a custom block and used it in the stage [required to complete objective 1]. They used `pen down' and added `move', `turn', and `change' statements accordingly [required to complete objective 4] within a nested loop implemented with two `repeat' statements. In the `move' statement, Indigo used an initialized variable, `Length' [required to complete objective 3], and incremented the value of `Length' by 10 at each iteration. However, the rotation count used in the nested loop was wrong. One of the `repeat' statements should have the count of rotations and the other should have a constant 4, indicating the 4 sides of the square drawn at each rotation. The objective detection system detected the use of the nested loop and marked objective 2 green [FP case]. The code was able to draw a Squiral. However, the implementation was not completely correct. But, once the student Indigo got 4 objectives correct, they submitted their solution.
We observed cases where students reached a correct solution, but could not recognize the correct solution because their completed objectives were not detected by the system. They continued working on the assignment for a longer time than was necessary. We also observed cases when students got 4 objectives correct but their solution was not even working, i.e. their code was not able to draw a Squiral because of organizational or syntactical problems. Only in these cases, the students realized completing the objectives is not enough. Thus, they modi ed the code overriding the objective detectors to reach a working solution. Our third case study showed that students who got 4 objectives correct with a working code submitted their solutions even if their code was not fully correct according to researchers. In these cases, all the students relied on objective detection and submitted partially-correct solutions. These case studies potentially indicate students' high reliance on objective detection feedback, since they did not seem to question the feedback system as long as they had a working program that drew the correct shape. However, students did not rely on their own skill to determine when their code was correct. Furthermore, students sometimes even ignored the produced output that showed they had a working solution, in the cases when objectives were not detected.
To answer RQ3, we investigated how the correctness of objective detection at early phases of a problem solving attempt impacted later phases of the attempt. We divided the total time each student spent on the system into three phases [Figure 9] - a) Phase A : when objectives were detected for the rst time; b) Phase B : changes in previously detected objectives were detected. c) Phase C : students spent time in the system but no change in the objectives were detected. For example, a student got objectives 1, 3, and 4 marked green within 20 minutes of starting the attempt [Phase A]. Then the student continued working for another 10 minutes [Phase B] but did not see a new objective [objective 2] go green. However, the previously detected objectives were broken and corrected several times. Finally, the student spent another 5 minutes [Phase C] when no change in any of the objectives were detected. We tried to relate correct, incorrect, early, and late detection ratios in phase A with the active and idle time spent in phases A, B, and C to understand if correct, incorrect, early, or late detection regulates the time or e ort students put on the assignment.
Active and Idle Time Observed in Phase A: In this phase, objectives got detected for the rst time by the system. Students were observed to take wide ranges of time before an objective was detected [Table 2]. We plotted average active and idle time against correct objective detection ratios and observed that students with higher correct objective detection ratio have shorter phase A in terms of active time [Figure 10a]. For 1 of the students, this phase did not occur because an objective was never detected in the student's code. In this phase, only a few cases were found when the students had idle time. 16 out of 27 students did not have any idle time at all. 7 students had idle time ranging from 3-5 minutes. The rest of the 4 students had idle time ranging from 10 - 24 minutes. We observed that a higher early detection rate (over 25%) has a decreasing trend in average idle time [Figure 10b]. This may potentially indicate that positive feedback can be motivating to students, even if it is provided early.
Active and Idle Time Observed in Phase B: For 11 students, phase B did not occur at all, due to either the fact that an objective was never detected, or the students submitted their code after all of the objectives were detected for the rst time in phase A. 10 students spent >0 - <10 minutes, and 6 students spent >10 - 35 minutes in phase B. 4 of the 6 students who spent a higher amount of time in this phase B had a high early detection ratio at phase A (50-75%), and 1 student had a high incorrect detection ratio (50%). These students did not have correct solutions, even if some or all of the objectives got detected in phase A. In this phase B, 21 out of 27 students had no idle time. 6 students had idle time ranging from 3 to 24 minutes. As we plotted average active and idle time in phase B against the correct and incorrect objective detection ratio in phase A, we observed a higher correct detection rate (>25%) in phase A seemed to decrease the active time spent in phase B [Figure 11a]. This means that correct detections helped students complete their programs more quickly. Correct objective detections in phase A pushed the students towards the end of their attempt. However, incorrect objective detection in phase A decreased idle time in phase B and caused the students to continue actively working [Figure 11b]. Active and Idle Time Observed in Phase C: This phase C indicates the time when no change was detected in any of the objectives. In this phase C, one of the following scenarios occur : 1) Most of the objectives were detected in earlier phases; the student had a working solution and was making minor modi cations without impacting the objectives; or 2) One, more than one, or all of the objectives were undetected and the student was working on the assignment but submitted the attempt without another objective being detected. We observed that when the rst scenario occurs, students spent only a few minutes in phase C and submitted their code. 18 out of 27 students spent only 0.1-8 minutes in the system after scenario 1 occurred: even if the objective detection was wrong, the student relied on the system, and submitted their code. We observed a higher early detection ratio in phase A led to decreased average active time in phase C [Figure 7]. The remaining 9 students spent 12-56 minutes in this phase. Scenario 2 played out for 7 of these 9 students, who all submitted the program with some incomplete objectives.
The results of our analysis showed that the active and idle time spent in Phase B and Phase C are associated with the quality of detection in Phase A in some cases. We observed that correct objective detection in Phase A that led to a working solution pushed students to nish their attempt, making Phases B and C shorter. Whereas, incorrect objective detection in Phase A that led to a non-working solution decreased the idle time observed in Phase B and students worked more. However, the active time in such cases varied from student to student, and depended on the extent to which objective detection went wrong. Our case studies presented in Section 6 demonstrated indication of students' signi cant reliance on the feedback. This reliance on the system interacted di erently depending on whether the feedback was correct or incorrect, and whether or not the student code output appeared correct, and these di erences are re ected in students' response or e ort in terms of active and idle time.
Our case studies and analysis demonstrated that, although our feedback system could guide students to complete a programming assignment, it could mislead students to do extra work, submit a partial correct solution, or to end up at a non-working solution in the event it provides inaccurate feedback. However, the visual feedback of objectives going green [even for an incorrect or too-early detection] reduced idle time, giving us an indication that the feedback was motivating for students. All these observed impacts could be the result of students' high reliance on the feedback system. To prevent the negative impacts generated in the event of incorrect detections, mechanisms to prevent incorrect detections must be explored. We observed our system to fail in the event of new student behaviours. Since we built interventions that use data from prior students and new students behave in new ways, the system has not had time to learn and adapt based on their behavior. Thus, any system like ours must have an iterative process for integrating new behaviors that may arise from diversity in students and instructors.
This paper presents case studies to provide important insights into the impacts of positive feedback on novice programmers from multiple perspectives. We present case studies that shed light on how the feedback system helped two students to complete a programming task who failed to complete the task on their rst attempts without feedback. While these scenarios highlighted the usefulness of positive feedback, our case studies surrounding the event when the system could not provide accurate feedback provided insights on what impact feedback failures can have on students' responses. These insights can be highly useful to decide on measures to mitigate an adverse impact or to formulate adaptations to handle unexpected behaviors. This may involve expressing a con dence level for detectors, or inviting students to self-explain how and why their solutions are correct. The primary contributions of this work are: 1) Case Studies demonstrating how the positive feedback system can induce a working solutions for a programming task; 2) Case studies and code trace-based analyses that gave important insights on how a data-driven positive feedback system impacts students' behaviour when the system goes wrong. Our results show interesting relationships between correctness of the provided feedback and the time students spent on the task or in the system. In our future work, we plan to explore these impacts in larger controlled studies and on other programming tasks, and we also plan to explore how we can adapt our system to balance students' understanding of their own code with reliance on feedback, to promote learning. 10. ACKNOWLEDGMENTS This material is based upon work supported by the National Science Foundation under grants 1623470.