As programming activities become integrated into K-12 classrooms, teachers with minimal prior programming experience will instruct students of varying backgrounds. This usually takes the form of Live Coding, programming on a display in front of students while they follow along. While e ective, Live Coding prohibits teachers from seeing students current progress. Though researchers can observe instruction and can tell teachers when students need the pace altered, this help will not always be available. To improve and scale insights and investigate how teacher instruction in uences student programming behavior, we perform several analyses to visualize the in uence of instruction on student assignment progression throughout the class period. We present case studies featuring teacher and student programming behaviors. We end on possible means our methodology can be extended to create dashboard and other data-driven tools.
1. INTRODUCTION
1To address inequities in access to computing education [
If children get experience with programming, it is usually
within elective computing classes or outside of school
activities [
Teacher's self-e cacy and discomfort are important factors
when trying to get core subject teachers to add
computing into their classes. Our prior work has demonstrated
the success of lessons often hinge on a teachers comfort
with instructing, and often their self-e cacy is very low
as a novice programmer[
There are dozens of systems that analyze and collect student
programming data (an introductory review can be found
here: [
The dataset used in this analysis came from several
implementations of a 4-day computing-integrated science lesson.
Nine instructors across three schools led 19 middle-grade
classes in a `Food Webs' unit using a block-based
programming environment, Cellular [
The Food Webs unit is a 4-day activity designed to meet the state-level education standards for 6th-grade science classrooms on the topic of Energy Transfer in a food web. On the rst day of the activity, students use pseudocode to complete scienti c worksheets on de ning terminology and relationships between organisms in a food chain. Students work as a class to abstract the behaviors that an organism in the food chain might exhibit (eat, grow, move, etc.). On day two, teachers lead students in using their previous pseudocode to model the behavior of sunlight as an abiotic factor in the food web and its e ect on producers (plants) using the Cellular programming environment. Using a companion worksheet, students manipulate energy values in the code to see its e ects. On day three, teachers help students add in code for a primary consumer (bunny), again exploring the propagation of energy in the food chain. Finally, on day four, teachers instruct students on how to create the basic functionality for a secondary consumer (fox) as shown in Figure 1. Students then set up their own scienti c experiment determining an independent and dependent variable, as well as their hypotheses on what will happen. Students are encouraged to run several trials of their experiments before declaring a de nitive conclusion.
The Food Webs curriculum was chosen as we have data from the largest set of unique teachers as well as the largest number of student participants. We choose the 4th day rather than the other two programming days for our analysis because as part of our design-based implementation research, we implemented di erent versions of Days 2 and 3 for di erent teachers to experiment with di erent instructional scaffolding techniques. However, every teacher implemented the same nal activity. As Day 4 is also the nal day of the activity, teachers have had multiple class periods of experience instructing programming assignments and Day 4 serves as a culmination of their nal abilities.
The primary data used for our analyses is environment trace data from the consenting students working within the assignment. The 19 classroom implementations of the assignment resulted in over 90,000 interactions from 287 unique students. To train the teachers in instructing, instructor guides were developed. These gave the interactions with the environment (code additions) necessary to complete the assignments, included images of example nal code, and a suggested order of how to add code in their environment for instruction. Using these instructor guides, we decompose the assignment into ve unique, mutually exclusive features. These features demonstrate the ve additions that teachers led students into creating in the Cellular Environment. These features can be detected using an auto-grader type system and we can determine at any given point in a students programming trace whether or not that feature is present or absent in the code. Figure 1 presents a short description of each feature alongside the corresponding code blocks. Feature 1 tasks students to create a local variable, Energy, that will represent how much energy a Fox agent has in the Food Web. The instructions explicitly state that the variable must be local as it represents an individual fox's current energy. Furthermore it must be named \Energy" due to a case-sensitive dependency in custom blocks introduced later in the program. Feature 2 tasks students to make the Fox move within a forever loop, as to simulate a Fox moving around in a biome. The 3rd feature has students decrement the Energy variable inside the forever loop to simulate the Fox losing energy with each move. The 4th feature introduces the rst If block - If the energy of the Fox is low (but not 0), the Fox \eats a bunny" in the environment and gains some energy. The nal feature, Feature 5, is a second If block where if the Fox's energy is less than 0, the Fox will die. We use code feature detection to determine, for each code state in a student's code-trace, whether or not each feature is present or absent in a student's code. We then use this information to develop the following three analyses: 1. Feature-Time-Completion - A histogram that visualizes when students complete a given feature relative to the feature completion time of the instructor. 2. Cumulative-Feature-Completion - A stair-step graph indicating when students add features to their code during the class relative to the instructor. 3. Feature-Path-Graph - A state-space representation outlining what features students add in what order, showing the varying paths they take compared to the path taken by the instructor.
Our rst methodology, Feature Time Completion, produces
a histogram showing the timing in which students complete
a feature relative to when the instructor completes the
feature. In the three graphs in Figure 2, we present exemplar
cases of the \I do, We do, You do" instructional strategies
introduced in the Gradual Release of Responsibility (GRR)
model [
Our second representation shows how students cumulatively add features to their code relative to each other as well as relative to the instructor. Using the timestamps of when students add their next feature to their code, we are able to chart out in a Stair-step fashion students progressing through the assignment, going from having nothing complete to having more and more of the 5 features added. We are also able to see based o the nal placement of each line, how many features students ended the lesson with. In the 3 cases and 4 classrooms described below, each student trace is an individual stair-step function. Instructor feature completion is denoted by the dashed vertical lines, and comparing when teachers and students complete features (in combination with the additional classroom observation data) allows us to understand di erent instructional patterns and their consequences.
Figure 3 describes one teacher doing the Food Web Assignments with their class. After the teacher completes their rst feature (with the majority of the class following very closely, within 2 minutes of him completing it) 15 minute without progress pasess. A strand of students make progress independently of the teacher, but the majority are stuck at only completing one feature. From classroom observations, we know that the majority of students had followed the instructor in naming their variable \energy" (though the instructional guide says to name it \Energy" as custom functions expect that name and are case sensitive). A strand of students are able to recognize the error from their student guides and are able to independently make progress. At around 10:00, the teacher xes the error, and announces it to the class). The wide range of students jumping in progress at varying di erent times re ects a change in instructional style adopted by the teacher. As it was a simple naming x that a ected every other Feature, xing it resulted in making progress on multiple features near simultaneously (seen in the tall jumps in each stair step graph).
We now present a case study of the same instructor teaching two di erent periods. As part of the faded sca olding teaching approach, a research member (Teacher C) taught (a) TeacherD - I do (b) TeacherA - We do (c) TeacherH - You do the rst class period (Period 2) where Teacher E observed. Teacher E then taught Period 4. Teacher E requested that Teacher C instruct the next period (Period 5) to observe once more how the lesson should go and then afterwards, the teacher instructed Period 6. From Figure 4, we can see the di culties that Teacher E had with their rst attempt teaching. Teacher E completes all of her features for the class after nearly 45 minutes of instruction with sometimes nearly 8 minute gaps between feature additions. Many students are intently following Teacher E adding a feature when she does, but the number of bands that seem to be moving independently before the teacher suggests that the students were at varying paces not at the instructors. This is backed by the classroom observations with researchers noting that some students were ahead of the teacher and the teacher actually asked students for help on what to add next in the sequence. In the next attempt, Figure 5, Teacher E was able to complete features much faster, adding 4 of the 5 necessary features in an 8 minute span of time. This latter period dramatically changes the pace at which the students add the features, and the closeness of bands suggests that students were more intently following the instructors pace. There are common patterns that can be seen across both graphs. One is the presence of outlier students who complete things at a dramatically faster pace than the teachers. From the two graphs, it also should be noted that the teacher does not demonstrate the last feature to the class (unlike in her previous teaching period), and many of her students therefore do not get all 5 features in their code. We do acknowledge two outliers, one who jumps 4 features (around 11:15) and the other who more gradual progress, but still faster than the group. Classroom observations con rm that that student was copying code that the previous instructor had left on the projector during the transition period.
We now highlight a period taught by Teacher F, a research instructor with a large amount of prior informal computer science teaching experience. Teacher F elected to lead the assignment without demonstrating to the students what the code should look like. Instead, she instructed students in what features they should be adding next, facilitating, and helping students by walking around the room. After each feature, she would then recap verbally with the whole class to review what they needed to have done in order to add the feature before moving on to the next one. We can see that student progress is very independent of one another, with the timing of each successive feature addition (i.e. when students go from having 1 to 2 features) varying widely. While students add their rst feature within 5 minutes of each other, Students add their 2nd, 3rd, and 4th feature within 10 minutes of each other. This re ects the choice of instruction style, with Teacher F walking around the room independently helping students. In contrast to the end of Teacher G's lecture, progress in Teacher F's class is limited, usually, to only adding the next feature and nothing else. This re ects the teaching style of only giving students limited information about what the next tasks are, focusing the instruction on one feature at a time. Teacher F's students that complete their fth feature (adding death), do it near simultaneously, re ecting the change in instruction style recorded where the teacher instructs the class in completing the feature together.
In our nal visualization, we remove the temporal information of the last two graph-types and focus solely on Featurepaths, the order in which students add features to their code. In Figure 7, we highlight three Feature State representations of three classes taught by three di erent instructors (Teacher C, B, and H). Each state in the graph represents a unique set of features complete by a student. For example, following the path of Teacher B (the states marked 'TB') in the second graph, the states progress from the one with 0 features to the one having only feature 1, to having both feature 1 and 2, to having features 1,2, and 3, and ending having features 1 through 4. Conversely, the state spaces for Teachers H and C show di erent sequences of feature additions (for C in the order 2,1,3,4,5 and for H in the order 4,2,3,5). Common state transitions, (those where a large proportion of students transitioned) are marked with the Feature that was added during the transition. The size of each state represents the proportion of students in the class that visited that state. As some students do not nish the assignment fully (getting \stuck" at certain points) we represent these as \stuck-states" in which a grayed proportion of the state represents how many students who visited that state did not The three feature graphs show di erent instructional strategies at play. The rst, TC, show most students following the path of the teacher, adding features as they do. However, it also includes many individual paths taken by students not necessarily following the teachers instruction sequence explicitly, but following the general direction, denoted by the number of 'con uence' points or rejoining back into the Teachers state of various di erent paths. This classroom had the largest proportion of students actually nish the assignment compared to the other two classrooms. Teacher B's more narrow state-space with the largest states being those that the teacher traversed suggest that most students were following along with the instructors code additions (corroborated by Figure 4 as well as observations), adding in the order that they did. While Teacher H had a subset of students add features in the order they did as well as a subset chosing a separate path, these paths seemed to converge in state 2,3,4 then leading to the addition of Feature 5. While Teacher H and C both show variance in student feature paths, it should be noted that both Teacher H and B did not add all 5 features to their code (Teacher H did not add Feature 1, and Teacher B did not add feature 5). As such, all students in Teacher B's classroom and all but one student in Teacher H's class failed to complete all features of the assignment highlighting scenarios in which intent direct mimicry did not fully bene t students.
As shown in Figure 2, while there are indeed examples of \I do" sequences, where a teacher demonstrates a task and then students replicate it, we also nd examples of \We do" and \You do" sequences rounding out Pearson's Gradual Release of Responsibility model. Our observations triangulate the \We do" pattern to actually involve the class working together with the teacher to solve a problem, as opposed to Pearson's more traditional interpretation of the teacher working with just a small group of students. We nd \I do" and \We do" strategies often adopted for the rst features that students work on together in the assignment with the instructor with \You do" strategies prominent in features later in the assignment (the last features students work on). This would corroborate teaching progression strategies pushed for in the GRR model, transitioning students from \I do" to \We do" to \You do" feature completion throughout the course of an assignment.
By separating out the individual traces of each student into their own function, we can see how a student temporally progresses through the assignment in relation both to the teacher and the other students in the class. From the student perspective, we see many examples of students who work in lock-step with the instructor (or with each other), adding features as they do. But we also see students who trail and lag behind as well as students who rapidly add features before the instructor has even introduced them. This di erentiation is to be expected in a core classroom where students have varying access to computing outside of school and therefore di erent experiences with block-based programming environments. The ability to identify students who are not following along with the instruction, either as the pace is too fast or too slow, can aid instructors in identifying when to create more extension activities to high achievers or when to give more support to students in need.
The state-space focuses on which features are being added in what order, and can grant information about the way teachers set up instruction (how they choose to pace the students through the task). Very narrow state-space graphs like shown for Teacher B can demonstrate a classroom that is explicitly following along with the instructions as sequenced by the teacher (even if they might be working on the tasks at di erent times). Conversely, wider state-spaces like those shown for Teachers C and H can be more indicative of classrooms where students had more agency over the order in which they choose tasks. We make no argument that more explicit instruction, or a higher path \follow" rate leads to either more or less students completing the assignment. Teacher B and Teacher H had a high proportion of students following along with their path, however, this might have left students unable to complete the assignment independently as neither teacher added the fth and nal feature.
Looking through the teacher perspective, we can identify patterns indicative of increased teacher discomfort. As mentioned in Teacher G's case study, as he erred in making the variable, he his following students all at-lined in their coding progress. This highlighted the loss of the teacher as the \keeper" of knowledge and exposed the vulnerability that they had to problem-solve along with the students, thus incurring pedagogical emotional discomfort. Similar signs of discomfort are suggested in Teacher E's case study where there are long pauses between teacher actions and student reactions. These graphs contrast Teacher F's, the experienced computing instructor, which had students clustered on the same features, moving at a steady interval. Teacher F did not feel the need to explicitly direct students with live coding, but instead was able to guide their exploration through facilitation. Conversely, we also discovered evidence of change in instructional practice, both within a class period and between them. In many cases, these changes lead to more students following along and getting through the material. As the anxieties that novice teachers tend to have focus on not doing things 'correctly', showing evidence of positive change between implementations could potentially inspire con dence in novice teachers that they are capable of instructing computing curricula.
The drawn conclusions that we outline in our case studies are supported in part by our classroom observations, corroborating what we nd within the data-driven visualizations. Because of the nuanced and varied ways in which classroom behavior can manifest within the data, it is important that we frame these results with our understanding of what happened within the classroom. As such, we understand the reliance our methodology has on observational data, and make no advocacy that our method acts as a complete replacement for it. In the same way that no single one of the three graphs tell the full set of information of a classroom, we believe that these quantitative analysis methods act as an additional tool to act alongside qualitative data collection methods, ones that scale much better though than the observational-based ones. We also understand that other limitations present themselves with aspects of our methodology such as only focusing on one lesson with only 5 possible features. In addition, only having the set of consenting students out of the entire classroom trace data could have created a sampling bias that might alter any and all of our graphs and therefore our analysis. We intend to try and o set these e ects by replicating this with the multitude of lessons that are being developed by our teacher partners in the upcoming academic year. Our methodology described is agnostic to the assignment, the features, and how they are tested for (though we acknowledge that the feature state points are only as good as the tests you run on the code).
Our initial results provide a foundation for further analysis into in uence of one actor (e.g. teacher) on another (e.g. student) during programming. For example, as more schools begin to adopt pair programming models of instruction, it would be worth investigating these collaborative interaction e ects. This current work has practical applications for our context in teacher training. As these graphs, charts, and other gures can be generated rather quickly, teacherfriendly versions can serve as additional means of explaining to teachers how the experience went and how the classroom acted. This data can help teachers inform their practice and provide immediate or post-hoc help. Further, additional processing can translate many of these graphs into animations that can give a 'live feel' of how the instruction played out in real time. Setting times for feature completion targets for an instructor can keep them on pace in tight classroom periods, and windows for instructor-student feature completion can be used to determine if the class is following along intently. Often, the instructor participants in this study looked to researchers for a simple 'Thumbs Up or Thumbs Down' measure to see if they were doing things correctly and kids were following along. If a simple noti cation such as the one described were implemented into a dashboard system, we can imagine it bene ting instructors of K-12 programming.
Mehak Maniktala (mmanikt@ncsu.edu), Danielle Boulden (dboulde@ncsu.edu), Eric Wiebe (wiebe@ncsu.edu) and Ti any Barnes (tmbarnes@ncsu.edu). This material was supported by the National Science Foundation under grant 1837439. Anything expressed in this material does not necessarily reect the views of the NSF.