=Paper=
{{Paper
|id=Vol-2734/paper9
|storemode=property
|title=Data-Driven Approaches for Exploring the Effects of Teacher Instruction on Student Programming Behaviors
|pdfUrl=https://ceur-ws.org/Vol-2734/paper9.pdf
|volume=Vol-2734
|authors=Nicholas Lytle,Veronica Catete,Yihuan Dong,Mehak Maniktala,Danielle Boulden,Eric Wiebe,Tiffany Barnes
|dblpUrl=https://dblp.org/rec/conf/edm/LytleCDMBWB20
}}
==Data-Driven Approaches for Exploring the Effects of Teacher Instruction on Student Programming Behaviors==
https://ceur-ws.org/Vol-2734/paper9.pdf
Data-Driven Approaches for Exploring the Effects of
Teacher Instruction on Student Programming Behaviors
Nicholas Lytle Veronica Catete Yihuan Dong
NC State University NC State University NC State University
nalytle@ncsu.edu vmcatete@ncsu.edu ydong2@ncsu.edu
ABSTRACT as a teacher are programming is correct. Though obser-
As programming activities become integrated into K-12 class- vations provide teachers with information on how students
rooms, teachers with minimal prior programming experience are following along with the lesson, this methodology is not
will instruct students of varying backgrounds. This usually scalable and we seek more quantitative data-driven method-
takes the form of Live Coding, programming on a display ologies to investigate student programming behavior. To
in front of students while they follow along. While effective, understand the effects of instructional practices, we exam-
Live Coding prohibits teachers from seeing students current ine student and teacher coding artifacts, utilizing the logged
progress. Though researchers can observe instruction and interactions of the coding environment. We are interested
can tell teachers when students need the pace altered, this in ways to visualize how students follow along with teacher
help will not always be available. To improve and scale instruction. We perform exploratory analyses to identify
insights and investigate how teacher instruction influences patterns in student behavior concerning completion timing,
student programming behavior, we perform several analyses paths, and rates. Using visualizations, we present case stud-
to visualize the influence of instruction on student assign- ies of teachers instructional style grounded through class-
ment progression throughout the class period. We present room observations. We believe these data-driven approaches
case studies featuring teacher and student programming be- could lead to both scaleable offline and online analysis of
haviors. We end on possible means our methodology can be programming instruction, informing the development of in-
extended to create dashboard and other data-driven tools. telligent dashboards designed to aid both novice and expert
programming teachers during live coding instruction.
Keywords
block-based programming, instructor influence,Live Coding 2. BACKGROUND
2.1 Computing in STEM Education
1. INTRODUCTION If children get experience with programming, it is usually
1
To address inequities in access to computing education [17], within elective computing classes or outside of school activ-
teachers and researchers have begun partnerships to put ities [17]. To reach all students, computing must be inte-
computing activities in required courses. Teachers new to grated into required K-12 courses. In this age-range, curric-
programming exhibit levels of pedagogical discomfort with ula that use block-based programming languages are often
instructing programming assignments. Our prior work has employed [8]. Block-based programming languages such as
demonstrated that teachers new to computing curricula of- Snap [9] are increasingly popular and have been shown to
ten adopt instructional styles for lessons that are more aligned scaffold novices into learning programming in STEM con-
with their traditional teaching methods, which for middle texts better than traditional text-based languages [22]. While
school grades is often direct instruction[3]. Direct instruc- block-based environments can support novice students, teach-
tion is often done in the form of ’Live-Coding’, showing the ers must be trained in computing topics and integrated com-
process of developing the final code. This process leaves puting lessons must be developed for their classrooms. These
teachers without a sense of how the class is doing, whether are usually both accomplished during teacher professional
or not they are following along, or whether or not what they development seminars [12], though this might not be acces-
sible for all teachers. Another avenue like our work[3] is
1
Copyright ©2020 for this paper by its authors. Use per- design-based implementation research [6], where we collab-
mitted under Creative Commons License Attribution 4.0 In- orate with in-service teachers to iteratively build and refine
ternational (CC BY 4.0). a computing-integrated activity, actively focusing on pro-
fessional development and teacher comfort. These iterative
designs have made us choose to create lessons where teachers
instruct through live coding.
2.2 Instructing Programming in K-12
Teacher’s self-efficacy and discomfort are important factors
when trying to get core subject teachers to add comput-
ing into their classes. Our prior work has demonstrated
the success of lessons often hinge on a teachers comfort
with instructing, and often their self-efficacy is very low here: [11]). In programming data-mining, The sequence
as a novice programmer[15, 3]. Frykholms research backs of all interactions within the environment is called a trace,
this by suggesting a teachers self-efficacy filter in combi- which comprises a list of all states a student environment
nation with tolerance for discomfort will affect their will- is in and the interactions that connect them. An important
ingness to adopt new materials and curricula [7] such as subset of this trace is all interactions that result in differ-
computing. To mitigate discomfort, teachers tend to em- ing code (e.g. adding, deleting, and relabeling code), called
ploy more instructional-based practices such as focused or a code-trace and all the different code-states (i.e. unique
guided instruction that contrasts the constructionist and code) that a user progresses through on the way to complete
open-ended practices loudly promoted in computing [13]. a problem. This code trace is used for intelligent program-
However, at the middle school level, evidence from controlled ming environments as input for the creation of data-driven
studies demonstrate that direct, strong instructional guid- next-step hints [19] as well as other instructional support
ance proves more efficient in terms of long-term learning like worked examples [23]. While useful for data-driven al-
than constructivist-based, minimal guidance [13]. This is gorithms, the size of the space of all possible code-states is
further supported by Sweller’s cognitive load theory which in practice incredibly large even for a small programming
suggests free exploration of a highly complex learning envi- problem [24]. As such, methods have been developed to col-
ronment may generate a heavy working memory load that lapse the state-space of programming problems to something
is actually detrimental to learning [21]. These prior general more manageable for analysis. Prior work has represented
findings apply directly to programming instruction. A study the varying pathways of student attempts using a feature-
by Lin and Dalbey [14] shows medium and lower perform- state representation [24, 16]. In this manner, an assignment
ing students do better with explicit instruction which has no can be seen as a set of features, and a student’s current
negative impacts on high performing students. Husic further state in the assignment can be represented by which features
argues that beginners lack a repertoire of useful approaches the student has present or absent in their code. Rui et al
to thinking about and learning programming. To build this, [24] demonstrated techniques to visualize student pathways
teachers of introductory classes need to be specific when pro- through a programming assignment by representing which
viding integrated information, problem-solving support, and features students progressively added to to their code until
feedback [10]. Direct instruction reduces the working mem- reaching a final solution. Lytle et al extended this method-
ory load of the students and lets them concentrate on the ology to be useful in informing curricula design by looking
tasks at hand [14]. Thus, for introductory students, early at common pitfalls in student programming pathways [16].
in the learning curve, teachers should provide a supportive Additional effort has been placed recently in using student
scaffolding to help students carry out a task. Accordingly, feature completion information to help the instructor dur-
when teachers provide scaffolding, they generally carry out ing classroom implementation [5]. Dashboards have been
parts of the overall task that students cannot yet manage. developed that use programming trace or compilation in-
formation to aid instructors in finding students in need of
This research suggests K-12 teachers might benefit from help[5]. Diana et al’s dashboard system is able to use fea-
adopting the pedagogical technique of Live Coding. Live tures of code-trace data to display to an instructor which
Coding, often found in university programming courses, has students are lagging behind in the lesson (having fewer fea-
an instructor start from scratch (or starter code) and build tures complete) in order to aid teachers in selecting which
a program related to the material in front of the students[2]. students to help or group together to peer-help [5]. While
Unlike code snippits, displaying the construction of code useful, this requires that the dashboard be visible to the in-
gives students insight into the programming process [2] and structor at all times which may take away from the time
can usually be done in a manner that allows students to fol- spent live coding. We wish to extend these methodologies
low along creating an active learning context. Studies have in order to develop ways to provide post-hoc analysis of a
demonstrated the benefits of live coding for instructing in classroom implementation as well as in the future, provide
block-based programming languages [20]. Recent work has unintrusive immediate support to instructors live coding in
focused on intelligent dashboard support for Live Coding K-12 environments.
[4], though Chen et al. focus on making the process easier
rather than analyzing the dynamics between instructor and
students. It should be noted that nowhere in definitions 3. METHODS
for live coding does it specify that students follow along.
However, given the research described above, we feel that
3.1 Data Collection
The dataset used in this analysis came from several imple-
active, follow-along participation driven by instructors will
mentations of a 4-day computing-integrated science lesson.
provide the best platform for students to learn in these con-
Nine instructors across three schools led 19 middle-grade
texts. Additionally, those who have some prior experience
classes in a ‘Food Webs’ unit using a block-based program-
will be able to independently move on without the instruc-
ming environment, Cellular [1] (example code is shown in fig-
tor, while novices can follow along intently step by step.
ure 2). Cellular is a block-based language to scaffold novices
Given this instructional technique and the classrooms that
learning to code, and uses an agent-based environment to aid
have already participated in this style, we seek to use ad-
in the modelling of scientific phenomena. Approximately
ditional advances in programming trace data analytics to
480 students participated in the integrated activity, how-
understand classroom dynamics during live coding sessions.
ever, we are only analyzing trace log data from 287 consent-
ing students with complete participation over the four days.
2.3 Programming Trace Data and Analytics Classroom observations of each initiative were conducted by
There are dozens of systems that analyze and collect student researchers who sat in each classroom watching the teacher
programming data (an introductory review can be found instruct and aiding them as requested if an issue arose.
Table 1: Participants by instructor (* = researcher) culmination of their final abilities.
Instructor School Sections Usable N
Teacher A* I 1 10
3.3 Data Analysis
The primary data used for our analyses is environment trace
Teacher B I 1 9
data from the consenting students working within the as-
Teacher C* II 2 27
signment. The 19 classroom implementations of the assign-
Teacher D II 4 72
ment resulted in over 90,000 interactions from 287 unique
Teacher E II 3 61
students. To train the teachers in instructing, instructor
Teacher F* II 1 11
guides were developed. These gave the interactions with
Teacher G III 1 16
the environment (code additions) necessary to complete the
Teacher H II 5 62
assignments, included images of example final code, and a
Teacher J III 1 19
suggested order of how to add code in their environment
Total I-2, II-15, III-2 19 287
for instruction. Using these instructor guides, we decom-
pose the assignment into five unique, mutually exclusive
features. These features demonstrate the five additions that
A breakdown of participating instructors and schools is avail- teachers led students into creating in the Cellular Environ-
able in Table 1. Demographic data for each of the three ment. These features can be detected using an auto-grader
schools is provided in Table 2 below. Data for each of the type system and we can determine at any given point in a
science classes mirrors the overall school demographics. In- students programming trace whether or not that feature is
structors were all trained in the Food Webs course materials present or absent in the code. Figure 1 presents a short de-
before teaching their class. In schools I and II, Research scription of each feature alongside the corresponding code
instructors (Teacher A, C, and F) led the first class pe- blocks.
riod each morning. Teacher instructors led the remaining
class periods. There were a total of 6 researcher-led (or Non
In-Service instructors) class sessions and 13 teacher-led (In-
Service Instructors) class sessions.
3.2 Curriculum
The Food Webs unit is a 4-day activity designed to meet the
state-level education standards for 6th-grade science class-
rooms on the topic of Energy Transfer in a food web. On
the first day of the activity, students use pseudocode to com-
plete scientific worksheets on defining terminology and re-
lationships between organisms in a food chain. Students
work as a class to abstract the behaviors that an organism
in the food chain might exhibit (eat, grow, move, etc.). On
day two, teachers lead students in using their previous pseu-
docode to model the behavior of sunlight as an abiotic factor
in the food web and its effect on producers (plants) using Figure 1: Food Webs Day 4 student coding tasks.
the Cellular programming environment. Using a companion
worksheet, students manipulate energy values in the code to Feature 1 tasks students to create a local variable, Energy,
see its effects. On day three, teachers help students add in that will represent how much energy a Fox agent has in the
code for a primary consumer (bunny), again exploring the Food Web. The instructions explicitly state that the vari-
propagation of energy in the food chain. Finally, on day able must be local as it represents an individual fox’s cur-
four, teachers instruct students on how to create the basic rent energy. Furthermore it must be named “Energy” due
functionality for a secondary consumer (fox) as shown in to a case-sensitive dependency in custom blocks introduced
Figure 1. Students then set up their own scientific experi- later in the program. Feature 2 tasks students to make the
ment determining an independent and dependent variable, Fox move within a forever loop, as to simulate a Fox moving
as well as their hypotheses on what will happen. Students around in a biome. The 3rd feature has students decrement
are encouraged to run several trials of their experiments be- the Energy variable inside the forever loop to simulate the
fore declaring a definitive conclusion. Fox losing energy with each move. The 4th feature intro-
duces the first If block - If the energy of the Fox is low (but
The Food Webs curriculum was chosen as we have data from not 0), the Fox “eats a bunny” in the environment and gains
the largest set of unique teachers as well as the largest num- some energy. The final feature, Feature 5, is a second If
ber of student participants. We choose the 4th day rather block where if the Fox’s energy is less than 0, the Fox will
than the other two programming days for our analysis be- die. We use code feature detection to determine, for each
cause as part of our design-based implementation research, code state in a student’s code-trace, whether or not each
we implemented different versions of Days 2 and 3 for differ- feature is present or absent in a student’s code. We then
ent teachers to experiment with different instructional scaf- use this information to develop the following three analyses:
folding techniques. However, every teacher implemented the
same final activity. As Day 4 is also the final day of the ac-
tivity, teachers have had multiple class periods of experience 1. Feature-Time-Completion - A histogram that visu-
instructing programming assignments and Day 4 serves as a alizes when students complete a given feature relative
Table 2: Student demographics at each of the studied schools.
School # Students White Asian Black Hispanic Multiracial Female Low Income
School I 526 31.4% 6.7% 30.8% 24.3% 6.3% 48.3% 47.7%
School II 815 39.5% 5.9% 22.5% 25.8% 5.8% 48.1% 51.0%
School III 919 22.0% 2.6% 42.3% 28.1% 4.7% 52.2% 65.6%
to the feature completion time of the instructor. by Teacher B. Here, colored points are shown in order to
be able to track students across different features. It should
2. Cumulative-Feature-Completion - A stair-step graph be noted that outlier points are removed for scaling pur-
indicating when students add features to their code poses (5 total features are completed by 3 students past the
during the class relative to the instructor. 100 second marker). Teacher B completed 4 features in in-
3. Feature-Path-Graph - A state-space representation tended sequential order (F1,F2,F3,F4) which is important
outlining what features students add in what order, to note as this was not always the case (as we will see later).
showing the varying paths they take compared to the Most students complete the same feature within a minute
path taken by the instructor. of when Teacher B completes the feature (minus the visibile
and non-visible outliers).
4. RESULTS
4.1 Feature Time Completion 4.2 Cumulative Feature Completion
Our second representation shows how students cumulatively
Our first methodology, Feature Time Completion, produces
add features to their code relative to each other as well as
a histogram showing the timing in which students complete
relative to the instructor. Using the timestamps of when
a feature relative to when the instructor completes the fea-
students add their next feature to their code, we are able
ture. In the three graphs in Figure 2, we present exemplar
to chart out in a Stair-step fashion students progressing
cases of the “I do, We do, You do” instructional strategies
through the assignment, going from having nothing com-
introduced in the Gradual Release of Responsibility (GRR)
plete to having more and more of the 5 features added. We
model [18]. This is a common instructional strategy taught
are also able to see based off the final placement of each line,
to teachers, that progresses a teacher from modeling a step
how many features students ended the lesson with. In the 3
to students (I do) to completing a step together with stu-
cases and 4 classrooms described below, each student trace is
dents (We do) to finally allowing students to try on their
an individual stair-step function. Instructor feature comple-
own (You do), giving the correct answer after their attempts.
tion is denoted by the dashed vertical lines, and comparing
The histograms shown show the timing of when the students
when teachers and students complete features (in combina-
complete the feature relative to when the teacher completes
tion with the additional classroom observation data) allows
the feature (minute 0). In subfigure 2a we can see evidence
us to understand different instructional patterns and their
of focused instruction, or “I do”, where the teacher demon-
consequences.
strates procedures before students attempt to solve problems
on their own. The graph is right-skewed and close to the cen-
ter signifying students completing the task directly after the 4.2.1 Teacher G
instructor. The second strategy, guided collaborative learn- Figure 3 describes one teacher doing the Food Web Assign-
ing, or “We do”, is characterized by students working as a ments with their class. After the teacher completes their
class to complete the feature while being guided along by the first feature (with the majority of the class following very
instructor. An exemplar of this strategy is shown in graph closely, within 2 minutes of him completing it) 15 minute
‘2b’. This has students completing the feature very close without progress pasess. A strand of students make progress
(within 5 minutes) of the instructor centering around the independently of the teacher, but the majority are stuck
instructor’s completion time. This suggests an instructional at only completing one feature. From classroom observa-
strategy where students are following along and completing tions, we know that the majority of students had followed
the assignment with the instructor in a “we do” fashion. Fi- the instructor in naming their variable “energy” (though the
nally, the last graphs depict independent learning, or a “You instructional guide says to name it “Energy” as custom func-
do” strategy where students attempt to solve the problem tions expect that name and are case sensitive). A strand of
before reviewing as a group. Graph ‘2c’ shows the major- students are able to recognize the error from their student
ity of students completing the feature prior to the instruc- guides and are able to independently make progress. At
tor demoing, with a subset of students (around 25%) com- around 10:00, the teacher fixes the error, and announces it
pleting the feature after the instructor reviews the solution. to the class). The wide range of students jumping in progress
The median student (the red-dashed line) is either right, at varying different times reflects a change in instructional
near-center, or left-adjusted for the three cases respectively. style adopted by the teacher. As it was a simple naming fix
Classroom observations confirm that the Teacher scaffolded that affected every other Feature, fixing it resulted in mak-
students by providing students with the blocks they would ing progress on multiple features near simultaneously (seen
need to complete the feature (but not giving the full an- in the tall jumps in each stair step graph).
swer or how to combine them). Many students, given the
appropriate building blocks, were then able to assemble the 4.2.2 Teacher E: Period 4 vs 6
answer very quickly (10 minutes before the demonstration). We now present a case study of the same instructor teach-
We turn to Figure 4, which provides an alternative represen- ing two different periods. As part of the faded scaffolding
tation of the same information for one class period taught teaching approach, a research member (Teacher C) taught
(a) TeacherD - I do (b) TeacherA - We do (c) TeacherH - You do
Figure 2: Gradual Release of Responsibility as shown by student interaction data
do not get all 5 features in their code. We do acknowledge
two outliers, one who jumps 4 features (around 11:15) and
the other who more gradual progress, but still faster than
the group. Classroom observations confirm that that stu-
dent was copying code that the previous instructor had left
on the projector during the transition period.
Figure 3: A flat-line after Teacher G’s first task - students
caught in the same error the Teacher does while live coding.
the first class period (Period 2) where Teacher E observed.
Teacher E then taught Period 4. Teacher E requested that
Teacher C instruct the next period (Period 5) to observe Figure 4: Teacher E’s first attempt teaching the lesson.
once more how the lesson should go and then afterwards,
the teacher instructed Period 6. From Figure 4, we can see
the difficulties that Teacher E had with their first attempt
teaching. Teacher E completes all of her features for the
class after nearly 45 minutes of instruction with sometimes
nearly 8 minute gaps between feature additions. Many stu-
dents are intently following Teacher E adding a feature when
she does, but the number of bands that seem to be moving
independently before the teacher suggests that the students
were at varying paces not at the instructors. This is backed
by the classroom observations with researchers noting that
some students were ahead of the teacher and the teacher
actually asked students for help on what to add next in the
sequence. In the next attempt, Figure 5, Teacher E was able
to complete features much faster, adding 4 of the 5 neces-
sary features in an 8 minute span of time. This latter period
dramatically changes the pace at which the students add the
Figure 5: Teacher E’s second attempt teaching the lesson.
features, and the closeness of bands suggests that students
were more intently following the instructors pace. There are
common patterns that can be seen across both graphs. One 4.2.3 Teacher F
is the presence of outlier students who complete things at We now highlight a period taught by Teacher F, a research
a dramatically faster pace than the teachers. From the two instructor with a large amount of prior informal computer
graphs, it also should be noted that the teacher does not science teaching experience. Teacher F elected to lead the
demonstrate the last feature to the class (unlike in her pre- assignment without demonstrating to the students what the
vious teaching period), and many of her students therefore code should look like. Instead, she instructed students in
what features they should be adding next, facilitating, and
helping students by walking around the room. After each
feature, she would then recap verbally with the whole class
to review what they needed to have done in order to add the
feature before moving on to the next one. We can see that
student progress is very independent of one another, with
the timing of each successive feature addition (i.e. when
students go from having 1 to 2 features) varying widely.
While students add their first feature within 5 minutes of
each other, Students add their 2nd, 3rd, and 4th feature
within 10 minutes of each other. This reflects the choice of
instruction style, with Teacher F walking around the room
independently helping students. In contrast to the end of
Teacher G’s lecture, progress in Teacher F’s class is lim-
ited, usually, to only adding the next feature and nothing
else. This reflects the teaching style of only giving students
limited information about what the next tasks are, focus-
ing the instruction on one feature at a time. Teacher F’s
students that complete their fifth feature (adding death),
do it near simultaneously, reflecting the change in instruc-
tion style recorded where the teacher instructs the class in
completing the feature together.
Figure 7: Feature-Path Graphs for Students (and teachers)
within sample classes for Teachers C, B, and H.
progress. We also denote the beginning and end state with a
special circle state and star-shaped state respectively. This
is similar to the representation introduced in Rui et al [24].
The three feature graphs show different instructional strate-
gies at play. The first, TC, show most students following the
path of the teacher, adding features as they do. However,
it also includes many individual paths taken by students
not necessarily following the teachers instruction sequence
explicitly, but following the general direction, denoted by
Figure 6: Teacher F instructs without demoing steps. the number of ’confluence’ points or rejoining back into the
Teachers state of various different paths. This classroom had
the largest proportion of students actually finish the assign-
4.3 Feature-Path-Graph ment compared to the other two classrooms. Teacher B’s
In our final visualization, we remove the temporal informa- more narrow state-space with the largest states being those
tion of the last two graph-types and focus solely on Feature- that the teacher traversed suggest that most students were
paths, the order in which students add features to their code. following along with the instructors code additions (corrob-
In Figure 7, we highlight three Feature State representations orated by Figure 4 as well as observations), adding in the
of three classes taught by three different instructors (Teacher order that they did. While Teacher H had a subset of stu-
C, B, and H). Each state in the graph represents a unique set dents add features in the order they did as well as a subset
of features complete by a student. For example, following chosing a separate path, these paths seemed to converge in
the path of Teacher B (the states marked ’TB’) in the sec- state 2,3,4 then leading to the addition of Feature 5. While
ond graph, the states progress from the one with 0 features Teacher H and C both show variance in student feature
to the one having only feature 1, to having both feature 1 paths, it should be noted that both Teacher H and B did
and 2, to having features 1,2, and 3, and ending having fea- not add all 5 features to their code (Teacher H did not add
tures 1 through 4. Conversely, the state spaces for Teach- Feature 1, and Teacher B did not add feature 5). As such,
ers H and C show different sequences of feature additions all students in Teacher B’s classroom and all but one stu-
(for C in the order 2,1,3,4,5 and for H in the order 4,2,3,5). dent in Teacher H’s class failed to complete all features of
Common state transitions, (those where a large proportion the assignment highlighting scenarios in which intent direct
of students transitioned) are marked with the Feature that mimicry did not fully benefit students.
was added during the transition. The size of each state rep-
resents the proportion of students in the class that visited
that state. As some students do not finish the assignment 5. DISCUSSION
fully (getting “stuck” at certain points) we represent these 5.1 Feature Time Completion
as “stuck-states” in which a grayed proportion of the state As shown in Figure 2, while there are indeed examples of
represents how many students who visited that state did not “I do” sequences, where a teacher demonstrates a task and
then students replicate it, we also find examples of “We do” discomfort are suggested in Teacher E’s case study where
and “You do” sequences rounding out Pearson’s Gradual Re- there are long pauses between teacher actions and student
lease of Responsibility model. Our observations triangulate reactions. These graphs contrast Teacher F’s, the experi-
the “We do” pattern to actually involve the class working enced computing instructor, which had students clustered
together with the teacher to solve a problem, as opposed on the same features, moving at a steady interval. Teacher
to Pearson’s more traditional interpretation of the teacher F did not feel the need to explicitly direct students with
working with just a small group of students. We find “I live coding, but instead was able to guide their exploration
do” and “We do” strategies often adopted for the first fea- through facilitation. Conversely, we also discovered evidence
tures that students work on together in the assignment with of change in instructional practice, both within a class pe-
the instructor with “You do” strategies prominent in fea- riod and between them. In many cases, these changes lead
tures later in the assignment (the last features students work to more students following along and getting through the
on). This would corroborate teaching progression strategies material. As the anxieties that novice teachers tend to have
pushed for in the GRR model, transitioning students from focus on not doing things ’correctly’, showing evidence of
“I do” to “We do” to “You do” feature completion throughout positive change between implementations could potentially
the course of an assignment. inspire confidence in novice teachers that they are capable
of instructing computing curricula.
5.2 Cumulative Feature
By separating out the individual traces of each student into 5.5 Limitations
their own function, we can see how a student temporally The drawn conclusions that we outline in our case studies
progresses through the assignment in relation both to the are supported in part by our classroom observations, corrob-
teacher and the other students in the class. From the stu- orating what we find within the data-driven visualizations.
dent perspective, we see many examples of students who Because of the nuanced and varied ways in which classroom
work in lock-step with the instructor (or with each other), behavior can manifest within the data, it is important that
adding features as they do. But we also see students who we frame these results with our understanding of what hap-
trail and lag behind as well as students who rapidly add pened within the classroom. As such, we understand the
features before the instructor has even introduced them. reliance our methodology has on observational data, and
This differentiation is to be expected in a core classroom make no advocacy that our method acts as a complete re-
where students have varying access to computing outside of placement for it. In the same way that no single one of the
school and therefore different experiences with block-based three graphs tell the full set of information of a classroom,
programming environments. The ability to identify students we believe that these quantitative analysis methods act as
who are not following along with the instruction, either as an additional tool to act alongside qualitative data collec-
the pace is too fast or too slow, can aid instructors in identi- tion methods, ones that scale much better though than the
fying when to create more extension activities to high achiev- observational-based ones. We also understand that other
ers or when to give more support to students in need. limitations present themselves with aspects of our method-
ology such as only focusing on one lesson with only 5 possi-
5.3 Feature-Path ble features. In addition, only having the set of consenting
The state-space focuses on which features are being added students out of the entire classroom trace data could have
in what order, and can grant information about the way created a sampling bias that might alter any and all of our
teachers set up instruction (how they choose to pace the graphs and therefore our analysis. We intend to try and
students through the task). Very narrow state-space graphs offset these effects by replicating this with the multitude of
like shown for Teacher B can demonstrate a classroom that is lessons that are being developed by our teacher partners in
explicitly following along with the instructions as sequenced the upcoming academic year. Our methodology described is
by the teacher (even if they might be working on the tasks agnostic to the assignment, the features, and how they are
at different times). Conversely, wider state-spaces like those tested for (though we acknowledge that the feature state
shown for Teachers C and H can be more indicative of class- points are only as good as the tests you run on the code).
rooms where students had more agency over the order in
which they choose tasks. We make no argument that more 5.6 Future Work
explicit instruction, or a higher path “follow” rate leads to ei- Our initial results provide a foundation for further analy-
ther more or less students completing the assignment. Teacher sis into influence of one actor (e.g. teacher) on another (e.g.
B and Teacher H had a high proportion of students following student) during programming. For example, as more schools
along with their path, however, this might have left students begin to adopt pair programming models of instruction, it
unable to complete the assignment independently as neither would be worth investigating these collaborative interaction
teacher added the fifth and final feature. effects. This current work has practical applications for
our context in teacher training. As these graphs, charts,
5.4 Instruction and other figures can be generated rather quickly, teacher-
Looking through the teacher perspective, we can identify friendly versions can serve as additional means of explaining
patterns indicative of increased teacher discomfort. As men- to teachers how the experience went and how the classroom
tioned in Teacher G’s case study, as he erred in making the acted. This data can help teachers inform their practice
variable, he his following students all flat-lined in their cod- and provide immediate or post-hoc help. Further, addi-
ing progress. This highlighted the loss of the teacher as the tional processing can translate many of these graphs into
“keeper” of knowledge and exposed the vulnerability that animations that can give a ’live feel’ of how the instruction
they had to problem-solve along with the students, thus in- played out in real time. Setting times for feature comple-
curring pedagogical emotional discomfort. Similar signs of tion targets for an instructor can keep them on pace in tight
classroom periods, and windows for instructor-student fea- [11] P. Ihantola, A. Vihavainen, A. Ahadi, M. Butler,
ture completion can be used to determine if the class is fol- J. Börstler, S. H. Edwards, E. Isohanni, A. Korhonen,
lowing along intently. Often, the instructor participants in A. Petersen, K. Rivers, et al. Educational data mining
this study looked to researchers for a simple ’Thumbs Up and learning analytics in programming: Literature
or Thumbs Down’ measure to see if they were doing things review and case studies. In Proceedings of the 2015
correctly and kids were following along. If a simple notifi- ITiCSE on Working Group Reports. ACM, 2015.
cation such as the one described were implemented into a [12] R. Jocius, D. Joshi, Y. Dong, R. Robinson, V. Cateté,
dashboard system, we can imagine it benefiting instructors T. Barnes, J. Albert, A. Andrews, and N. Lytle. Code,
of K-12 programming. connect, create: The 3c professional development
model to support computational thinking infusion. In
6. ADDITIONAL AUTHORS Proceedings of the 51st ACM Technical Symposium on
Mehak Maniktala (mmanikt@ncsu.edu), Danielle Boulden Computer Science Education, SIGCSE 20. ACM, 2020.
(dboulde@ncsu.edu), Eric Wiebe (wiebe@ncsu.edu) and Tiffany [13] P. A. Kirschner, J. Sweller, and R. E. Clark. Why
Barnes (tmbarnes@ncsu.edu). This material was supported minimal guidance during instruction does not work:
by the National Science Foundation under grant 1837439. An analysis of the failure of constructivist, discovery,
Anything expressed in this material does not necessarily re- problem-based, experiential, and inquiry-based
flect the views of the NSF. teaching. Educational psychologist, 41(2):75–86, 2006.
[14] M. C. Linn and J. Dalbey. Cognitive consequences of
7. REFERENCES programming instruction: Instruction, access, and
[1] B. M. Aidan Lane and J. Mullins. Simulation with ability. Educational Psychologist, 20(4):191, 1985.
Cellular A Project Based Introduction to [15] N. Lytle, V. Cateté, D. Boulden, Y. Dong,
Programming. BlockBooks Series. Monash University, J. Houchins, A. Milliken, A. Isvik, D. Bounajim,
Melbourne, Australia, first edition, 2012. Online: E. Wiebe, and T. Barnes. Use, modify, create:
https://github.com/MonashAlexandria/snapapps. Comparing computational thinking lesson progressions
[2] J. Bennedsen and M. E. Caspersen. Revealing the for stem classes. In Proceedings of the 2019 ACM
programming process. In ACM SIGCSE Bulletin, Conference on Innovation and Technology in
volume 37, pages 186–190. ACM, 2005. Computer Science Education, pages 395–401. ACM,
[3] V. Cateté, N. Lytle, Y. Dong, D. Boulden, B. Akram, 2019.
J. Houchins, T. Barnes, E. Wiebe, J. Lester, B. Mott, [16] N. Lytle, V. Cateté, Y. Dong, D. Boulden, B. Akram,
et al. Infusing computational thinking into middle J. Houchins, T. Barnes, and E. Wiebe. Ceo: A
grade science classrooms: lessons learned. In triangulated evaluation of a modeling-based ct-infused
Proceedings of the 13th Workshop in Primary and cs activity for non-cs middle grade students. In
Secondary Computing Education, page 21. ACM, 2018. Proceedings of the ACM Conference on Global
[4] C. Chen and P. J. Guo. Improv: Teaching Computing Education, CompEd ’19. ACM, 2019.
programming at scale via live coding. In Proceedings [17] J. Margolis. Stuck in the shallow end: Education, race,
of the Sixth Annual ACM Conference on Learning at and computing. MIT Press, Cambridge, MA, 2010.
Scale, L@S ’19, New York, NY, USA, 2019. ACM. [18] P. D. Pearson and M. C. Gallagher. The instruction of
[5] N. Diana, M. Eagle, J. Stamper, S. Grover, reading comprehension. Contemporary educational
M. Bienkowski, and S. Basu. An instructor dashboard psychology, 8(3):317–344, 1983.
for real-time analytics in interactive programming [19] T. W. Price, Y. Dong, and D. Lipovac. isnap: towards
assignments. In Proceedings of the Seventh intelligent tutoring in novice programming
International Learning Analytics & Knowledge environments. In Proceedings of the 2017 ACM
Conference, pages 272–279. ACM, 2017. SIGCSE Technical Symposium on Computer Science
[6] B. J. Fishman, W. R. Penuel, A.-R. Allen, B. H. Education. ACM, 2017.
Cheng, and N. Sabelli. Design-based implementation [20] A. Ruthmann, J. M. Heines, G. R. Greher, P. Laidler,
research: An emerging model for transforming the and C. Saulters II. Teaching computational thinking
relationship of research and practice. National society through musical live coding in scratch. In Proceedings
for the study of education, 112(2):136–156, 2013. of the 41st ACM technical symposium on Computer
[7] J. Frykholm. Teachers’ tolerance for discomfort: science education, pages 351–355. ACM, 2010.
Implications for curricular reform in mathematics. [21] J. Sweller. Cognitive load during problem solving:
Journal of Curriculum and Supervision, 2004. Effects on learning. Cognitive science, 12(2), 1988.
[8] S. Grover, R. Pea, and S. Cooper. Factors influencing [22] D. Weintrop and U. Wilensky. Comparing block-based
computer science learning in middle school. In and text-based programming in high school computer
Proceedings of the 47th ACM technical symposium on science classrooms. ACM Transactions on Computing
computing science education. ACM, 2016. Education (TOCE), 18(1):3, 2017.
[9] B. Harvey, D. Garcia, J. Paley, and L. Segars. [23] R. Zhi, T. Price, S. Marwan, Y. Dong, N. Lytle, and
Snap!:(build your own blocks). In Proceedings of the T. Barnes. Toward data-driven example feedback for
43rd ACM technical symposium on Computer Science novice programming. International Educational Data
Education, pages 662–662. ACM, 2012. Mining Society, 2019.
[10] F. T. Husic, M. C. Linn, and K. D. Sloane. Adapting [24] R. Zhi, T. W. Price, N. Lytle, Y. Dong, and
instruction to the cognitive demands of learning to T. Barnes. Reducing the state space of programming
program. Journal of Educational Psychology, 1989. problems through data-driven feature detection.