Next-step programming hints have attracted considerable research attention in recent years, with many new techniques being developed for a variety of contexts. However, evaluating next-step hints is still a challenge. We performed a pilot study in which teachers (N = 7) rated automatic nextstep hints, both quantitatively and qualitatively, providing reasons for their ratings. Additionally, we asked teachers to write a free-form hint themselves. We found that teachers tended to prefer higher level hints over syntax-based hints, and that the di erences between hint techniques were often less important to teachers than the format of the generated hints. Based on these results, we propose modi cations to next-step hint strategies to increase their similarity to human teacher feedback, and suggest this as a potential avenue for improving their e ectiveness.
Joint Proceedings of the Workshops of the 14th International Conference on
Educational Data Mining (EDM 2021); Copyright ©2021 for this paper by
its authors. Use permitted under Creative Commons License Attribution 4.0
International (CC BY 4.0)
utilize historical student data and, as such, can be fully
automated [
Our main contribution in this paper is to combine quantitative ratings with qualitative explanations. In other words, we do not only investigate di erences in teacher ratings, but also why teachers preferred some hints over others. To this end, we performed a survey with N = 7 teachers, asking them to grade next-step hints generated by three di erent methods across three programming tasks in Python. Our overarching research questions are: RQ1 Do teachers' ratings di er between hint methods? RQ2 Do automatic hints align with teacher hints? RQ3 What are teachers' reasons for preferring some hints over others? This paper is set out as follows: Section 2 discusses related work in more detail, Section 3 describes the setup of our study, Section 4 describes the results and, nally, Sections 5-6 discuss and summarize the implications of our work.
Prior work on evaluating next-step hints broadly falls into
three categories: technical criteria, outcomes for students,
and expert opinions [
Technical criteria are mostly concerned with the
availability of hints and motivated by the cold start problem, i.e.
the problem that data-driven hint generation requires a
certain set of data to become possible [
In our work, we combine aspects of this prior work with
qualitative questions. In particular, we use a variation of the
three scales of Price et al. [
In this section, we cover the setup for our survey, beginning with the programming data sets we used, followed by the mechanism to select speci c examples, the hint methods, and the recruitment for the survey itself.
In order to provide realistic stimulus material, we selected our programs from three real-world, large-scale data sets of program traces in introductory programming. Namely, we considered data from the 2018 (beginner challenge) and 2019 (beginner and intermediate challenges) National Computer Science School (NCSS)1, an introductory computer science curriculum for (mostly Australian) school children in grades 5-10. 12; 876 students were enrolled in the beginners 2018 challenge, 11; 181 students in the beginners 2019 challenge, and 7; 854 students in the intermediate 2019 challenge. Each challenge consisted of about twenty- ve programming tasks in ascending di culty, each of which were annotated with unit tests. In all cases, we only considered submissions, i.e. 1https://ncss.edu.au programs that students deliberately submitted for evaluation against unit tests.
Our goal in this study was to evaluate the quality of
automatic hints in a range of realistic situations where students
were likely to need help and where feedback generation was
non-trivial. For the purpose of this study, we considered a
program as indicative of help-need if at least ve students
who submitted this program failed the same or more unit
tests in the next step of their development. This is in line
with prior work of [
In the end, we selected for each of the three challenges the program which maximized the tree edit distance to frequent programs and indicated help-need. The resulting submissions are shown in Figure 1, alongside with a description of the respective programming task and an example solution.
We considered three techniques to produce next-step hints.
Firstly, we used one-nearest neighbor (1NN) prediction [
Distance was measured according to the tree edit distance,
as used e.g. by [
Secondly, we used the continuous hint factory (CHF) [
Finally, we used the ast2vec neural network [
In all cases, the hint was formatted as a program which the recipe task You're opening a boutique pie shop. You have lots of crazy pie ideas, but you need to keep them secret! Write a program that asks for a pie idea, and encodes it as the numeric code for each letter, using the ord function. Print the code for each letter on a new line.
recipe submission
1 msg = input ( 'Pie idea : ')
2 code = ord ( msg [0])
3 code1 = ord ( msg [
recipe solution 1 msg = input ( 'Pie idea : ') 2 for a in msg : 3 print ( ord (a )) anagram task Let's make a computer program that only knows how to say 'hi': It doesn't matter what you type in, it should still print 'Hi, I am a computer'! To make it a bit more exciting, though, we'll add an Easter Egg. The word 'anagram' should trigger a secret message: Hi, I am a computer! What are you? anagram Nag a ram! Hi, I am a computer!
anagram submission 1 print ( 'Hi , I am a computer ! ') 2 raining = input ( ' What are you ? ') 3 if raining == 'a dog ': 4 print ( 'Hi , I am a computer ! ') 5 if raining == ' anagram ': 6 print ( 'Nag a ram ! ') 7 print ( 'Hi , I am a computer ! ') 8 if raining == 'a person ': 9 print ( 'Hi , I am a computer ! ')
anagram solution
1 print ( 'Hi , I am a computer ! ')
2 word = input ( ' What are you ? ')
3 if word == ' anagram ':
4 print ( 'Nag a ram ! ')
5 print ( 'Hi , I am a computer !)
scoville task scoville submission
The Scoville scale measures the spiciness of 1 jalapeno = int ( input (" What is the SHU value ? " ))
chilli peppers or other spicy foods in Scoville 2 if jalapeno <= 10000:
heat units (SHU). For example, a jalapen~o has 3 print ( 'Nam will enjoy this ! ')
a range between 1,000 to 10,000, and a ha- 4 else :
banero is between 100,000 and 350,000! Di er- 5 print ( ' This food is too spicy for everyone ! ')
ent people have di erent tolerances to eating 6
chilli peppers. Nam's parents cook with a lot 87 isfugasruygar=y in<t=( 1i2n0p:ut (" What is the SHU value ? " ))
of chilli, and so she enjoys eating foods with 9 print ( 'Nam will enjoy this ! ')
a SHU value less than 10000. Michael likes it 10 print ( ' Michael will enjoy this ! ')
less spicy, and only enjoys eating foods with 11 else :
a SHU value less than or equal to 120. Write 12 print ( ' This food is too spicy for everyone ! ')
a program to read in the SHU value for some
food, and print out who will enjoy the food. scoville solution
For example:
What is the SHU value? 5000
Nam will enjoy this!
and another example:
What is the SHU value? 120
Michael will enjoy this!
Nam will enjoy this!
If neither Michael nor Nam will enjoy the food,
your program should output: This food is too
spicy for everyone :(
1 shu = int ( input (" What is the SHU value ? " ))
2
3 if shu < 10000:
4 print ( 'Nam will enjoy this ! ')
5 if shu < 120:
6 print ( ' Michael will enjoy this ! ')
7 if shu >= 10000:
8 print ( ' This food is too spicy for everyone :( ')
1 a = input ( 'Pie idea : ')
1NN 2 b = [ ord (x) for x in a]
3 print (*b , sep ="\n")
1 x = x( '<string > ')
2 x = x(x [0])
3 x = x(x [0])
CHF 4 x = x(x [0])
5 x = x(x [0])
6 for x in x
7 x(x , keyword = '<string > ')
1 msg = input ( 'Pie idea : ')
2 code = ord ( msg [0])
3 code1 = ord ( msg [
We used a random sample of 30 student traces from the same task as training data to simulate a `cold start' with only a classroom-sized training data set.
The resulting hints of all three methods for all three tasks are shown in Figure 2.
For our study, we recruited N = 7 teachers from programming courses in Australia and Germany. Recruitment was performed via e-mail lists with a survey link. Teachers could then voluntarily and anonymously complete the survey in Microsoft forms. Participants were rst asked about their experience as programming teachers. Six participants responded that they had taught more than three courses, and one participant that they had taught between one and three courses. Participants with no experience were excluded from the study.
We acknowledge that our recruitment strategy has limitations: While we can be reasonably certain that only experienced programming teachers took part, we have no information about the speci c courses they taught and whether that matches up with the kind of programming task we investigated in our study. Further, self-selection bias may have occurred as we did not employ a random recruitment strategy.
Next, we presented the rst programming task (the recipe
task) with the o cial task description from the National
Computer Science School, the example solution, and the
student's submission (refer to Figure 1, top left). We asked the
teachers whether they thought the student needed help in
this situation (on a four point scale), why (as a free text
eld), and what edit they would recommend to guide the
student (free text eld). Further, we presented the three
automatic hints in Figure 2 (top left) and asked the teacher
how relevant each hint was, how useful it was, and how
much the student could learn from it, all on a ve-point
scale. We de ned `relevant' as `addressing the core problem
of the student's program' and `useful' as `getting closer to
a correct solution'. These scales correspond to the scales of
relevance and progress proposed by [
Finally, we asked the teachers whether they would prefer not to give any of the hints and why (free text). We repeated all questions for the anagram and scoville tasks. To avoid ordering bias, the order of hint methods was shu ed randomly for each participant.
We present our results beginning with the teacher's assessment of whether hints were needed at all, followed by the hints given by teachers, and we conclude with the assessment of teachers for the automatic hints.
For each of the programs in Figure 1, we rst asked the teachers how much help a student in this situation would 0 1 2 help need 3 need on a four-point scale, ranging from 0 (\The student is on the way to a correct solution and does not need help.") to 3 (\The student seems to have crucial misconceptions and should start from scratch.").
Figure 3 shows the distribution of responses for each task. Most teachers agreed on response 1 for all tasks (\The student is on the way to a correct solution but could bene t from a hint."). This indicates that our automatic selection indeed identi ed examples which indicated help-need. We also asked teachers why they believed that the student did or did not need help. In response to this question, most teachers appeared to analyze which high-level concepts the student had already understood - judging from their program - and which concepts were still missing or were misunderstood. For example, one teacher responded for the recipe task: \They know how to input, they know how to do the ord, they know they need to move through the string, they have just forgotten that there is a 'short cut' to do this in a loop.", and another teacher responded for the anagram task: \The student is not seeing the general rule of the program and is trying to cover possible cases by hand." Further, multiple teachers responded to this question with suggestions on how to provide further guidance and help to the student, such as \It just hasn't clicked that there are other possible inputs that they need to account for. Keeping this fresh in their mind should quickly lead to a solution." or \I think that they should mess around a bit more, but they should get a hint that the input function should only be used once for this problem.".
Next, we asked teachers how they would recommend editing the student's program next. Interestingly, teachers generally did not give hints on a syntactic level. In fact, some of them stated explicitly that they thought this was not helpful (e.g. \I would not give students exact syntax because then they blindly follow without understanding"). However, in some cases, they did suggest lines to delete. We found that teacher feedback tended to t into four general categories: A suggesting a missing concept, such as a for-loop, an else statement, or a combination of if-statements. For example, \When you have a line of code that you are task recipe anagram scoville repeating it can be useful to use a loop like `for' or `while'. This also allows you to repeat the code within the body of the loop for a exible number of times." B explaining or hinting at situations when the program will not work as expected. E.g. \You're almost there. However, it will only do the right thing when the user writes `a dog', `anagram' or `a person'. You can improve it so that it says `Hi, I am a computer!' every time, no matter what the user says." C suggesting the student solve a simpler problem rst. E.g. \Suggest that they delete all but the rst line and try and print out each letter one at a time." D suggesting that the student has something unnecessary in their program, or directly telling them to delete it. E.g. \Remove the two irrelevant if statements leaving only the correct `easter egg' statement"
Our third set of questions for each task concerned the rating of automatic hints (refer to Figure 2) according to relevance, usefulness, and learning, each on a ve-point scale from -2 (\not at all") to +2 (\very"). Table 2 shows the average ratings ( standard deviation) given by the teachers. We observe that 1NN is generally regarded as relevant and useful, which can be explained by the fact that it always recipe anagram scoville recommended a correct solution for the tasks in our examples (refer to Figure 2 and 1). However, teachers believed that students would not learn particularly much from these hints (rating around zero). For CHF, we observe strongly negative ratings in all three criteria for the recipe and anagram task, where CHF did not provide function and variable names (refer to Figure 2), but positive scores for the scoville task where it selected a correct solution. On that task, it received even higher scores than 1NN. Ast2vec received negative scores on the recipe task, and scores around zero for all criteria on all other tasks.
Finally, we asked teachers if there were any hints they would prefer not to give, and why. Figure 4 shows how often each method was named for each task (where lower is better). Regarding the reasoning, one teacher always responded that \I would not give students exact syntax because then they blindly follow without understanding.", which excludes all automatic hints. Further, 1NN was often named because it \shows a valid solution. Students may copy the hint code and then use it - without understanding what it does.". The hints provided by CHF for the rst two tasks did not include variable and function names (refer to Figure 2), which lead teachers to exclude it because it \does not look syntactically valid, and is incomprehensible.". Ast2vec was named least often, albeit by a narrow margin. Reasons for naming it were that it is \not helpful for developing student understanding", would require additional explanation, or even \harm the learning of the student."
In this section, we interpret the results in light of our three research questions: Do ratings di er between methods? Do automatic hints align with teacher hints? And what are teachers' reasons for preferring some hints over others? RQ1: Do teacher’s ratings differ between hint methods?. We do observe systematic di erences between hint methods in terms of ratings. However, these di erences appear to be driven less by the underlying algorithm, but rather by two factors: a) whether a correct solution was selected or a partial solution, and b) whether the hint was presented as a program with function and variable names or not. teachers only gave high scores for usefulness and relevance if a correct solution was given and gave very low scores if function and variable names were missing. They Maybe you could try a for-loop.
What happens when the user types `apple' ? Try doing this task on for-loops rst Can you think of a way to reduce the number of print statements? What happens when the user types `cat' ? Can you think of a way to use fewer if statements? What happens when the user types 120? How can you reduce the number of input calls? (and/or) Try using only one variable. never gave high scores for learning.
RQ2: Do automatic hints align with teacher hints?. We observe that teacher hints do not directly align with automatic hints because teachers generally suggested hints which were higher-level, like adding missing concepts or deleting lines to get to a more compact program. More generally, we identi ed four categories (refer to Section 4.2) which cover the kinds of edits teachers would have given themselves. To align teacher hints to automatic hints, we could employ automatic heuristics which post-process next-step hints, such as: A - Missing Concepts Compare the nodes of the student's current AST to that of the next step, then suggest concepts corresponding to missing nodes.
B - Mishandled Situations Apply test cases to the current and the predicted program, then suggest the student focus on inputs that work for the next step but not the current step C - Simpler Problem Similarly to A, rst identify missing concepts in the student's work, then suggest easier programming tasks that contain this concept (e.g. from earlier in the course).
D - Deletions Compare the current step to the next step and, if lines are deleted, ask the student if these lines are necessary.
Applying these strategies, the automated hints for the tasks in Figure 2 might then become the hints shown in Table 3, which align better with the hints given by teachers. We note in passing that abstraction may also make it easier to provide helpful hints because the hint method does not need to get every detail (such as function or variable names) right, merely the rough direction. For example, we notice that ast2vec uses the wrong string in line 5 of its hint and the wrong comparison constant in line 4 of Figure 2 (bottom). This would not be an issue in the abstracted hint. Still, we acknowledge that this approach has limitations: For strategy A and C, we implicitly assume that 'concepts' coincide with syntactic building blocks, e.g.: `have you tried adding a for loop?'. This assumption likely breaks down in more advanced programming classes.
RQ3: What are teachers’ reasons for preferring some hints over others?. Teachers mostly explained hints by concepts that were still missing in a program (like loops), undesired functional behavior for additional cases, or superuous code. These reasons align with the hints the teachers gave. Importantly, many teachers also emphasized that the student had already gotten many things right, indicating that teachers were motivated to preserve the progress that had already been made. The main reason for not providing a hint appeared to be that the teachers were concerned that the hint may not lead to any learning, either because the hint was syntactic, and hence not abstract enough, because the hint was a correct solution, or because the hint was `incomprehensible' due to missing function or variable names. Overall, the reasons provided by teachers underline our nding that teachers do not only care about the content of a next-step hint but also { perhaps mainly { how it is communicated, i.e. with a high-level explanation instead of a syntactic edit and without revealing the solution.
We performed a survey with N = 7 teachers to evaluate hints from three hint methods (1NN, CHF, and ast2vec) to investigate three research questions: Do quantitative ratings di er between methods? Do automatic hints align with teacher hints? And what are teachers' reasons for preferring some hints over others? We found that teachers generally had a low opinion of syntactic next-step hints, irrespective of the method. Di erences in ratings could be explained by two factors: Whether the hint was a correct solution (then it was regarded relevant and useful) or not (then all ratings were around zero) and whether the hint used human-readable variable and function names or not (then the ratings became strongly negative). Instead of syntactic hints, teachers preferred higher-level hints which suggested a missing concept, pointed out inputs which were not appropriately covered by the current program, suggested a simpler problem rst, or proposed to remove super uous lines.
Finally, the main concern of teachers for not giving hints was
students' learning. They disregarded both syntactic hints
as well as showing a correct solution because they were
concerned that students might navely apply the hint without
re ecting on it su ciently. This points to a potential gap in
current next-step hint approaches, which are mainly focused
on suggesting changes, but less on inviting re ection or
abstracting to a higher level. For introductory programming
courses, it may be su cient to just post-process syntax-level
hints with simple heuristics { such as the ones proposed in
the previous section. Additionally, one can introduce textual
explanations as suggested by [