-Systems for automatic assessment of programming tasks have become a popular choice in programming courses as several advantages of using automatic assessment in teaching and learning process have been observed. One of the most important is the immediate feedback students get. However, the quality of the feedback is essential to achieve good learning results. At the University of Ljubljana we use our proprietary system called Project Tomo as a teaching tool. One of the most important aspects of our system is the possibility to return a detailed feedback and to analyze the student's solution since every submission is stored on the server. Until now we have collected more than 110,000 attempts along with their history. We are currently in the process of analyzing them. Currently we are concentrating on how to use this data to further improve the quality of the feedback given to the student. Some of the observations and the preliminary results are presented in the paper.
Teaching a beginner level programming course can be quite
challenging. Since programming is a skill, it can be learnt
best by solving as many programming tasks as possible. As
Lee and Ko write in [
Teachers should provide the students with the feedback
beyond one that normal tools (compilers, interpreters and run
time environments) provide. But the feedback must be
immediate, otherwise the students can get stuck which slows down
their progress considerably. Keuning, Jeuring and Heeren [
But providing timely and instant feedback in overcrowded
classrooms is a tough task. This calls for a tool that can
automatically provide immediate feedback. In their paper
Keuninig et all. [
The paper is organized as follows. In Section II we describe our SAAP solution called Project Tomo and its properties. In Section III we present the results of the analysis of students’ submissions and the conclusion in Section IV.
After evaluating several SAAP (a systematic review can
be found in [
So we developed a new web service for automatic
assessment called Project Tomo 2 [
solution, aiming at providing assistance for lab exercises where
students are required to solve numerous programming tasks.
We aimed towards the methods and tools that may help in
providing the necessary feedback to improve the support for
students during the learning of programming. So, our target
was the formative feedback [
The main design objectives in developing our SAAP service were: • Local execution, • Possibility to use any of the existing programming environments, • Being flexible enough to be functional with any programming language (currently we support Python, R and Octave) and • Providing as much flexibility as possible in administering tests to achieve giving the appropriate feedback.
The details for those decision are explained in [
The service works as follows: the students first download the files containing problem descriptions to their own computers. The files are opened in their preferred programming environment for the chosen programming language and the students start coding the solutions. Executing the files checks their solutions locally. If the server is available, the solutions are automatically stored on the server. The server also optionally checks for the validity of the solutions by comparing hashed output from students’ solution to the hashed output of the official solution.
This approach has several benefits: the service provides instant insight into the obtained knowledge to both student and teacher, all without disturbing the teaching process. There is also no need for powerful servers since all executions are done on the students’ computers.
As much flexibility as possible in administering tests was another vital feature. For instance, one of the requirements was that there should be a possibility to administer tests that can check whether a specific method was (or was not) used in a student’s submission. For example, if the student’s ability to write recursive programs is to be tested, non-recursive methods should not be accepted, even if they give expected results.
After providing the instructions for the task the teachers enter the expected solution followed by the tests that it has to pass (see Fig. 1). The solution is separated from the tests with the Check.part() command. It should be noted that the officially provided solution needs to pass the same tests as the students’ solutions. At first this approach seems slightly more demanding for the teachers compared to the traditional approach where the teachers only provide instructions in text form. However, this forces the teacher to test the quality of the instructions. Namely, it often happens that poorly formulated # ======================================================= # Computing distances # =========================================@000003======= # # Write a function dist(x1, y1, x2, y2) that returns the # distance between points (x1, y1) and (x2, y2). # # >>> dist(1, 2, 3, 4) # 2.82842712475 # ======================================================= def dist(x1, y1, x2, y2): ’’’Returns distance between two points’’’ return ((x1 - x2) ** 2 + (y1 - y2) ** 2) ** 0.5 Check.part() Check.equal(’dist(1, 2, 3, 4)’, 2 ** 1.5) Check.equal(’dist(0, 0, 3, 4)’, 5) problems only prove as being such during the attempt to solve them. This approach also ensures that the official solutions exist and work properly.
There are two commands that are most often used in testing. The most simple one tests the equality of the expected result with the result acquired by evaluating the given expression (see Fig. 1).
However, Tomo’s main strength is in the possibility that the teacher composes a test that goes beyond a direct comparison of the outputs. The tests have access to the source code of the submitted solution under Check.current_part[’solution’]. So it is simple to make tests that ensure that a solution did not use for or while loops (e.g. if the students are to write solutions in recursive style) - see Fig. 2.
Commands Check.out_file, Check.equal, and Check.run return True or False. Therefore, they can be used to determine if additional tests should be run or not. For instance, if the first test fails, the submission is clearly not valid and additional tests are not necessary. However, if the goal is to provide the students with detailed information on which test data their programs fail, as many tests as desired can be run.
There are other commands available to test the programs, for example commands that work with files. Since the validation is essentially a program written in a chosen programming language (Python for example) using the capabilities of the Check class, it can be made more advanced using all programming constructs that language offers. Thus, there are numerous possibilities available to prepare the appropriate feedback.
The basic feedback is the report on the success of the student’s attempt. This feedback is issued as soon as the students runs their solutions. Figure 3 shows that the solution of the first part of the task is accepted, the second one failed at (at least) one test and there has been no attempt to solve the third part of the task yet. 1. part has a valid solution. 2. part does not have a valid solution.
- Expression vsote([
We paid special attention to suitable wording of this basic feedback. One of the first versions of the systems declared “solution is correct”. But this is not in accordance with the premise that passing all the tests is not yet the proof of correctness. Therefore, we changed that into “solution is accepted”. Several teachers reported this change had positive influence on student’s awareness on what is the ’right solution’.
As explained before, the basic test is done with
Check.equal method. This method directly compares the
output from the official solution and the one (in [
And of course—most importantly—the teacher’s task is to react to events that may occur during exercises. And that is precisely the basic reason why we are developing the Project Tomo: to relieve the teachers of simple tasks and give them additional time to interact with the student during lab exercises.
In Project Tomo it is possible to give a student feedback not only when a given test fails but anytime during the test program. This can be used to notify the students that the solution has passed some test cases and that they are on a good track. This is achieved using the Check.feedback construct which prints out the string given. See example in Fig. 4. if Check.equal("""start(’miha’, ’meta’)""", 1):
Check.feedback("Bravo! Strings ’miha’ and" + " ’meta’ match in the first character")
A good way to learn is also to observe the official solution (see Fig. 5). In Project Tomo the teacher can decide when the students see it for every task. Currently the possibilities for official solution visibility are “always”, “never” and “after they have submitted a valid submission”.
The first option is rarely used since it gives the students an easy possibility to ’cheat’: if their solution is not accepted, they look at the official one and use obtained information to solver the task. The second one is used during exams, where the official solutions are made visible after the end of the exam and the third one is the default setting.
Till now, we have developed quite an extensive library of programming tasks with high-quality feedback. Despite that we are constantly adding new tests and feedbacks to the tasks. Ideas for additional tasks arise from observing the mistakes that students make while programming. Many of the mistakes are missed by the teacher since it is impossible to observe each and every student all of the time. Since Project Tomo stores the history of every submission, we can do that retroactively: checking the history of the students’ submissions we can analyze them, extract typical mistakes and use that knowledge to improve the quality of the feedback even further. So our workflow is as follows.
First the task is created by the teacher. The teacher tries to predict typical mistakes the students will make and include them into the test cases. The task is used in a teaching course and a lot of submissions are acquired from the students. Then these submissions are analyzed and if need for additional test cases is seen, they are added. Then the updated task is used in a teaching course and the entire process is repeated. So the quality of the feedback (and test cases) is checked and improved continuously.
Currently we are just starting a thorough analysis of the submissions. The goal of the first step of the analysis is to detect problematic tasks. Our assumption is that the exercises where the average number of unsuccessful attempts students made before the accepted one, is high, are prime candidates for being labeled as problematic. If we manage to improve the quality of the feedback for these tasks, the students would benefit the most.
We concentrated on the last year programming course, where all data is available. For each successful attempt we checked how many unsuccessful attempts had been made before the valid one. Using the above mentioned criteria we detected several tasks where the average number of unsuccessful submissions was higher than 10. We analyzed the source code of those unsuccessful attempts.
Since the number of attempts is quite large, we have only managed to analyze some of the tasks so far. One task that was particularly interesting was a simple one, where students had to print the amount of money on an bank account in a grammatically correct way. The average number of unsuccessful submissions for this task was higher than 15. When we analyzed the history of the 495 attempts made, we found out that the students had made two typical mistakes. The first one was that they were unaware of the grammar rules of their own mother tongue and the second one was that they were very careless with their output, which usually only slightly deviated from the official one. Both of them combined caused the students to submit a lot of attempts that only slightly deviated in the output from the official solution, but were rejected by Project Tomo. It looks like students did not manage to see the difference in the outputs since they submitted lots of solutions with seemingly random changes to the source code that did not really fix the problem.
Using the above data we added two additional feedbacks to the task. The first one informs the student of the necessary grammar rules in detail if the test detects that they are not respected. This will hopefully reduce the number of incorrect attempts since a student can fix all grammatical mistakes in one step. The second one deals with the sloppy outputs. On one hand it is good that students learn how to be accurate. On the other hand it can be very frustrating if one has been working on a task for hours without visible progress, even more so for beginners. So we decided to modify the comparison function between the expected and the given output so that it will show more clearly where the outputs differ. We hope this will reduce the number of unsuccessful attempts even further and teach the students how to be precise at the same time.
We plan to analyze more tasks in the similar manner and use the updated tasks during future programming courses. We hope to notice a reduction of unsuccessful attempts.
Our goal is to use our analysis results to improve the feedback for the most problematic tasks and use the improved exercises in the class next year.
Currently most of the analysis is done manually, which is a very slow process. In the future we plan to use machine learning algorithms to extract common patterns from unsuccessful submissions for the given task. This would save our time trying to analyze the history of all attempts and allow us to focus on the most common mistake patterns.
Also some aditional features in providing feedback are
planed. We are currently looking into the possibility of adding
an additional option when the official solution can be seen:
make it visible after a specified number of unsuccessful
attempts. This would allow the students to see the official
solution after they had made some real effort towards solving
the task but failed to provide a valid solution. However, we
have to ensure way of verifying that those attempts are “real”,
not merely faking some output in order to reach the required
number of submissions. Here we will probably use some of
the approaches suggested in literature, for example in [