In this paper, we describe our lessons learned during the introduction of automatically assessed programming exercises to a Bachelor's level course on algorithms and data structures in the Winter semester 2019/2020, which is yearly taken by around 300 students. The course used to mostly focus on theoretical and formal aspects of selected algorithms and data structures. While still maintaining the primary focus of a theoretical computer science course, we introduce a secondary objective of enhancing programming competence by giving practical programming exercises based on select topics from the course. With these assignments, the students should improve their understanding of the theoretical aspects as well as their programming skills. The programming assignments were given in regular intervals during lecture period with a thematic alignment between assignments and lectures. To compensate for the new set of tasks, the workload of assignments on theoretical aspect was reduced. We describe the di erent experiences and lessons learned through the introduction and conduction of these exercises. A user study with 44 participants shows that the introduction was perceived well by the students, although improvements are still possible, especially in the area of feedback to the students.
One of the key competences a student of computer science should possess at the end of his or her study should be the competence to write computer programs.
To support students in learning this important skills, many tools for
automatically assessed programming exercises have been developed over the last years
[
There are many publications about the details of di erent tools for automatic
assessment of programming tasks (e.g. see the reviews [
In comparison, literature of the actual experience of introducing these tools
into regular classes seems to be relatively sparse. However, there are publications
describing the experiences of introducing automatically assessed (programming)
tasks in regard to exercise design [
In our work, we use the CodeRunner tool for automatic assessment. The tool
was developed by Lobb and Harlow [
Currently, e-learning at Universitat Hamburg is mainly used for the distribution
of les (like lecture notes or exercise sheets) and for communication, to the
best knowledge of the authors. There are only few cases where the potential of
blended learning [
The goal of the Bachelor's level course Algorithmen und Datenstrukturen (algorithms and data structures) is to teach the students the principles of e cient algorithms, both in theoretical and practical terms. Each year, around 300 students participate in the course. Prior to the introduction of the programming exercises, the main focus of the module was set on the theoretical and formal aspects of selected algorithms and data structures. Because programming skills were mainly taught in di erent modules, the practical aspects (sample applications, programming tasks) were not discussed. With this development, our goal is to blur the distinction of theoretical and practical courses, thereby allowing students to implement theoretical concepts from scratch.
In the course, the available e-learning platform Moodle4 was previously used
for sharing documents and for communication through a forum. In addition,
students were able to check the progress on their course achievements, these,
however, had to be input manually by the instructors. The introduction of online
tests in the form of automatically assessed programming exercises is a novelty
for the course. The implementation of these new exercises was done using the
CodeRunner plugin [
By developing these programming exercises, we wanted to allow the students to deepen their understanding of the algorithms and data structures discussed in the lecture. This was done by letting the students implement di erent algorithms and sometimes let them use the algorithms to solve di erent tasks. One welcomed side e ect was to improve the programming skills of the students through these exercises. To compensate for the additional work caused by the new exercises, the workload of assignments in the area of theory had to be reduced.
All exercises were created based on the topics of the lecture. We created the di erent tasks by rst de ning the requirements of the tasks. Based on these, we chose suitable algorithms and data structures for the programming tasks. Those were transformed into the actual task, the test cases and an example solution. The same procedure was used to create example programs for the actual lecture itself.
It was required for the students to pass the exercises in order to complete the course. As such, the students were externally motivated to complete the programming exercises. One example of such a task from the point of view of the students (including short explanations of all important user interface elements) can be seen in Fig. 1. In total, 10 tasks were created, which were combined into 6 blocks. The students could choose whether they wanted to use Java or Python, for each block the better result was counted. For each block, the students were given two weeks to complete the tasks. For each task, the students had 10 tries to develop a correct solution that passes all test cases, however, they could also test their solution on a smaller set of pre-test cases. The test cases were composed of corner cases (e.g. empty input, maximum input value), normal cases and random tests. The random tests prevented the students from hard-coding the test results 4 https://moodle.org/ into their programs. All test cases were restricted in execution time and memory usage, however, the provided limitations were more than enough to pass all test cases even with ine cient solutions. Feedback to the students was only send through the result of the test cases, since manual feedback would have put a lot of additional work on the instructors and thus was not feasible. In addition, a sample solution was provided for each task. The average length of the provided sample solution, including source code comments, amounted to 24.5 and 19.7 lines of code for Java and Python, respectively, with both peaking at 41. All tasks were perceived as easy by all instructors.
To facilitate communication with the students, multiple ways of communication were o ered, both for announcements as well as for questions. These include a mailing list, Moodle-based communication (a forum as well as announcements) and special tutorials, which could be joined by students at will.
While the technical setup did not pose notable issues and overall the students were able to use the system and achieve their learning goals, we encountered some issues. These are described below. 5.1
Due to the curricular structure of the Universitat Hamburg as well as possible extracurricular activity, the knowledge of the students when starting the course is highly diverse. Firstly, the students are enrolled in di erent study programs. Because of this, there is not one single programming language everyone is trained in. As a consequence, we had to develop the tasks in di erent programming languages (Java and Python), which signi cantly increased the e orts, as this does not only imply creating the tasks twice, but also requires modelling this on the side of automatic score reporting. In addition, the students greatly diverged in programming skill levels. While some perceived the tasks as quite di cult, there was also a smaller group who found the tasks to be extremely easy. 5.2 CodeRunner adds an extra layer of abstraction between the student and the system on which the code is actually run. This extra layer caused many problems for the students.
Once, it is not directly visible what exactly the underlying system is doing, and especially what e ect the di erent user interface elements have on the system. To reduce di culties, we used di erent counter-measures: a live demonstration at the beginning of the semester, as well as a user manual that students could consult on any questions. Still, the students had problems with the user interface in the rst weeks.
In addition, errors in students' solutions are not easy to debug. Although any compiler errors or failed test cases are shown, the execution of the source code could not directly be analysed by standard tools like a debugger. It has been proven helpful to provide special source code les, which allowed the students to develop solutions on their own computer by emulating the behaviour of the system.
Finally, students were quick to blame the system for any error instead of searching them in their own solution. For example, one student has blamed the system for not allowing enough execution time for his solution although he programmed an in nite loop in his solution. Because of this and similar cases, we had a high demand of support (see below). 5.3
We could observe the problems of many students to apply the knowledge they gained in the lecture to the programming exercises. As a result, many students tried to use their self-developed, creative, algorithms instead of using the algorithms presented during the lecture. This was also the case for tasks like 'Implement algorithm X'. Often, these algorithms had many problems in di erent cases (especially in corner cases), which causes malfunction in both normal execution and our test cases. Because of this, it is proven to be especially important to cover each possible cause of errors with its own respective test case, some of which were hard to anticipate. This way, students could analyse each failed test case individually and easily nd their errors. Whenever there was a cause of error we did not anticipate (and therefore did not have a test case) we could observe the students having more problems.
In addition, there were cases where a student had problems with the random test cases while at the same time they passed all other test cases. This shows two things: There might be some hidden problems in the student's solution, and there were some test cases we had missing. While we plan to improve our test cases by collecting these issues, it is not always possible to avoid such problems since tasks might have to be changed each year in order to ensure unseenness. 5.4
Although there were no major problems with the programming exercises and the systems ran stable, there was still a high demand of support from the students in the form of questions and support requests whenever they were not able to solve an issue on their own. This includes for example questions concerning the interpretation of the given task, technical di culties, issues with their solution, and organizational questions. Since most of the aforementioned problems and resolutions were very speci c to the students' solutions, the support was highly individual and therefore caused high time and e ort demands. 6
To evaluate the acceptance of the programming exercises by the students, we
conducted a user study. The study is carried out following Kreidl [
Out of the 300 students that partook in the course, 44 students additionaly participated in the user study on a voluntary basis. As can be seen in Tab. 1, the programming exercises were accepted well by the students in general (all values are around 2). However, improvements can be made especially in the area of feedback to the students (variables availability of tasks and learning causes and feedback to the students ). The value intensity of usage is low in comparison to the others, however, this is expected, since it was intended that the students do the programming exercises only a single time.
We also evaluated the amount and di culty of the tasks. The students could rate the di culty on a scale of 1 (too easy) to 5 (too di cult), with 3 equal to being adequate. The amount of tasks was evaluated on a similar scale, from 1 (too many) to 5 (too few), with 3 equal to being a good number of tasks. The tasks received a di culty of 3.0 ( = 0:7) whereas the amount received a rating of 2.7 ( = 0:7). This shows that our tasks where perceived as having the right number and di culty for the course. 7
In this paper, we described our experiences of introducing automatically assessed programming exercises in a Bachelor's level course focusing on algorithms and data structures in computer science with around 300 yearly participants. The course is mostly focused on theoretical and formal aspects. Overall, the introduction of the programming exercises was successful, although we experienced some di culties in the area of the mixed prior knowledge of participants, students' creativity, the extra abstraction layer of CodeRunner, and a high demand of support. A user study shows that the programming exercises were accepted by the students, although there is still room for improvement especially in the area of feedback to the students concerning the speci c issues of their solutions.
Currently, it is planned to continue the programming exercises in next year's course. Improvements are especially planned for including better feedback. Since manual feedback by instructors is not feasible for the course, it is planned to improve feedback by both improving the test cases and the feedback included in the test cases (e.g. purpose of the test case and common mistakes).
Acknowledgements. This research was supported by MINTFIT Hamburg. MINTFIT Hamburg is a joint project of Hamburg University of Applied Sciences (HAW), HafenCity University Hamburg (HCU), Hamburg University of Technology (TUHH), University Medical Center Hamburg-Eppendorf (UKE) as well as Universitat Hamburg (UHH) and is funded by the Hamburg Authority for Science, Research and Gender Equality.