-We supplement our software engineering lectures and exercises with facultative online programming assignments. Based on our experience with previous systems, we developed INLOOP, a solution tailored to our needs. The core of the system is a test runner component that supports virtually any kind of solution assessment through a generic interface. The system supports automated plagiarism checks and grading, which we currently use to award students bonus points for written exams. In addition, it is directly integrated with a version control system to facilitate peer-reviewed development of assignments by our team of instructors and to enable continuous publishing of assignments. The resulting workflow helped to improve the quality of our assignments and tests. The architecture of INLOOP is characterized by asynchronous solution processing and takes advantage of container technology to achieve scalability, robustness and isolation, while maintaining a small footprint. So far, our approach is a success: we observed an increasing number of students who actively practiced programming, and they clearly performed better in the written exam. The poster presents the architecture of INLOOP and the implemented continuous publishing workflow.
The motivation of the INLOOP approach is to improve teaching in software engineering by taking advantage of interactive technologies. Our goals can be described from four perspectives: Students should be given an easy-to-use environment to practice their software development skills. They should receive immediate, detailed and user-friendly feedback for submitted solutions. Instructors should be given tools to manage complex programming assignments and handle increasing numbers of students, because at such a scale, individual feedback on solutions cannot be manually provided. Operators are primarily concerned about the robustness and scalability of the system. Researchers and Developers want a system that is easy to comprehend and extend.
Since 2006, we have been supplementing our software engineering lectures and exercises with facultative online programming assignments. The course focuses on objectoriented modeling and programming using UML and Java. Our system uses public JUnit tests to automatically verify solutions submitted by the students; the task complexity ranges from basic to demanding, where students have to implement a design given by a UML diagram. We award students one exam bonus point for each demanding task solved.
In INLOOP, tasks are organized into categories and have
optional publication dates and deadlines, which can be used to
restrict the user’s ability to view a task or submit solutions for
it. The test runner of INLOOP supports virtually any kind of
solution assessment, i.e., any build, test or analysis tool that is
available for Linux, in a secure and robust manner. Instructors
specify the test environment for a task in a declarative fashion,
and the system builds a test image that is used by the test
runner. By default, the test output is shown to the user as-is, but
pretty-printer modules may transform it into HTML fragments
that can be integrated into the user interface. Additionally,
the system uses an offline version of the JPlag plagiarism
checker [
A distinctive feature of INLOOP is its direct integration
with the Git version control system: all task artifacts
(metadata, description, diagrams, tests, etc.) are retrieved from a
remote repository, transformed if necessary, and published. For
example, such transformations include building the test image,
rendering Markdown task texts, and creating ZIP archives of
supplementary material. This process is triggered by a Git push
directly in the IDE on the instructor’s computer; a feature we
refer to as continuous publishing (Fig. 1). Before, we had to
manually edit and publish tasks in the backend of a web
application, which was an error-prone process. Our approach has
the following advantages: 1) the process is fully automated,
2) all task artifacts are stored in one place, 3) changes can be
tracked, and 4) task artifacts can be developed in collaboration.
Another key characteristic is that continuous publishing can be
combined with continuous integration tests and a branching
model such as Gitflow [
1Although INLOOP does not change the status of existing submissions when a new task revision is published, maintaining a stable task set during a semester is advisable to avoid student confusion.
INLOOP is designed as a multi-process application (Fig. 2). The core components are the web application and the background workers, both implemented in Python. An event-based job broker and a relational database are used to exchange data between the processes. The web application is a standard Django application, but instead of blocking on long-running test runner jobs, it submits them to a job queue. The jobs are picked up by a configurable number of background workers, and are executed inside their own, ephemeral Docker container. Apart from the necessary input and output, the containers are completely isolated from each other, the network, and the host system; in addition, strict resource limits are enforced.
Having bonus points as an incentive seems to be a good motivation for our students. Hence, we observed an increasing number who practiced their development skills with INLOOP. In the summer term 2017, 11074 solutions were submitted, up to 419 per day in the week before the written exam. Students who actively practiced programming online clearly performed better in the written exam. But in our setting, plagiarism checks remain a necessity: out of 1838 valid solutions, 127 (6,9%) have been identified as plagiarism, and some of them were complete copies.
With continuous publishing, we eliminated inconsistencies by generating all published content from a single source repository. The streamlined workflow enabled us to spend more time on actual task development, instead of time-consuming and error-prone manual publishing. Furthermore, our chosen branching model made it easy to incorporate urgent bug fixes into the current “stable” set of tasks, while at the same time being able to prepare larger source code transformations for the next semester.
There are many online programming systems available. We
are not aware of one that incorporates a continuous publishing
workflow based on version control, but the idea has been
used in the past in other publishing systems, e.g., the Open
Journal System [
We plan to integrate static code analysis and style checks
into INLOOP and ultimately reject code that violates a
predefined coding standard. In the same spirit, the current binary
solved/not solved test result could also be enhanced to give
grades based on the code quality metrics calculated by such
tools. To make it more challenging for students to solve tasks
by just satisfying static test data, our unit tests could be
complemented with property-based testing. Another promising
idea, inspired by Web-CAT [