-Courses related to software testing education, at the university level, in most cases have a learning outcome requiring from students to understand and apply a set of test design techniques upon completing the course. The problem, however, remains on how to both effectively and efficiently evaluate if a student has accomplished the stated outcome. By purely looking at the final resulting set of the test cases provided by a student, it is not evident which, if any, test design technique was used to derive them. In this paper, we are presenting a rather simple but effective method of collecting video assignment submissions from students instead of a traditional source code and tests solution. This way, the teacher could rather quickly and in detail gather evidence that student indeed obtained the knowledge needed for passing the stated learning outcome.
With an ever-growing number of university students, who are studying software engineering, the need for efficient and effective methods of evaluating homework assignments is evident. By efficient, here we are mostly referring to the effort spent on grading these assignments by the examiner (or the teacher) and such an effort is often expressed in the time needed to complete it. In addition, another type of resources could be spent in grading, like using specific hardware where assignment has to be downloaded or a specific service (as in cloud-based) that has to be accessed. When referring to the effective evaluation of homework assignments, here we are mostly interested in how to easily detect deviations between the provided assignment and a given learning outcome for which the assignment was designed.
Software Testing education, as an important aspect of software engineering, can not be isolated from this problem. Practical assignments, in which it is often asked from students to apply specific test design techniques on a given system under test, could rather easily grow in complexity. But more importantly, in contrast to other disciplines, having only one correct solution in this field is rather unusual. This problem highlights the importance of evaluating how students applied a given test design technique rather than its outcome.
In this paper, we are presenting a rather simple but effective method of collecting video assignment submissions from students instead of a traditional source code and tests solution. This way, the teacher could rather quickly (by going fastforward) and in detail (by pausing and replaying certain parts) gather evidence that student indeed obtained the knowledge needed for passing the stated learning outcome. In addition, by sharing videos among students via course social media channels, motivation for students to provide quality solutions is even higher.
In the following section, we are discussing existing reports on advancing test education with respect to efficient assignment grading process, as well as some approaches to using video as a media for assignments. Section III address the peculiarities of software testing assignments and in particular discuss how to evaluate correct usage of the test-driven development approach. Section IV presents the process used for video assignments in one instance of a Software Verification & Validation course. Section V discusses threats to the general applicability of this approach with final conclusions and future work ideas presented in Section VI.
When it comes to increasing efficiency of software
programming assignments evaluation, several approaches have been
proposed [
Using videos as a media for student assignments has been
already proven successful [
However, to the best of our knowledge, we have not seen the usage of video techniques in the grading of software testing education assignments. Also, it is important to note here (a) A new test is added (b) A new code is added that we are discussing students video production using video recording devices (camcorders). In our case, when discussing video assignments, we are referring to computer desktop video recording production.
Learning outcomes presents an important instrument in ensuring the quality of a course at the higher level educational institution. For that reason, it is rather significant to properly design evaluation assignments and match them to the learning outcome.
For example, within the course on Software Verification & Validation, given at Ma¨lardalen University, we have defined, among others, the following learning outcome:
The idea is very simple, we expect from our students to know how to apply test design techniques. However, the problem arises when we try to evaluate if students have correctly applied a particular test design technique. Simple because our assignments are designed as follows:
By providing us with the Triangle.java and TriangleTest.java files, we, the teachers, have no ability to validate if students indeed applied test-driven development approach and what guidance they used when designing the tests. There is an evident need to have a more detailed view of the actual process conducted by students. In the following section, we discuss further why is this concern important in a test-driven development education.
Test-driven development process introduces an interesting
paradigm for software development [
• Red: The developer should first write a test and execute it to see it fails, and thus visualise that there is a problem which needs to be fixed. • Green: Afterwards, the developer has to create a minimal code needed to make the previously failing test pass. Only once all the tests are passing, the developer can proceed to the following step. • Refactor: After few iterations on the code it is important to refactor and improve the design of code. No new functionalities are added and existing test cases are ensuring that the current functionality is not corrupted.
Although not a direct test design technique, TDD does represent an important aspect of overall code quality. It is necessary to have the students exposed to it in order to better appreciate the effort of proper test design as well as test automation. However, due to the fact that the TDD concept is new to students and sometimes even considered rather unorthodox, we need to make sure that the students are indeed following it properly.
In this section, we are presenting our experience from applying the video assignment approach to an instance of the Software Verification & Validation course, given at Ma¨lardalen University in 2017. At the very beginning of the course, once the students are introduced to the fundamentals of software testing and in particular the test case design when using the test-driven development approach, an assignment was given to them in the form of a video submission. They were expected to record their desktop screen with Eclipse1 integrated development environment being open for the whole duration until the assignment is completed. The video is then to be posted on an internal social network at Ma¨lardalen University (Yammer2) within a group dedicated to students of this course. Figure 1 depicts a screenshot of one student submission as an example. Although students did not expect this type of assignment submission and had not previously done anything similar to it, they still managed to get the videos posted on time and with a rather good quality. Many of the solutions were also correct, but for a few, the teacher had to further discuss students’ submissions with them.
At least one of the students made a mistake by writing first code and then tests. When asked about this, the student admitted that he completely forgot that the assignment should be done by following the TDD approach, and rather soon resubmitted his new video solution, which was correct this time. However, the most interesting issue that was discussed with students, was the interpretation of the following TDD rule:
An important aspect of the success of video assignments is the tooling support for it. There exist several commercials, free or open source solutions for screen recording software. We did not impose on students the usage of any specific tool but rather suggested an existing open source solution that works on most platforms students could use without bringing any additional cost to them. The tool in question was Open Source Broadcaster (OBS).3 Although several students did follow the suggestion to use OBS software, others tried out different software they sought more suitable for the job.
Providing timely feedback to students assignments is an
important factor in the success of a teaching method. To
the developer has to create a minimal code needed improve efficiency of this process, teachers have applied
to make the previously failing test pass. several rather interesting ideas. Plimmer and Mason proposed
On several occasions, it was noticeable that students are not the usage of a paperless environment for annotation of students
really following this rule and as soon as they have a failing assignments [
One of the general threats to introducing new types of
assignments at a higher education institution is the phenomenon
of plagiarism. At Ma¨lardalen University there is a
recommendation to design assignments such that they discourage
any possibility of plagiarism appearing. However, as stated
by Leah [
Another interesting aspect to discuss regarding general applicability of video assignments is its usage by blind or visually impaired users. In previous instances of Software Verification & Validation courses we have witnessed challenges with respect to tooling support for blind and visual impaired persons. For example, one of the assignments includes code and test analysis using structural (code) coverage techniques. However, most tools for coverage provides colour-coded information regarding the state of a singe statement and/or branch. This rendered to be completely useless for some blind students that we had in a course. Indeed, this aspect has to be further investigated and better understand to what degree it is possible for blind and visual impaired users to perform desktop video recording.
Homework assignments are classified as active learning type of approach to knowledge building process. This is where students have to actively work on solving a dedicated practical problem given to them. However, they have to do that alone with no immediate help from the teacher or other peers in the course. In order to truly evaluate if the homework assignment has been conducted correctly, the teacher needs to see the process of creating it, rather than focusing on the actual solution or its outcome.
In this paper, we have presented an approach of using video assignments in grading students submissions, especially within the software testing education. By having a video submission from students, the teachers can potentially better utilise their time when evaluating if a solution for a given assignment fulfils the learning outcome of the course. Within the course of Software Verification and Validation at Ma¨lardalen University, this approach was tested by applying it to the TDD assignment. As a result, it was noticed that students to some extent do not follow the rules of TDD and could potentially accommodate the wrong process for using it. With the help of video assignments, this was noticed early in the course and students had the time to discuss and reflect on the mistakes done.
For the next instance of the course, the plan is to integrate video assignments within the learning platform itself rather than using social platform for it. In addition, a short introductory video could be made for students on how to create their own videos and motivate why this way of working is beneficial for students as well as teachers.
This work was supported by the Swedish Knowledge Foundation (KKS) through projects FuturE and PROMPT - Professional Master’s in Software Engineering. The author would like to thanks to the colleagues and students at Ma¨lardalen University for their inspiration and support.