=Paper=
{{Paper
|id=Vol-2066/isee2018paper01
|storemode=property
|title=Appreciate the Journey not the Destination - Using Video Assignments in Software Testing Education
|pdfUrl=https://ceur-ws.org/Vol-2066/isee2018paper01.pdf
|volume=Vol-2066
|authors=Adnan Causevic
|dblpUrl=https://dblp.org/rec/conf/se/Causevic18
}}
==Appreciate the Journey not the Destination - Using Video Assignments in Software Testing Education==
https://ceur-ws.org/Vol-2066/isee2018paper01.pdf
Appreciate the Journey not the Destination - Using
Video Assignments in Software Testing Education
Adnan Čaušević
Software Testing Laboratory, Mälardalen University
Västerås, Sweden
adnan.causevic@mdh.se
Abstract—Courses related to software testing education, at the by sharing videos among students via course social media
university level, in most cases have a learning outcome requiring channels, motivation for students to provide quality solutions
from students to understand and apply a set of test design is even higher.
techniques upon completing the course. The problem, however,
remains on how to both effectively and efficiently evaluate if a In the following section, we are discussing existing reports
student has accomplished the stated outcome. By purely looking on advancing test education with respect to efficient assign-
at the final resulting set of the test cases provided by a student, ment grading process, as well as some approaches to using
it is not evident which, if any, test design technique was used to video as a media for assignments. Section III address the
derive them. In this paper, we are presenting a rather simple but peculiarities of software testing assignments and in particular
effective method of collecting video assignment submissions from
students instead of a traditional source code and tests solution. discuss how to evaluate correct usage of the test-driven de-
This way, the teacher could rather quickly and in detail gather velopment approach. Section IV presents the process used for
evidence that student indeed obtained the knowledge needed for video assignments in one instance of a Software Verification
passing the stated learning outcome. & Validation course. Section V discusses threats to the general
applicability of this approach with final conclusions and future
I. I NTRODUCTION work ideas presented in Section VI.
With an ever-growing number of university students, who
are studying software engineering, the need for efficient and II. R ELATED W ORK
effective methods of evaluating homework assignments is When it comes to increasing efficiency of software program-
evident. By efficient, here we are mostly referring to the effort ming assignments evaluation, several approaches have been
spent on grading these assignments by the examiner (or the proposed [1]–[5]. Most of these approaches are focusing on
teacher) and such an effort is often expressed in the time the automated grading of students code by utilising a back-
needed to complete it. In addition, another type of resources end testing system. In some cases, evaluation is done on the
could be spent in grading, like using specific hardware where tests as well in the form of coverage reports and also using
assignment has to be downloaded or a specific service (as in static analysis as a mean to provide code styling conformance
cloud-based) that has to be accessed. When referring to the checks. But, most of these approaches are focusing on grading
effective evaluation of homework assignments, here we are the final assignment submission of a student. As we discuss
mostly interested in how to easily detect deviations between further in this paper it is evident that omitting the process,
the provided assignment and a given learning outcome for used by a student in creating the assignment, may hinder
which the assignment was designed. correct evaluation of obtained knowledge in applying test
Software Testing education, as an important aspect of design techniques. Consequently, we are proposing the usage
software engineering, can not be isolated from this problem. of video assignment submissions as a mean to overcome this
Practical assignments, in which it is often asked from students aspect of the students grading.
to apply specific test design techniques on a given system Using videos as a media for student assignments has been
under test, could rather easily grow in complexity. But more already proven successful [6], [7]. Leah, for example, pre-
importantly, in contrast to other disciplines, having only one sented how students of Adelphi University have used video
correct solution in this field is rather unusual. This problem recording devices and video editing software to perform as-
highlights the importance of evaluating how students applied signments related to their motor (physical) skills [6]. Kearney
a given test design technique rather than its outcome. and Schuck [7] discuss in details the nature of student video
In this paper, we are presenting a rather simple but effective projects and its relation to learning. Both students and teachers
method of collecting video assignment submissions from stu- in their studies have indicated that video projects of students
dents instead of a traditional source code and tests solution. do develop authentic learning, referring to students ability to
This way, the teacher could rather quickly (by going fast- connect their education to the real world.
forward) and in detail (by pausing and replaying certain parts) However, to the best of our knowledge, we have not seen the
gather evidence that student indeed obtained the knowledge usage of video techniques in the grading of software testing
needed for passing the stated learning outcome. In addition, education assignments. Also, it is important to note here
ISEE 2018: 1st Workshop on Innovative Software Engineering Education @ SE18, Ulm, Germany 4
(a) A new test is added (b) A new code is added
Fig. 1: Student writing a code to pass the previously failing test.
that we are discussing students video production using video developers are constantly building quality into their code.
recording devices (camcorders). In our case, when discussing However, the process itself is rather sensitive to deviations and
video assignments, we are referring to computer desktop video thus should be followed in a rather strict “red-green-refactor”
recording production. patter.
• Red: The developer should first write a test and execute
III. S OFTWARE T ESTING A SSIGNMENTS
it to see it fails, and thus visualise that there is a problem
Learning outcomes presents an important instrument in which needs to be fixed.
ensuring the quality of a course at the higher level educational • Green: Afterwards, the developer has to create a minimal
institution. For that reason, it is rather significant to properly code needed to make the previously failing test pass. Only
design evaluation assignments and match them to the learning once all the tests are passing, the developer can proceed
outcome. to the following step.
For example, within the course on Software Verification & • Refactor: After few iterations on the code it is important
Validation, given at Mälardalen University, we have defined, to refactor and improve the design of code. No new func-
among others, the following learning outcome: tionalities are added and existing test cases are ensuring
Upon fulfilling the course, the student should be that the current functionality is not corrupted.
able to apply various software testing techniques in Although not a direct test design technique, TDD does repre-
development projects. sent an important aspect of overall code quality. It is necessary
The idea is very simple, we expect from our students to know to have the students exposed to it in order to better appreciate
how to apply test design techniques. However, the problem the effort of proper test design as well as test automation.
arises when we try to evaluate if students have correctly However, due to the fact that the TDD concept is new to
applied a particular test design technique. Simple because our students and sometimes even considered rather unorthodox,
assignments are designed as follows: we need to make sure that the students are indeed following
Create a working Eclipse project for the Triangle it properly.
problem.You need to provide both the program and
the test cases.You have to create tests before the code IV. V IDEO S UBMISSIONS
(Test-driven Development approach). In this section, we are presenting our experience from
By providing us with the Triangle.java and TriangleTest.java applying the video assignment approach to an instance of the
files, we, the teachers, have no ability to validate if students Software Verification & Validation course, given at Mälardalen
indeed applied test-driven development approach and what University in 2017. At the very beginning of the course, once
guidance they used when designing the tests. There is an the students are introduced to the fundamentals of software
evident need to have a more detailed view of the actual testing and in particular the test case design when using the
process conducted by students. In the following section, we test-driven development approach, an assignment was given
discuss further why is this concern important in a test-driven to them in the form of a video submission. They were ex-
development education. pected to record their desktop screen with Eclipse1 integrated
development environment being open for the whole duration
A. Grading the TDD process
until the assignment is completed. The video is then to be
Test-driven development process introduces an interesting posted on an internal social network at Mälardalen University
paradigm for software development [8]. By enforcing devel-
opers to write tests prior to the code, in small time cycles, 1 www.eclipse.org
ISEE 2018: 1st Workshop on Innovative Software Engineering Education @ SE18, Ulm, Germany 5
Fig. 2: Teacher using telestrator while providing video feedback to students
(Yammer2 ) within a group dedicated to students of this course. A. Tooling
Figure 1 depicts a screenshot of one student submission as An important aspect of the success of video assignments
an example. Although students did not expect this type of is the tooling support for it. There exist several commercials,
assignment submission and had not previously done anything free or open source solutions for screen recording software. We
similar to it, they still managed to get the videos posted on did not impose on students the usage of any specific tool but
time and with a rather good quality. Many of the solutions rather suggested an existing open source solution that works
were also correct, but for a few, the teacher had to further on most platforms students could use without bringing any
discuss students’ submissions with them. additional cost to them. The tool in question was Open Source
At least one of the students made a mistake by writing Broadcaster (OBS).3 Although several students did follow the
first code and then tests. When asked about this, the student suggestion to use OBS software, others tried out different
admitted that he completely forgot that the assignment should software they sought more suitable for the job.
be done by following the TDD approach, and rather soon re-
submitted his new video solution, which was correct this time. B. Video Feedback
However, the most interesting issue that was discussed with
Providing timely feedback to students assignments is an
students, was the interpretation of the following TDD rule:
important factor in the success of a teaching method. To
the developer has to create a minimal code needed improve efficiency of this process, teachers have applied
to make the previously failing test pass. several rather interesting ideas. Plimmer and Mason proposed
On several occasions, it was noticeable that students are not the usage of a paperless environment for annotation of students
really following this rule and as soon as they have a failing assignments [9]. Morris and Chikwa [10] evaluated if using
test, they write a code that goes way beyond the scope of that audio comments instead of a paper based student feedback
test. This results in having written a code that was not fully would increase the overall students’ grades. Their findings
tested. This was discussed in details with students in one of the pointed out no significant difference although students were
follow-up lectures after the assignment deadline. Essentially, positive regarding audio feedback.
their inexperience with TDD has led to making such a mistake. To support students in creating videos, the teacher in the
Still, having video assignments, it made it easier for teachers course has also made a few. Figure 2 picture a screenshot of a
to detect this deviation and address it as early as possible pre-recorded video feedback provided to students. Essentially
such that incorrect application of TDD practice in future is it is a video of a teacher watching and commenting on other
hopefully prevented. students videos. Our idea was that we cannot expect students
2 www.yammer.com 3 www.obsproject.com
ISEE 2018: 1st Workshop on Innovative Software Engineering Education @ SE18, Ulm, Germany 6
to do something that we, the teachers, are not ready to use In this paper, we have presented an approach of using video
ourselves. Video feedback was accepted by students as a rather assignments in grading students submissions, especially within
positive approach in giving a timely feedback. the software testing education. By having a video submission
from students, the teachers can potentially better utilise their
V. T HREATS TO A PPLICABILITY time when evaluating if a solution for a given assignment
fulfils the learning outcome of the course. Within the course of
In this section, we present two potential threats to a general
Software Verification and Validation at Mälardalen University,
applicability of using video assignments in software testing
this approach was tested by applying it to the TDD assignment.
education.
As a result, it was noticed that students to some extent do not
One of the general threats to introducing new types of as-
follow the rules of TDD and could potentially accommodate
signments at a higher education institution is the phenomenon
the wrong process for using it. With the help of video
of plagiarism. At Mälardalen University there is a recom-
assignments, this was noticed early in the course and students
mendation to design assignments such that they discourage
had the time to discuss and reflect on the mistakes done.
any possibility of plagiarism appearing. However, as stated
For the next instance of the course, the plan is to integrate
by Leah [6], students who submitted video assignments had
video assignments within the learning platform itself rather
to actually do them by themselves as they did not had a
than using social platform for it. In addition, a short intro-
possibility to reuse already completed assignment from a
ductory video could be made for students on how to create
previous year students. We noticed the same pattern in our
their own videos and motivate why this way of working is
study as students could not just copy & paste the code and test
beneficial for students as well as teachers.
cases from another student pier. Instead they had to write first
test and then the minimum code making that test pass. It would ACKNOWLEDGEMENT
require a significant skill to be able to decompose an existing This work was supported by the Swedish Knowledge Foundation
code and test solution into a TDD pattern (reverse engineering (KKS) through projects FuturE and PROMPT - Professional Master’s
the TDD process). However, this is only applicable to the TDD in Software Engineering. The author would like to thanks to the
related assignments. For any other type of video assignments colleagues and students at Mälardalen University for their inspiration
and support.
plagiarism opportunities do exist and they should be treated
accordingly. One approach would be to record a video camera R EFERENCES
stream overlay of the student during the completion of their [1] S. Elbaum, S. Person, J. Dokulil, and M. Jorde, “Bug hunt: Making early
task. However, this approach would increase the privacy issue software testing lessons engaging and affordable,” in Proceedings of the
and it remains unknown what effect it might have on the 29th International Conference on Software Engineering, ser. ICSE ’07.
Washington, DC, USA: IEEE Computer Society, 2007, pp. 688–697.
quality of the assignment. [2] J. Spacco and W. Pugh, “Helping students appreciate test-driven de-
Another interesting aspect to discuss regarding general velopment (tdd),” in Companion to the 21st ACM SIGPLAN Symposium
on Object-oriented Programming Systems, Languages, and Applications,
applicability of video assignments is its usage by blind or ser. OOPSLA ’06. New York, NY, USA: ACM, 2006, pp. 907–913.
visually impaired users. In previous instances of Software Ver- [3] S. H. Edwards and M. A. Perez-Quinones, “Web-cat: Automatically
ification & Validation courses we have witnessed challenges grading programming assignments,” in Proceedings of the 13th Annual
Conference on Innovation and Technology in Computer Science Educa-
with respect to tooling support for blind and visual impaired tion, ser. ITiCSE ’08. New York, NY, USA: ACM, 2008, pp. 328–328.
persons. For example, one of the assignments includes code [4] A. Allowatt and S. H. Edwards, “Ide support for test-driven development
and test analysis using structural (code) coverage techniques. and automated grading in both java and c++,” in Proceedings of the 2005
OOPSLA Workshop on Eclipse Technology eXchange, ser. eclipse ’05.
However, most tools for coverage provides colour-coded infor- New York, NY, USA: ACM, 2005, pp. 100–104.
mation regarding the state of a singe statement and/or branch. [5] I. Hernn-Losada, C. Pareja-Flores, and J. . Velzquez-Iturbide, “Testing-
This rendered to be completely useless for some blind students based automatic grading: A proposal from bloom’s taxonomy,” in
2008 Eighth IEEE International Conference on Advanced Learning
that we had in a course. Indeed, this aspect has to be further Technologies, July 2008, pp. 847–849.
investigated and better understand to what degree it is possible [6] L. H. Fiorentino, “Digital video assignments: Focusing a new lens on
for blind and visual impaired users to perform desktop video teacher preparation programs,” Journal of Physical Education, Recre-
ation & Dance, vol. 75, no. 5, pp. 47–54, 2004.
recording. [7] M. Kearney and S. Schuck, “Spotlight on authentic learning: Student
developed digital video projects,” Australasian Journal of Educational
VI. C ONCLUSION AND F UTURE W ORK Technology, vol. 22, no. 2, pp. 189–208, 2006.
[8] K. Beck, Test Driven Development. By Example (Addison-Wesley Sig-
Homework assignments are classified as active learning nature). Addison-Wesley Longman, Amsterdam, 2002.
type of approach to knowledge building process. This is where [9] B. Plimmer and P. Mason, “A pen-based paperless environment for
annotating and marking student assignments,” in Proceedings of the 7th
students have to actively work on solving a dedicated practical Australasian User Interface Conference - Volume 50, ser. AUIC ’06.
problem given to them. However, they have to do that alone Darlinghurst, Australia, Australia: Australian Computer Society, Inc.,
with no immediate help from the teacher or other peers in the 2006, pp. 37–44.
[10] C. Morris and G. Chikwa, “Audio versus written feedback: Exploring
course. In order to truly evaluate if the homework assignment learners? preference and the impact of feedback format on students?
has been conducted correctly, the teacher needs to see the academic performance,” Active Learning in Higher Education, vol. 17,
process of creating it, rather than focusing on the actual no. 2, pp. 125–137, 2016.
solution or its outcome.
ISEE 2018: 1st Workshop on Innovative Software Engineering Education @ SE18, Ulm, Germany 7