=Paper=
{{Paper
|id=Vol-2358/paper-07
|storemode=property
|title=Using Mini-Projects to Teach Empirical Software Engineering
|pdfUrl=https://ceur-ws.org/Vol-2358/paper-07.pdf
|volume=Vol-2358
|authors=Michael Felderer,Marco Kuhrmann
|dblpUrl=https://dblp.org/rec/conf/seuh/FeldererK19
}}
==Using Mini-Projects to Teach Empirical Software Engineering==
https://ceur-ws.org/Vol-2358/paper-07.pdf
Using Mini-Projects to Teach
Empirical Software Engineering
Michael Felderer1 and Marco Kuhrmann2
1
University of Innsbruck, michael.felderer@uibk.ac.at
2
Clausthal University of Technology, kuhrmann@acm.org
Abstract lines exist, such as for systematic reviews [16, 28],
Empirical studies have become a central element of systematic mapping studies [23, 24], multi-vocal re-
software engineering research and practice. Yet, teach- views [10, 11], or surveys [12, 21]. All these instru-
ing the instruments of empirical software engineering ments are meant to support researchers and practi-
is challenging, since students need to understand the tioners alike in conducting empirical studies and to
theory of the scientific method and also have to de- ground their work and decisions in evidence.
velop an understanding of the application of those in- Conducting empirical studies is challenging and
struments and their benefits. In this paper, we present requires careful preparation and a disciplined work
and evaluate an approach to teach empirical software approach. Quite often, students consider empirical
engineering with course-integrated mini-projects. In studies of little to no help when it comes to software
mini-projects, students conduct small empirical stud- development and project work, since the relation to
ies, e.g., surveys, literature reviews, controlled ex- actual development tasks is not obvious. Yet, many
periments, and data mining studies in collaborating of today’s applications rely on data, e.g., machine
teams. We present the approach through two im- learning systems like text and speech recognition, IoT
plementations at two universities as a self-contained devices, and autonomous cars. Empirical methods
course on empirical software engineering and as part as such are about data analysis and, thus, provide
of an advanced software engineering course; with 101 a suitable approach to teach data analysis—or data
graduate students in total. Our evaluation shows a engineering in general—that is a core competence
positive learning experience and an improved under- in data-intensive applications. Furthermore, modern
standing of the concepts taught. More than a half software development paradigms, such as DevOps
of the students consider empirical studies helpful for including continuous integration and deployment, uti-
their later careers. Finally, a qualitative coding and lize data, e.g., to analyze a system’s performance, to
a statistical analysis showed the proposed approach predict defects, and to make informed decisions in
beneficial, but also revealed challenges of the scien- the development process as practiced in continuous
tific work process, e.g., data collection activities that experimentation [5]. Therefore, it is necessary for
were underestimated by the students. teachers to open the students’ minds for a rigorous
and evidence-based work approach.
1 Introduction In this paper, we present and evaluate the concept of
Empirical software engineering aims at making soft- course-integrated mini-projects to teach empirical soft-
ware engineering claims measurable, i.e., to analyze ware engineering instruments. Our approach helps
and understand phenomena in software engineering students learn how to conduct empirical studies and
and to evaluate software engineering approaches and understand the instruments and challenges coming
solutions, and to ground decision-making processes along with such studies. Collaborating project teams
in evidence. For this, an extensive portfolio of instru- conducting small empirical studies form the basis of
ments for empirical software engineering has been our approach. We implemented the approach in two
developed. For instance, Wohlin et al. [27] provide a courses at the University of Southern Denmark (2016,
collection of instruments, e.g., controlled experiments, 68 students) and University of Innsbruck (2017, 33
surveys and case studies, to be used for empirical stud- students). Mini-projects allow students to learn em-
ies in software engineering. Kitchenham et al. [13] pirical instruments by practically applying them. Our
extended these instruments by a detailed guideline for evaluation shows a positive learning experience and
conducting systematic reviews. For most of the basic an improved understanding of (empirical) software
instruments used in empirical software engineering engineering concepts. More than half of the students
today, extended and more detailed (pragmatic) guide- perceive empirical studies helpful for their later ca-
V. Thurner, O. Radfelder, K. Vosseberg (Hrsg.): SEUH 2019 75
Using Mini-Projects to Teach Empirical Software Engineering
Michael Felderer und Marco Kuhrmann, Uni Insbruck & TU Clausthal
reers. Our evaluation also shows that notably data out actual research to practice the application of the
collection activities (e.g., for surveys and experiments) empirical instruments. In our previous work [17–19],
are underestimated by the students. Our findings thus we presented different, self-contained classroom ex-
lay the foundation for improving research-oriented periments and developed a guideline to select the best-
courses that require data and data analysis. fitting study type for a specific context [6]. In [14], we
The remainder of the paper is organized as follows: introduced a teaming model that helps implementing
Section 2 gives an overview of background and re- empirical studies in larger project courses.
lated work. Section 3 describes the mini-project ap- We contribute a generalized concept grounded in
proach, and Section 4 presents the approach’s evalua- [6, 14] that allows for including empirical studies as
tion based on two implementations at the University course units. We implemented and evaluated our
of Southern Denmark and the University of Innsbruck approach in a course on empirical software engineer-
respectively. We conclude the paper and discuss future ing and an advanced course on software engineering
work in Section 5. and demonstrate how to implement and scale course-
integrated empirical studies.
2 Background and Related Work
Using empirical studies in software engineering ed-
3 Course-Integrated Mini-Projects
ucation is not a new idea [2]. However, empiri- We present the course-integrated mini-projects ap-
cal studies—notably (controlled) experiments—are proach in Section 3.1. The presentation includes the
mainly used as a tool to support research using stu- description of the team setups, project and task de-
dents as subjects, but got little appreciation as a teach- scriptions, and examples for which we present details
ing tool in software engineering in the first place. in Section 3.2. Section 3.3 demonstrates two inte-
That is, students only get in touch with empirical gration strategies: the first integration strategy is a
studies as subjects in an empirical inquiry, and they self-contained course on empirical software engineer-
have to carry out tasks, e.g., in an experiment as for ing [14] and the second strategy is a topic-specific
instance reported in [7, 8, 18, 20]. Yet, teaching em- part of an advanced software engineering course.
pirical software engineering as a subject requires a 3.1 Mini-Projects and Project Teams
setup in which empirical studies are the main sub-
Figure 1 shows the general organization model for the
ject or at least provide a significant contribution to
mini-project approach. A MiniProject has a Topic, a
a course. In this regard, Wohlin [26] proposes three
Schedule, and optional Reference Literature and
levels for integrating empirical studies in software
Input Data. It is always carried out using at least one
engineering courses: (i) integration in software en-
Method, e.g., an experiment [27], a case study [25],
gineering courses, (ii) as a separate course, and (iii)
and a survey [21]. Finally, every mini-project consists
as part of a research method course. Wohlin men-
of a Project Team and an Advisory Team, which we
tions that introducing empirical software engineering
describe in more detail in the following.
will provide more opportunities to conduct empirical
studies in student settings, but that educational and 1 1..*
research objectives need to be carefully balanced. Dil- Mini Project Result
lon [4] comes to the same conclusion and considers - Topic [1]
- Schedule [1]
a successful observation of a phenomenon as part of - Method [1..*]
joint research
- Reference Literature [*]
an empirical study not be an end in itself. Students - Input Data [*]
I
need time to get familiar with ideas and concepts 1 1
request
1
advice 0..*
associated with the phenomenon under observation.
Advisory Team Project Team
Finally, Parker [22] considers experiments distinctive provide 0..*
1 1..* 1
and more participative. Students are likely to remem- advice
1..* S
ber lessons associated with experiments. shared
Advisor 2..* research design
In this paper, we present an approach that considers Student
empirical studies major subjects of a course and that Teacher Practice Team
uses such studies as teaching tool. Referring to estab- External Advisor Method Team
lished learning models such as Bloom’s Taxonomy of
Service Team
Learning [1] and Dale’s Cone of Learning [3], we aim
to include as many active learning parts as possible
in the courses. Still, we use passive learning methods Figure 1: Overall organization model of mini-projects.
to transfer knowledge about theoretical basics, such
as methods and their application contexts. Address-
ing the active learning levels, however, is challenging. Advisory Teams An advisory team bundles all advi-
In “ordinary” software engineering education, project sors involved in a specific mini-project—usually one
courses are used to train software project work. For or two persons. An Advisor is either the teacher of
empirical studies, it is required that the students carry the course or an external advisor, such as an external
V. Thurner, O. Radfelder, K. Vosseberg (Hrsg.): SEUH 2019 76
Using Mini-Projects to Teach Empirical Software Engineering
Michael Felderer und Marco Kuhrmann, Uni Insbruck & TU Clausthal
Style Description Item Description
Isolated The “normal” way of doing a mini-project is Metadata This section contains all information rele-
the isolated way of working. Isolated means vant to a task, e.g., hand-in date.
that a project team has a self-contained task Title A project needs a telling title and an ID
that can be worked on without any interaction Context This section briefly describes the context of
with other teams. the project and provides a short summary
Joint This style is applied if project teams collabo- of the basic tasks. Recommendation: the
ratively work on a joint (research) project. A context section should be treated as an ab-
complex project is broken down into a num- stract, such that it can be used as a teaser
ber of smaller projects. Project teams thus and a small piece of information, e.g., used
have to be coordinated in terms of task distri- in a course management system.
bution, scheduling, and result synthesis. Work The detailed work description contains at
Shared This style is applied if project teams competi- Description least:
tively work on the same (research) topic. Two 1. A detailed task list.
or more teams are assigned the same task;
team-specific methods can be varied and re- 2. A list of input/reference material.
sults can be compared. That is, this style helps 3. A list of deliverables to be shipped.
conducting controlled experiments or imple-
Note: The level of detail depends on the
menting independently conducted studies.
actual task, i.e., for an “explorative” task,
the description needs to be more open while
Table 1: Overview of the different interaction styles a specific development task requires a more
among mini-project teams. detailed task description.
Schedule The basic schedule lists all deadlines and
the respectively expected results.
project topic sponsor [14]. Besides offering and pro- Related Our concept allows for collaborative and
moting project topics, advisors regularly interact with Projects competitive work (Figure 1 and Table 1). If
the project teams for which they handle individual such a collaborative/competitive work style
support requests, and they provide general technical is implemented, this section provides the in-
and methodical support. formation about the other teams. If method
or service teams provide useful services to
a project, such teams are referred here too.
Project Teams A project is composed of all students Literature This section lists selected reference litera-
working on a specific problem. Project teams can ture relevant to a project.
interact with each other. We distinguish the three in-
teraction styles isolated, joint, and shared, which are Table 2: Structure of a mini-project task description
explained in Table 1. Furthermore, we distinguish (recommended minimal elements).
three types of project teams according to the type
of task they are working on: a Practice Team per-
forms an “active” task, e.g., a development task or a fied right here; alternatively, a separate catalog of re-
research task. A Method Team deals with methodologi- sults has to be provided, e.g., including templates and
cal expertise, i.e., it develops competencies regarding mandatory/recommended outlines. Relevant types
specific (scientific) methods and offers “consultancy for project outcomes are, e.g., (research) data1 , es-
services” in terms of applying a specific method “right” says or reports, presentations, tutorials, and software.
to practice teams. Finally, a Service Team develops The second important item of the task description is
skills in more general topics, such as data analysis or the list of related projects. For instance, if a task is a
presentation, and offers respective “services” to other collaborative task, this list refers to all related projects
teams—practice teams and method teams alike. that contribute to the overall project goal. Further-
3.2 Project- and Task Descriptions more, this list also refers those method and service
teams that provide useful support, e.g., if the project
Every mini-project is supposed to produce at least
is concerned with developing and conducting a survey,
one Result. In this section, we provide a blueprint
this list can refer to a method team that is focused
for a 1-page project- and task description, which also
on the theoretical aspects of survey research. Finally,
illustrates the manifestation of the different attributes
the task description can also be used to develop a
of the class Mini Project (Figure 1).
checklist, which is used for the final submission. This
For every project, a description that includes tasks,
checklist helps students to check if their delivery pack-
dates, and expected results is necessary. Table 2 pro-
age is complete, and it helps teachers validating the
vides a summary and a description of task-description
delivery and grade its components. Figure 2 shows
items that we consider relevant. The work descrip-
tion requires special attention as it comprises the de- 1 Recommendation: If students conduct a research task, analyzed
tailed activity list, input material, and the description data that is necessary for the project documentation must always
of expected results. The expected results are speci- be complemented with the original raw data.
V. Thurner, O. Radfelder, K. Vosseberg (Hrsg.): SEUH 2019 77
Using Mini-Projects to Teach Empirical Software Engineering
Michael Felderer und Marco Kuhrmann, Uni Insbruck & TU Clausthal
Study Type SCM ASE
Gr Top Gr Top
Theory (tutorial) 3 9 9 7
Experiment 3 1 1 3 2 1
Survey 3 4 2 3 3 1
Systematic Review 3 3 2 3 1 1
Mapping Study 3 1 1 7
Simulation 3 2 2 7
Data Mining Study 7 3 7 3
Table 3: Overview of the study types implemented
including the number of groups for a specific method
(Gr) and the number of topics per study type (Top).
Introduction and
Fundamentals
Lecture Exercise
Presentation and
Scientific Writing
Model
Paper Reviews
Introduction to
Figure 2: Example of a mini-project task description. Empirical Research
Self-directed learning: working on the selected topics,
e.g., SLRs or surveys (incl. presenting and writing) Presentations: Theory and
Tutorial Topics
a practically used example of a task description as
described in Table 2. This task description is taken Status Control and
Guest Lectures theory experts
from the course given at University of Southern Den- help practice
mark and describes a survey research task, which Status Control and teams…
was performed collaboratively with two external re- Guest Lectures
searchers [9]. This particular project was a collabora-
tive project. Specifically, two teams (No. 15 and 16; Presentations: Secondary Studies
see the related projects part) were assigned one task,
but had to conduct the survey with different target
Presentations: Experiments and
groups. Each team submitted its own data and report, Simulations
but, both teams gave only one joint presentation.
Presentations: Survey Research
3.3 Course-Integration Strategies
We describe two implementation strategies for the Group Papers’
presented approach using two graduate courses. For Peer Review
these implementations, we provide a short summary
of the respective courses and their learning goals, and Evaluation and
Wrap-Up
we provide an overview of the projects implemented
in the respective courses (Table 3). Furthermore, this
section lays the foundation for the approach’s evalua- Figure 3: Overview of the topics and the general orga-
tion, which is presented in Section 4. nization of the SCM course.
3.3.1 Implementation as a Self-Contained
Empirical Software Engineering Course cally, the major learning goals of the SCM course were
defined as follows:
A course on the Scientific Method (SCM, University
of Southern Denmark, SDU, 2016) implemented the • After the course, students know the basic terminol-
presented approach as a self-contained master-level ogy and the key concepts of the scientific method.
course. Figure 3 illustrates the overall organization of
the SCM course showing the introduction parts and • After the course, students know and understand
the active learning/project parts. the most important empirical research methods.
The goal of the SCM course was to teach empirical • After the course, students have shown their ability
software engineering as main subject by letting the stu- to practically apply one research method, conduct
dents perform small studies themselves [14]. Specifi- and report on a small research project.
V. Thurner, O. Radfelder, K. Vosseberg (Hrsg.): SEUH 2019 78
Using Mini-Projects to Teach Empirical Software Engineering
Michael Felderer und Marco Kuhrmann, Uni Insbruck & TU Clausthal
In total, 30 research topics were presented to the stu- Overview Software
dents. According to their preferences, students could Engineering
Technical Software Engineering
apply for up to three topics, which built the founda- Empirical Methods in
tion for the final team setup. Finally, 20 projects were Software Engineering
started with two to three students each. Besides the
Software Process
theoretical topics, five research methods were covered
Lectures
Models Complementing
by the projects (Table 3). Material/Activities:
The SCM course implements the concept from Fig- Modeling Languages
Topic-specific
ure 1 as follows: method teams became theory teams empirical studies
for those students that did not want to carry out Model and tutorials
a study, but wanted to learn more details about a Transformations
specific method or technique, e.g., the systematic re-
view method [13]. Service teams became cross-cutting Predictive Models
topics, e.g., SLRs or surveys (incl. presenting and
teams that build up a specific expertise and consulted
Self-directed learning: working on the selected
Overview: Empirical Methods in Software
theory and practice teams. The 20 teams were con- Engineering (Experiments, Surveys, Data
nected with each other, e.g., a theory team supported Mining, Secondary Studies)
one or many practice teams, and both were supported
by cross-cutting teams. The teacher supervised the Presentation of Project Topics
individual teams as well as the groups of collaborating
writing)
teams. The teams were formed right in the first ses- Topic Selection and Team Building
sion of the course and, thus, the projects became the Initial Project Feedback
main subjects to build the learning experience upon.
3.3.2 Integration in an Advanced Software Weekly Standup (Project Progress)
Engineering Course
A course on Advanced Software Engineering (ASE, Uni- Final Project Presentations
versity of Innsbruck, UI, 2017) implemented the pre-
Final Project Feedback
sented approach as part of a master-level course on
software engineering in which empirical studies com- Project Evaluation
plemented the (technical) software engineering topics.
These technical software engineering topics were or-
ganized around the concept of models in software Figure 4: Overview of the topics and the general orga-
engineering and covered software process models (in- nization of the ASE course.
cluding agile process models), modeling languages
including UML and DSLs, model transformations as
well as predictive models, e.g., for defect prediction.
Figure 4 illustrates the overall organization of the ASE 13 projects were started with two to three students
course showing the introduction parts and the active each. The projects covered four research methods and
project parts. six topics (Table 3).
The overall goal of this course was to teach students
advanced topics in software engineering and to let stu- The concept from Figure 1 was implemented as fol-
dents make the experience of the value that empirical lows: the 13 project teams were formed as practice
studies have to support software engineering activi- teams. Since the empirical studies were designed to
ties. Specifically, the major learning goals of the ASE complement selected topics of the ASE course, no ex-
course were defined as follows: plicit method teams or service teams have been formed.
That is, all practice teams worked on specific topics
• After the course, students know and understand and developed the technical skills and parts of the
different advanced software engineering concepts. required methodological skills themselves. Additional
methodological skills were delivered to the teams by
• After the course, students know the basic empir-
the teachers and guest lectures, who also acted as
ical research methods and know how to utilize
advisors. Teams were formed when the mini-projects
empirical studies in the different software engi-
were assigned to the students.
neering activities.
• After the course, students have shown their abil-
ity to practically apply one research method to a 4 Evaluation
specific software engineering activity, conduct and
report on a small research project. In this section, we present the research design and
evaluation strategy in Section 4.1, the results and a
In total, eight topics have been proposed to the stu- discussion in Section 4.2, and threats to validity in
dents and students could apply for the topics. Finally, Section 4.3.
V. Thurner, O. Radfelder, K. Vosseberg (Hrsg.): SEUH 2019 79
Using Mini-Projects to Teach Empirical Software Engineering
Michael Felderer und Marco Kuhrmann, Uni Insbruck & TU Clausthal
Research Questions course, and the second data collection, again, in the
course’s final evaluation when the mini-projects have
RQ 1 Do course-integrated empirical studies (mini-
projects) help students improving their work ap- been finished. All four questionnaires including a
proach? We aim to study if course-integrated summary are available online2 .
empirical studies (mini-projects) help students Our questionnaire design allows for a 2-staged data
better understand the value of structured scien- collection that helps observing changing student per-
tific work approaches. For this, we investigated ceptions and evaluating the courses over time. The
the following detailed questions: questionnaires share a number of questions to allow
RQ 1.1 Do mini-projects support a better/more effective for comparing courses, the implementations of our
learning? approach, and to qualitatively analyzing the student
RQ 1.2 Do mini-projects support a better understanding feedback. As a learning from the SCM data collection,
of concepts?
the two ASE questionnaires put more emphasis on the
RQ 1.3 What are the perceived learnings of mini-
single phases of the scientific workflow, e.g., by specif-
projects?
ically asking for challenges and difficulties regarding
RQ 2 Do course-integrated mini-projects help students the design of research instruments and conducting
better understand the role of empirical studies? the data collection. Different to the questionnaires
We aim to study the general attitude towards
used in the SCM course, is the ASE questionnaires,
empirical studies, i.e., do students change their
students were asked to provide nicknames (to ensure
attitude once they actively conducted an em-
pirical study. For this, we investigated two anonymity), such that tracking individual students
detailed questions: was possible to evaluate specific ratings and to evalu-
RQ 2.1 Do mini-projects change the attitude towards ate such ratings over time.
empirical studies?
RQ 2.2 Do course-integrated empirical studies studies Analysis Procedures All four questionnaires pro-
help understanding challenges (revealing mis- duce quantitative and qualitative data. For the quanti-
conceptions)? tative analysis, we primarily use descriptive statistics
RQ 3 What are the perceived pros and cons of the to analyze the four measurements individually, over
mini-project approach? We aim to study dis- time per course, and for analyzing both courses. Fur-
/advantages perceived by students that partici- thermore, due to the questionnaire’s evolution, for
pated in mini-projects. the ASE course we could conduct additional inferen-
tial statistical analyses, e.g., hypothesis testing and
Table 4: Overview of the research questions studied. correlation analysis.
To qualitatively analyze the data, we used the free-
text answers provided by the students. For the ASE
4.1 Research Design and Evaluation course, an analysis of general learning and learning
Strategy outcomes was performed using the questions for the
We evaluated our approach in two master-level expected learning outcomes, and the questions about
courses at two universities and by surveying the par- the learning regarding the mini-project topics and
ticipating students. In this section, we present our the way of conducting empirical research (Appendix;
research questions, outline the survey instrument, and variables MP6 –MP8 ). An overall analysis of the course
describe the data collection strategies. as such (in both courses) was performed using the
questions for the courses’ appropriateness, the lectures
Research Questions To evaluate the proposed mini- and exercises, and the perceived relation to practice
project approach, we aim at answering the research (Appendix; variables GC2 –GC4 ; interpreted as school
questions listed in Table 4. Our three top-level re- grades). The coding of the feedback into categories
search questions address the (general) learning ex- was jointly performed by the two authors.
perience, the usefulness of the approach in terms of
improving the understanding of the role of empirical Validity Procedures To mitigate threats and to en-
studies, and the perception concerning the pros and sure the validity of the instrument, we reused a ques-
cons of the approach presented. tionnaire design that was already applied to other
courses and that received an external quality assur-
ance [15, 18]. The original questionnaire design was
Data Collection To collect the data, we (initially)
extended by specific questions to evaluate the suitabil-
developed two online questionnaires for the SCM
ity of the mini-project approach.
course based on Google Forms [14]. The first question-
naire was used in a mid-term evaluation; the second
(extended) questionnaire was used for the final eval- Demographics In total, 68 students were enrolled
uation. While preparing the ASE course, we revised in the SCM course of which 39 students participated
both questionnaires and conducted the first data col- 2 Appendix: https://kuhrmann.files.wordpress.com/2018/
lection before we started the mini-projects in the ASE 11/appendix-draft.pdf
V. Thurner, O. Radfelder, K. Vosseberg (Hrsg.): SEUH 2019 80
Using Mini-Projects to Teach Empirical Software Engineering
Michael Felderer und Marco Kuhrmann, Uni Insbruck & TU Clausthal
Category SCM, n=38 ASE, n=29 RQ 1.2: Support for a better understanding of con-
Mean SD Mean SD cepts A key to provide value to the students is to
Course complexity 2.87 0.66 2.62 0.55 make software engineering concepts better/easier to
Course speed 3.05 0.76 2.83 0.38 understand. For this, students were asked to rate the
Course volume 2.58 0.91 2.10 0.76 statement: “The mini-projects helped me understanding
concepts better”, i.e., whether or not the understanding
Relation to Practice 2.11 0.99 2.34 1.03
of concepts of interest has been improved. In this con-
text, an investigation of the role of empirical studies is
Table 5: Results of the final course evaluation.
provided in Sect. 4.2.2. Again, the majority of the stu-
dents (92% for the SCM course and 69% for the ASE
in the initial evaluation, and 38 in the final evaluation. course; Figure 6) considers the mini-project approach
In the ASE course, 33 students were enrolled, and 29 advantageous for gaining a better understanding of
students participated in both evaluations. From the software engineering concepts.
29 ASE-students, 27 provided a nickname that was
used in the subject-based analyses. Table 5 gives an
ASE 6 14 5 3 1
overview of the general course evaluation. Students
of both courses perceived the course complexity, speed SCM 13 22 2 1
and volume as moderate and see a good relation of 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
the course to practice. Fully Agree Somewhat Agree Indifferent Somewhat Disagree Fully Disagree
4.2 Results and Discussion
This section presents the findings of the evaluation of Figure 6: Results for the improved understanding
our proposed course-integrated mini-project approach. from final questionnaires (SCM: n=38, ASE: n=29).
The following sections present the findings according
to the research questions described in Table 4.
4.2.1 RQ 1: Improved Work Approach RQ 1.3: Perceived Learning The question for the
With RQ1, we aim to study if course-integrated mini- perceived learning is answered using five statements
projects help students understand the value of a struc- (Appendix, MP2.4 –MP2.8 ). In particular, we were inter-
tured scientific work approach. To better structure ested to learn about the perceived impact to the later
the findings, we defined three sub-questions (Table 4), career (“The practiced scientific work approach will help
which we discuss in the following. me in my later career.”) and shareable expertise built
in the course (“I built a specific exercise that I could
share with other teams.”), and a retrospective rating
RQ 1.1: Support for a better learning This sub-
of the group work in the mini-projects (looking back:
question is addressed by the answers to the state-
“contributed to my learning expericence”, “team work
ment: “The mini-projects improve the learning experi-
[..] was good” and “collaboration [..] was good”).
ence”, which was quantitatively analyzed.
Figure 7 shows the aggregated results for the per-
ceived learnings. Approximately 63% of the SCM
ASE 9 8 6 5 1 students and 59% of the ASE students think that the
courses provide take-aways that will have a positive
SCM 17 19 1 1
impact on their later careers. Concerning the share-
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
able expertise, 53% of the SCM students state that
Fully Agree Somewhat Agree Indifferent Somewhat Disagree Fully Disagree
they have obtained knowledge and expertise that they
can share with others; 29% are indifferent. In the ASE
Figure 5: Results for the learning experience from course, even though the course has more “practical”
final questionnaires (SCM: n=38, ASE: n=29). elements, only 41% of the students think that they
built a shareable expertise, but 48% are indifferent.
Figure 5 shows the results (taken from the final Concerning the general perceived learning experi-
evaluation) for both courses and shows that 95% of ence and the teamwork within the project team, the
the SCM-students consider the mini-project approach vast majority of the students rate the courses as good
contributing to an improved perceived learning expe- and very good. However, the cross-team collaboration
rience (3% each rate the teaching format neutral or shows a different picture—notably in the SCM course
less effective than other teaching formats). For the in which interdisciplinary work was enforced by the
ASE course, 59% consider the mini-project approach course design. In the SCM course, 29% of the students
more effective, and 21% each rate it neutral or less considered the cross-team collaboration good to very
effective. In summary, mini-projects contribute to an good, but 55% rated the cross-team collaboration bad
improved perceived learning experience, especially in to very bad. Analyzing this phenomenon, we found
the SCM setting, but also in ASE, where mini-projects the necessity for the different teams to interact with
were only one part of the course. each other to obtain required knowledge from other
V. Thurner, O. Radfelder, K. Vosseberg (Hrsg.): SEUH 2019 81
Using Mini-Projects to Teach Empirical Software Engineering
Michael Felderer und Marco Kuhrmann, Uni Insbruck & TU Clausthal
Category Total SCM ASE
ASE 7 10 6 3 3
Help in later
Topic specific (i.e., mini-project topics) 16 2 14
career
SCM 9 15 9 2 3
Empirical methods (e.g. experiments) 16 6 10
Reporting findings (from studies) 13 13 0
Expertise to share
Importance, meaning (emp. research) 12 2 10
ASE 2 10 14 2 1
Data management (in studies) 11 1 10
Effort (to plan/conduct a study) 10 0 10
SCM 4 16 11 6 1
Technical skills 3 2 1
Soft skills 17 12 5
collaboration
ASE 8 8 7 2 4
Cro ss-team
SCM 3 8 6 14 7 Table 6: Qualitative feedback for the perceived learn-
ings of mini-projects from final questionnaires.
ASE 13 10 2 2 2
Teamwork
SCM 16 13 7 2
of data) and reporting results of empirical studies are
ASE 10 11 4 2 2 highlighted. Students experienced that conducting an
experience
Learning
empirical study causes effort and might be a complex
SCM 13 21 3 1 endeavor (“It is hard to get results, which have a strong
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% meaning”). On the other hand, students also built an
Fully Agree Somewhat Agree Indifferent Somewhat Disagree Fully Disagree understanding of the importance and the meaning of
empirical research in software engineering (“The topic
Figure 7: Results for the perceived leanings from final exists and is very useful if done correctly”). Finally, stu-
questionnaires (SCM: n=38, ASE: n=29). dents hardly report learnings regarding technical skills
(e.g., using LATEX or R). However, manifold learnings
about soft skills (e.g., reviewing techniques) are re-
ported, notably concerning teamwork and cross-team
teams by also trying to keep their own schedule the
collaboration (see also Figure 7).
most disappointing aspect. On the other hand, we
found an “understanding” for this kind of work, which 4.2.2 RQ 2: Improved Understanding
reflects reality in interdisciplinary collaboration and, This research question aims at investigating if students
thus, students eventually considered this a significant built an understanding of the role of empirical stud-
learning. Considering the ASE course, we wanted to ies. Specifically, if students consider empirical studies
learn whether a similar behavior can be observed. As a valuable instrument to complement the technical
Figure 7 shows, cross-team collaboration is still con- software engineering activities in a beneficial way.
sidered a problem, even though the heterogeneity of
the project teams and thus the need to collaborate RQ 2.1: Changed Attitude towards Empirical Stud-
was reduced. ies To learn about the students’ understanding of
An in-depth analysis of the perceived learnings of the value of empirical studies, we asked the students
mini-projects was performed by qualitatively coding whether their view on empirical studies has changed
the responses of the free-form text questions (GC1 : once they actively conducted an empirical study them-
“What was your major take-home asset [..]?”, MP7 : selves (“I like the mini-project part”).
“What did you learn about the topic of your research
project?” and MP8 : “What did you learn about empir-
ASE 5 10 3 9 2
ical research?”). For GC1 , 26 students from the SCM
course provided feedback. In the ASE course, 27 stu- SCM 9 23 3 1 2
dents provided feedback for MP7 and MP8 . In total, 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
we extracted 38 statements from the SCM course and Fully Agree Somewhat Agree Indifferent Somewhat Disagree Fully Disagree
60 statements from the ASE course (for both ques-
tions). The students’ statements were categorized
Figure 8: Results for the changed view on science
based on keywords, and the threshold for building a
from final questionnaires (SCM: n=38, ASE: n=29).
category was set to three references.
Table 6 provides the condensed qualitative feed-
Figure 8 shows that 84% of the SCM students
back on the perceived learnings in eight categories.
changed their view on science and the value of em-
Summarized, topic specific learnings (e.g., application
pirical studies after conducting an empirical study
of DSLs or comments in programming languages) as
themselves. The ASE course provides a different pic-
well as learnings related to the application of empiri-
ture. Still 52% of the students changed their view, but
cal methods (e.g., formulation of research questions
almost 38% did not change their view on science and
or application of specific empirical methods like sur-
empirical studies.
veys) were frequently mentioned. Also, data manage-
ment (i.e., data collection, preparation and analysis
V. Thurner, O. Radfelder, K. Vosseberg (Hrsg.): SEUH 2019 82
Using Mini-Projects to Teach Empirical Software Engineering
Michael Felderer und Marco Kuhrmann, Uni Insbruck & TU Clausthal
RQ 2.2: Challenges of Empirical Studies Besides
Final 3 6 7 12 1
Writing the
the general perception of science and empirical stud-
Report
ies, we are also interested in specific challenges. For Initial 3 3 8 11 2
this, we revised the questionnaire (see Appendix) and
Performing Data
added questions that explicitly address the scientific Final 1 6 11 8 3
Analysis
work process. Initial 1 4 10 8 3 3
Data Collection
Final 6 6 11 5 1
Activity Results p-value
Definition of research questions V = 80 0.5257 Initial 2 3 13 8 1 2
Working with scientific literature V = 55.5 0.4927
Implementation
Final 4 3 13 7 1 1
Design of research instruments V = 68 0.3254
Instrument
Implementation of instrument V = 43 0.7808
Initial 2 5 9 10 3
Data collection V = 135 0.0277
Performing data analysis V = 107 0.9529
Instruments
Final 2 7 6 12 1 1
Research
Design of
Writing the report V = 106 0.3698
Initial 2 4 6 10 4 3
Table 7: Results of the paired Wilcoxon signed-rank
Wo rking with
Final 3 4 9 10 1 2
Literature
Scientific
test for the ASE course (n=27).
Initial 7 8 11 3
To study the perceived challenges students face
Definition of
Final 2 8 6 11 2
Questions
Research
when implementing empirical studies, we asked the
students to rate the different parts of the scientific Initial 2 6 6 12 1 2
work process (Appendix, variables MP5.1 –MP5.7 ; see 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
also Table 7). Furthermore, 27 students enrolled in Straightforward Fairly easy Indifferent Somewhat difficult Very difficult Not applicable
the ASE course provided a nickname, which we used
to investigate challenges over time by evaluating the
Figure 9: Overview of the perceived challenges on the
initial and the final questionnaires. We performed
different scientific work activities in the ASE initial
a Wilcoxon signed-rank test to compare if there are
and final questionnaire (n=29).
significant differences between the initial perception
of the scientific work process and the final one after
the study has been performed. Table 7 shows the test ing the responses of the free-form text questions (GC2 :
results for the various parts of the work process. “Up to 5 things that were good” and GC3 : “Up to 5
The results show that only for the activity data col- things that were bad”).
lection there is a significant difference (p < 0.05). In the SCM course data for GC2 and GC3 was col-
This indicates the perceived difficulties and challenges lected in the initial and final questionnaire. For the
regarding the data collection changed for the partici- ASE course, data was collected in the final evalua-
pating students. A Spearman rank correlation coeffi- tion only. Hence, for the data analysis presented in
cient for the initial and final feedback on the level of the paper at hand, we only consider the data col-
challenges regarding the data collection is low (ρ = lected in both final evaluations. In the SCM course, 29
0.2775), which further indicates that there is only a students provided comments in the final evaluation.
weak positive correlation between between the data Respectively, 21 ASE students provided feedback on
collection challenges perceived initially and the chal- perceived dis-/advantages. In total, we extracted 72
lenges perceived at the end. Figure 9 shows the uncor- pro- and 42 con-statements from the SCM feedback,
related perceived challenges regarding the different and we extracted 54 pro- and 23 con-statements from
activities in the scientific work process (collected in the ASE feedback. Both feedback sets were catego-
the ASE course; initial and final questionnaire). rized and analyzed based on keywords (qualitative
We conclude that the data collection is an activity coding), whereas the threshold for a category was set
in empirical studies for which challenges can be easily to three mentions. Table 8 provides the aggregated
over- and underestimated. When teaching empirical qualitative feedback on the perceived pros and cons of
software engineering it is therefore important to put the mini-project approach in nine categories grouped
special emphasis on the important role of data collec- by SCM, ASE, and in total.
tion and its challenges to prevent students from over-
or underestimating the required effort. General Perception In summary, the mini-project
4.2.3 RQ 3: Perceived Dis-/Advantages approach was seen very positive. For both courses
The third research question aims to investigate the SCM and ASE, the students identified considerably
perceived advantages (“pros”) and disadvantages more pros than cons on the mini-projects. In both set-
(“cons”) of the mini-project approach. For this, we tings, i.e., SCM (a self-contained empirical software
qualitatively analyzed the students’ feedback by cod- engineering course) and ASE (a part of a software
V. Thurner, O. Radfelder, K. Vosseberg (Hrsg.): SEUH 2019 83
Using Mini-Projects to Teach Empirical Software Engineering
Michael Felderer und Marco Kuhrmann, Uni Insbruck & TU Clausthal
Category SCM ASE Total ture (“A little slow lectures”) and the general schedul-
- , - , - , ing and coordination of the lecture with other obli-
Structure, organisation 18 14 9 11 27 25 gations (“During examination time, time overlap with
Knowledge transfer 18 3 3 2 21 5 studying”).
Mini-projects 14 3 6 2 20 5
Research, tech. skills 6 2 8 1 14 3
Effort Finally, the effort caused by the courses was
Topics 6 4 7 0 13 4
Relevance 3 3 9 0 12 3 perceived negative in both settings. Feedback for the
Guest lectures 5 7 6 0 11 7 SCM course says “The volume does not fit well a 5 ECTS
Group work 2 1 6 1 8 2 point course” and, respectively, for the ASE course
Effort 0 5 0 6 0 11 “Sometimes a single task was too big”. This feedback re-
flects a side-effect of project work that typically causes
Table 8: Qualitative feedback on the perceived pros more effort than closed tasks—or even “listen-only”
and cons of mini-projects from final questionnaires. classes. However, such comments motivate a revision
of the projects, e.g., splitting tasks into smaller units
to better support continuous work and to reduce the
engineering course), the obtained research-related perceived effort in a short time frame, but still keep
and technical skills were perceived very positive. The the learning effect of performing empirical research
topics covered in the courses were positively evalu- as we received it also from the SCM comments “in my
ated as well; especially in the ASE course in which opinion learning the scientific method without working
mini-projects were integrated as a part of a software with the methods it’s only knowing about the methods,
engineering course and focused on current (hot) top- not learning them.”
ics in software engineering. Feedback and knowledge
transfer were especially highlighted and considered 4.3 Threats to Validity
positive in the SCM course with its different types of We discuss issues, which may have threatened the
collaborating teams (dedicated practice, method and construct, internal and external validity as well as
service teams), in-depth introductions to empirical measures to mitigate them.
methods, and introductions and exercises in scientific
reading and writing. Also, group work was generally
considered positive, yet, the ASE course with its more Construct Validity The construct validity might be
uniform teams was perceived more advantageous. As threatened by the two different implementations of
already discussed in Sect. 4.2.2, the setting from the the approach presented and the instrument used for
SCM course suffers from the teams’ heterogeneity and its evaluation. Although the two courses differed with
the necessity to establish a cross-team collaboration, respect to the applied integration strategy for mini-
which caused more effort for the project teams. projects, for both courses, we used the same yet tai-
lored questionnaire and combined the responses. To
increase construct validity, we developed the ques-
Guest Lectures In both courses, guest lectures were
tionnaire from an external source [15], which both
given by external researchers. Considering the out-
authors reviewed and evolved. Furthermore, we col-
comes from Table 8, guest lectures in the ASE course
lected and combined quantitative and qualitative data
were considered positive, whereas the guest lectures
to answer and discuss the different research questions.
in the SCM course received a more indifferent rating
(with a slight tendency towards a negative evaluation). Due to the overall setup, the questionnaires differed
We argue that this perception is related to the selec- between the courses. Hence, some analyses like the
tion of the speakers rather than the course setting, yet, individual-based investigation of challenges over time
this remains subject to further investigation. (RQ 2.2) were possible in the ASE course only. Fur-
thermore, analyses regarding perceived learnings of
the students had to be performed using different ques-
Course Organization From the organizational per-
tions (SCM: GC1 , ASE: MP7 and MP8 ; see Appendix),
spective, the courses were perceived indifferent and a
which was handled using a multi-staged coding pro-
relatively high number of positive and negative com-
cess that resulted in common categories.
ments was provided. On the positive side, students
highlighted the organization and structure in general
(“Organization was good”), and, in particular, also Internal Validity The internal validity might be
the course assessment mode (“Fair grading system”) threatened by the rather low number of participants
and the sequence of activities (“The mini-projects were and the participants’ self-reporting, which both might
structured well—good planning of when to do the work, affect the relationship between course-integrated mini-
when to make a presentation and when to turn in the projects and the investigated effects on learning and
paper”). On the negative side, the overall flow was understanding of concepts and the role of empirical
mentioned (“Things started to slow down way too much studies in software engineering. To mitigate these
after the first 5 lectures”) as well as the speed of the lec- threats, we studied the integration of mini-projects
V. Thurner, O. Radfelder, K. Vosseberg (Hrsg.): SEUH 2019 84
Using Mini-Projects to Teach Empirical Software Engineering
Michael Felderer und Marco Kuhrmann, Uni Insbruck & TU Clausthal
in two settings with an acceptable number of partici- increased importance of Data Science, Machine Learn-
pants (SCM: 39 and ASE: 29). We also introduced the ing and Artificial Intelligence courses or even programs
questionnaire to the students to minimize the risk of in general. In all these setting, effective and efficient
misinterpretation. data collection and preparation for further analysis
To triangulate the results of quantitative analysis is essential. Hence, we encourage other teachers to
and to investigate the relationship between course- put special emphasis on all data-related activities, not
integrated mini-projects and their effects holistically, only the data analysis.
we also applied qualitative analysis to analyze the In summary, students provided an overall positive
responses of the participating students. qualitative feedback on the course-integrated mini-
projects as well as on the skills achieved and their
External Validity The external validity might be relevance. This motivates to further explore and dis-
threatened by the issue of the rather low number of seminate the presented approach. However, on the
settings in which the course was performed and eval- downside, the students pointed out the overall flow of
uated. However, we implemented and evaluated the the courses and the perceived high effort to perform
course for each of the two course integration strategies mini-projects, which requires a further refinement of
proposed in Section 3.3, i.e., self-contained empiri- the courses, e.g., by splitting it into smaller tasks of
cal software engineering course and integration in uniform granularity. In future, we will therefore revise
software engineering course. Two researchers from our approach accordingly and further disseminate it.
two different institutions were involved in preparing, We also plan to perform further in-depth analyses of
conducting and evaluating the courses. Furthermore, course artifacts, e.g., the students’ final reports, as
we combined quantitative and qualitative data to get well as replications of the courses.
a broader view on teaching empirical software engi- Finally, this paper presents an approach that has
neering with course-integrated mini-projects. been implemented twice and it also provides initial
The participants’ self-reporting might also affect the data. To get further insights and to improve the data
generalizability of the results. To mitigate this threat, available, we cordially invite other teachers to adapt
we introduced the questionnaire to the students to and integrate our approach into their courses and to
minimize the risk of misinterpretation. Since the pa- share their experiences.
per at hand is an initial study, we consider further
in-depth analyses, e.g., of course artifacts like final References
reports, and replications of the courses as well as the [1] L. W. Anderson and D. R. Krathwohl, editors. A
transfer of the approach presented and its evaluation Taxonomy for Learning, Teaching, and Assessing:
as future work. A Revision of Bloom’s Taxonomy of Educational
Objectives, Abridged Edition. Pearson, 1st edition,
5 Conclusion 2000.
In this paper we presented and evaluated an approach [2] V. Basili, R. Selby, and D. Hutchens. Experi-
to teach empirical software engineering with course- mentation in software engineering. Trans. on
integrated mini-projects. In such mini-projects stu- Software Engineering, 12(7):733–743, 1986.
dents conduct small empirical studies in collaborat-
ing teams within a software engineering course or [3] E. Dale. Audiovisual methods in teaching. Dryden
a research methods course. We illustrated the ap- Press, 3 edition, 1969.
proach by presenting two integration strategies: first
a self-contained course on empirical software engi- [4] J. Dillon. A Review of the Research on Practical
neering given at the University of Southern Denmark Work in School Science. Technical report, King’s
and second as part of an advanced software engi- College, 2008.
neering course given at the University of Innsbruck.
[5] F. Fagerholm, A. S. Guinea, H. Mäenpää, and
Both implementations were complemented by a study
J. Münch. Building blocks for continuous experi-
that showed a positive learning experience and an
mentation. In Proceedings of the 1st International
improved understanding of (empirical) software engi-
Workshop on Rapid Continuous Software Engi-
neering concepts.
neering, RCoSE 2014, pages 26–35, New York,
More than half of the participating students state
NY, USA, 2014. ACM.
as a perceived learning of the course that empirical
studies are helpful for their later careers (systematic [6] F. Fagerholm, M. Kuhrmann, and J. Münch.
and evidence-driven work). In addition, a statistical Guidelines for using empirical studies in soft-
analysis revealed that especially the data collection ware engineering education. PeerJ Computer
activities are underestimated by the students, which Science, 3(e131), September 2017.
allows for future improvements of university courses.
We consider this aspect critical not only for empirical [7] D. Fucci and B. Turhan. A replicated experiment
software engineering in particular, but also due to the on the effectiveness of test-first development. In
V. Thurner, O. Radfelder, K. Vosseberg (Hrsg.): SEUH 2019 85
Using Mini-Projects to Teach Empirical Software Engineering
Michael Felderer und Marco Kuhrmann, Uni Insbruck & TU Clausthal
International Symposium on Empirical Software [18] M. Kuhrmann and J. Münch. When teams go
Engineering and Measurement, pages 103–112. crazy: An environment to experience group dy-
IEEE, Oct 2013. namics in software project management courses.
In International Conference on Software Engineer-
[8] D. Fucci, B. Turhan, and M. Oivo. Impact of pro- ing, ICSE, pages 412–421. ACM, May 2016.
cess conformance on the effects of test-driven
development. In International Symposium on Em- [19] M. Kuhrmann and J. Münch. Enhancing soft-
pirical Software Engineering and Measurement, ware engineering education through experimen-
pages 10:1–10:10. ACM, 2014. tation: An experience report. In 2018 IEEE Inter-
national Conference on Engineering, Technology
[9] V. Garousi, M. Felderer, M. Kuhrmann, and and Innovation (ICE/ITMC), pages 1–9, June
K. Herkiloğlu. What industry wants from 2018.
academia in software testing?: Hearing prac-
titioners’ opinions. In Proceedings of the 21st In- [20] K. Labunets, A. Janes, M. Felderer, and F. Mas-
ternational Conference on Evaluation and Assess- sacci. Teaching predictive modeling to junior
ment in Software Engineering, EASE’17, pages software engineers—seminar format and its eval-
65–69, New York, NY, USA, 2017. ACM. uation: poster. In Proceedings of the 39th In-
ternational Conference on Software Engineering
[10] V. Garousi, M. Felderer, and M. V. Mäntylä. The Companion, pages 339–340. IEEE Press, 2017.
need for multivocal literature reviews in soft-
[21] J. Linåker, S. M. Sulaman, R. M. de Mello, and
ware engineering: complementing systematic
M. Höst. Guidelines for conducting surveys in
literature reviews with grey literature. In Pro-
software engineering. Technical report, Lund
ceedings of the 20th International Conference on
University, January 2015.
Evaluation and Assessment in Software Engineer-
ing, page 26. ACM, 2016. [22] J. Parker. Using laboratory experiments to teach
introductory economics. Working paper, Reed Col-
[11] V. Garousi, M. Felderer, and M. V. Mäntylä. lege, http://academic.reed.edu/economics/
Guidelines for including grey literature and con- parker/ExpBook95.pdf, accessed 2014-10-23.
ducting multivocal literature reviews in software
engineering. Information and Software Technol- [23] K. Petersen, R. Feldt, S. Mujtaba, and M. Matt-
ogy, 2018. son. Systematic mapping studies in software
engineering. In International Conference on Eval-
[12] M. Kasunic. Designing an effective survey. Tech- uation and Assessment in Software Engineering,
nical report, Carnegie Mellon University Pitts- pages 68–77. ACM, 2008.
burgh Software Engineering Institute, 2005.
[24] K. Petersen, S. Vakkalanka, and L. Kuzniarz.
[13] B. A. Kitchenham, D. Budgen, and P. Brereton. Guidelines for conducting systematic mapping
Evidence-Based Software Engineering and System- studies in software engineering: An update. Inf.
atic Reviews. CRC Press, 2015. Softw. Technol., 64:1–18, August 2015.
[14] M. Kuhrmann. Teaching empirical software en- [25] P. Runeson, M. Höst, A. Rainer, and B. Reg-
gineering using expert teams. In SEUH, pages nell. Case Study Research in Software Engineer-
20–31, 2017. ing: Guidelines and Examples. John Wiley &
Sons, 2012.
[15] M. Kuhrmann, D. M. Fernández, and J. Münch.
Teaching software process modeling. In Interna- [26] C. Wohlin. Empirical software engineering:
tional Conference on Software Engineering, pages Teaching methods and conducting studies. In
1138–1147, 2013. Proceedings of the International Workshop on Em-
pirical Software Engineering Issues: Critical As-
[16] M. Kuhrmann, D. Mendez Fernández, and sessment and Future Directions, volume 4336 of
M. Daneva. On the pragmatic design of lit- LNCS, pages 135–142. Springer, 2007.
erature studies in software engineering: an
[27] C. Wohlin, P. Runeson, M. Höst, M. C. Ohlsson,
experience-based guideline. Empirical Software
B. Regnell, and A. Wesslén. Experimentation in
Engineering, 22(6):2852–2891, Dec 2017.
software engineering. Springer Science & Busi-
[17] M. Kuhrmann and J. Münch. Distributed soft- ness Media, 2012.
ware development with one hand tied behind [28] H. Zhang, M. A. Babar, and P. Tell. Identifying
the back: A course unit to experience the role relevant studies in software engineering. Infor-
of communication in gsd. In 1st Workshop on mation and Software Technology, 53(6):625–637,
Global Software Engineering Education (in con- 2011.
junction with ICGSE’2016). IEEE, 2016.
V. Thurner, O. Radfelder, K. Vosseberg (Hrsg.): SEUH 2019 86