Empirical studies have become a central element of software engineering research and practice. Yet, teaching the instruments of empirical software engineering is challenging, since students need to understand the theory of the scientific method and also have to develop an understanding of the application of those instruments and their benefits. In this paper, we present and evaluate an approach to teach empirical software engineering with course-integrated mini-projects. In mini-projects, students conduct small empirical studies, e.g., surveys, literature reviews, controlled experiments, and data mining studies in collaborating teams. We present the approach through two implementations at two universities as a self-contained course on empirical software engineering and as part of an advanced software engineering course; with 101 graduate students in total. Our evaluation shows a positive learning experience and an improved understanding of the concepts taught. More than a half of the students consider empirical studies helpful for their later careers. Finally, a qualitative coding and a statistical analysis showed the proposed approach beneficial, but also revealed challenges of the scientific work process, e.g., data collection activities that were underestimated by the students.
Empirical software engineering aims at making
software engineering claims measurable, i.e., to analyze
and understand phenomena in software engineering
and to evaluate software engineering approaches and
solutions, and to ground decision-making processes
in evidence. For this, an extensive portfolio of
instruments for empirical software engineering has been
developed. For instance, Wohlin et al. [
Conducting empirical studies is challenging and
requires careful preparation and a disciplined work
approach. Quite often, students consider empirical
studies of little to no help when it comes to software
development and project work, since the relation to
actual development tasks is not obvious. Yet, many
of today’s applications rely on data, e.g., machine
learning systems like text and speech recognition, IoT
devices, and autonomous cars. Empirical methods
as such are about data analysis and, thus, provide
a suitable approach to teach data analysis—or data
engineering in general—that is a core competence
in data-intensive applications. Furthermore, modern
software development paradigms, such as DevOps
including continuous integration and deployment,
utilize data, e.g., to analyze a system’s performance, to
predict defects, and to make informed decisions in
the development process as practiced in continuous
experimentation [
In this paper, we present and evaluate the concept of course-integrated mini-projects to teach empirical software engineering instruments. Our approach helps students learn how to conduct empirical studies and understand the instruments and challenges coming along with such studies. Collaborating project teams conducting small empirical studies form the basis of our approach. We implemented the approach in two courses at the University of Southern Denmark (2016, 68 students) and University of Innsbruck (2017, 33 students). Mini-projects allow students to learn empirical instruments by practically applying them. Our evaluation shows a positive learning experience and an improved understanding of (empirical) software engineering concepts. More than half of the students perceive empirical studies helpful for their later careers. Our evaluation also shows that notably data collection activities (e.g., for surveys and experiments) are underestimated by the students. Our findings thus lay the foundation for improving research-oriented courses that require data and data analysis.
The remainder of the paper is organized as follows: Section 2 gives an overview of background and related work. Section 3 describes the mini-project approach, and Section 4 presents the approach’s evaluation based on two implementations at the University of Southern Denmark and the University of Innsbruck respectively. We conclude the paper and discuss future work in Section 5. 2
Using empirical studies in software engineering
education is not a new idea [
In this paper, we present an approach that considers
empirical studies major subjects of a course and that
uses such studies as teaching tool. Referring to
established learning models such as Bloom’s Taxonomy of
Learning [
We contribute a generalized concept grounded in
[
We present the course-integrated mini-projects
approach in Section 3.1. The presentation includes the
description of the team setups, project and task
descriptions, and examples for which we present details
in Section 3.2. Section 3.3 demonstrates two
integration strategies: the first integration strategy is a
self-contained course on empirical software
engineering [
Figure 1 shows the general organization model for the
mini-project approach. A MiniProject has a Topic, a
Schedule, and optional Reference Literature and
Input Data. It is always carried out using at least one
Method, e.g., an experiment [
1 Advisory Team 1 1..* Advisor
Mini Project
- Topic [
1..*
2..*
1
1..* 1 0..* Project Team 1 0..*
joint research I shared S research design
Advisory Teams An advisory team bundles all advisors involved in a specific mini-project—usually one or two persons. An Advisor is either the teacher of the course or an external advisor, such as an external The “normal” way of doing a mini-project is the isolated way of working. Isolated means that a project team has a self-contained task that can be worked on without any interaction with other teams.
This style is applied if project teams collaboratively work on a joint (research) project. A complex project is broken down into a number of smaller projects. Project teams thus have to be coordinated in terms of task distribution, scheduling, and result synthesis.
This style is applied if project teams
competitively work on the same (research) topic. Two
or more teams are assigned the same task;
team-specific methods can be varied and
results can be compared. That is, this style helps
conducting controlled experiments or
implementing independently conducted studies.
project topic sponsor [
Project Teams A project is composed of all students working on a specific problem. Project teams can interact with each other. We distinguish the three interaction styles isolated, joint, and shared, which are explained in Table 1. Furthermore, we distinguish three types of project teams according to the type of task they are working on: a Practice Team performs an “active” task, e.g., a development task or a research task. A Method Team deals with methodological expertise, i.e., it develops competencies regarding specific (scientific) methods and offers “consultancy services” in terms of applying a specific method “right” to practice teams. Finally, a Service Team develops skills in more general topics, such as data analysis or presentation, and offers respective “services” to other teams—practice teams and method teams alike.
Every mini-project is supposed to produce at least one Result. In this section, we provide a blueprint for a 1-page project- and task description, which also illustrates the manifestation of the different attributes of the class Mini Project (Figure 1).
For every project, a description that includes tasks, dates, and expected results is necessary. Table 2 provides a summary and a description of task-description items that we consider relevant. The work description requires special attention as it comprises the detailed activity list, input material, and the description of expected results. The expected results are speciItem
Description Metadata This section contains all information relevant to a task, e.g., hand-in date.
Title A project needs a telling title and an ID Context This section briefly describes the context of the project and provides a short summary of the basic tasks. Recommendation: the context section should be treated as an abstract, such that it can be used as a teaser and a small piece of information, e.g., used in a course management system.
Work The detailed work description contains at Description least: Schedule Related Projects Literature 1. A detailed task list. 2. A list of input/reference material. 3. A list of deliverables to be shipped.
Note: The level of detail depends on the actual task, i.e., for an “explorative” task, the description needs to be more open while a specific development task requires a more detailed task description.
The basic schedule lists all deadlines and the respectively expected results.
Our concept allows for collaborative and competitive work (Figure 1 and Table 1). If such a collaborative/competitive work style is implemented, this section provides the information about the other teams. If method or service teams provide useful services to a project, such teams are referred here too.
This section lists selected reference
literature relevant to a project.
fied right here; alternatively, a separate catalog of
results has to be provided, e.g., including templates and
mandatory/recommended outlines. Relevant types
for project outcomes are, e.g., (research) data1,
essays or reports, presentations, tutorials, and software.
The second important item of the task description is
the list of related projects. For instance, if a task is a
collaborative task, this list refers to all related projects
that contribute to the overall project goal.
Furthermore, this list also refers those method and service
teams that provide useful support, e.g., if the project
is concerned with developing and conducting a survey,
this list can refer to a method team that is focused
on the theoretical aspects of survey research. Finally,
the task description can also be used to develop a
checklist, which is used for the final submission. This
checklist helps students to check if their delivery
package is complete, and it helps teachers validating the
delivery and grade its components. Figure 2 shows
1Recommendation: If students conduct a research task, analyzed
data that is necessary for the project documentation must always
be complemented with the original raw data.
a practically used example of a task description as
described in Table 2. This task description is taken
from the course given at University of Southern
Denmark and describes a survey research task, which
was performed collaboratively with two external
researchers [
We describe two implementation strategies for the presented approach using two graduate courses. For these implementations, we provide a short summary of the respective courses and their learning goals, and we provide an overview of the projects implemented in the respective courses (Table 3). Furthermore, this section lays the foundation for the approach’s evaluation, which is presented in Section 4.
A course on the Scientific Method (SCM, University of Southern Denmark, SDU, 2016) implemented the presented approach as a self-contained master-level course. Figure 3 illustrates the overall organization of the SCM course showing the introduction parts and the active learning/project parts.
The goal of the SCM course was to teach empirical
software engineering as main subject by letting the
students perform small studies themselves [
SCM Theory (tutorial) Experiment Survey Systematic Review Mapping Study Simulation Data Mining Study cally, the major learning goals of the SCM course were defined as follows:
After the course, students know the basic terminology and the key concepts of the scientific method. After the course, students know and understand the most important empirical research methods. After the course, students have shown their ability to practically apply one research method, conduct and report on a small research project. In total, 30 research topics were presented to the students. According to their preferences, students could apply for up to three topics, which built the foundation for the final team setup. Finally, 20 projects were started with two to three students each. Besides the theoretical topics, five research methods were covered by the projects (Table 3).
The SCM course implements the concept from
Figure 1 as follows: method teams became theory teams
for those students that did not want to carry out
a study, but wanted to learn more details about a
specific method or technique, e.g., the systematic
review method [
A course on Advanced Software Engineering (ASE, University of Innsbruck, UI, 2017) implemented the presented approach as part of a master-level course on software engineering in which empirical studies complemented the (technical) software engineering topics. These technical software engineering topics were organized around the concept of models in software engineering and covered software process models (including agile process models), modeling languages including UML and DSLs, model transformations as well as predictive models, e.g., for defect prediction. Figure 4 illustrates the overall organization of the ASE course showing the introduction parts and the active project parts.
The overall goal of this course was to teach students advanced topics in software engineering and to let students make the experience of the value that empirical studies have to support software engineering activities. Specifically, the major learning goals of the ASE course were defined as follows:
After the course, students know and understand different advanced software engineering concepts. After the course, students know the basic empirical research methods and know how to utilize empirical studies in the different software engineering activities.
After the course, students have shown their ability to practically apply one research method to a specific software engineering activity, conduct and report on a small research project.
In total, eight topics have been proposed to the students and students could apply for the topics. Finally, g n i r e e n i g n reE res taw tcu foS eL l a c i n h c e T
d ted an c g le in e t es sen th re irkngon il.(csnp ) :ow rvye iitg
n irnng rsuo rw lea sR d L tec ,.S re .g li-fed ,isec S tpo Complementing Material/Activities:
Topic-specific empirical studies
and tutorials Overview Software
Engineering Empirical Methods in Software Engineering
Software Process
Models Modeling Languages
Model Transformations Predictive Models Overview: Empirical Methods in Software Engineering (Experiments, Surveys, Data
Mining, Secondary Studies) Presentation of Project Topics Topic Selection and Team Building
Initial Project Feedback Weekly Standup (Project Progress)
Final Project Presentations
Final Project Feedback
Project Evaluation 13 projects were started with two to three students each. The projects covered four research methods and six topics (Table 3).
The concept from Figure 1 was implemented as follows: the 13 project teams were formed as practice teams. Since the empirical studies were designed to complement selected topics of the ASE course, no explicit method teams or service teams have been formed. That is, all practice teams worked on specific topics and developed the technical skills and parts of the required methodological skills themselves. Additional methodological skills were delivered to the teams by the teachers and guest lectures, who also acted as advisors. Teams were formed when the mini-projects were assigned to the students. 4
In this section, we present the research design and evaluation strategy in Section 4.1, the results and a discussion in Section 4.2, and threats to validity in Section 4.3. Research Questions Do course-integrated mini-projects help students better understand the role of empirical studies? We aim to study the general attitude towards empirical studies, i.e., do students change their attitude once they actively conducted an empirical study. For this, we investigated two detailed questions: Do mini-projects change the attitude towards empirical studies? Do course-integrated empirical studies studies help understanding challenges (revealing misconceptions)? What are the perceived pros and cons of the mini-project approach? We aim to study dis/advantages perceived by students that participated in mini-projects.
We evaluated our approach in two master-level courses at two universities and by surveying the participating students. In this section, we present our research questions, outline the survey instrument, and describe the data collection strategies.
Research Questions To evaluate the proposed miniproject approach, we aim at answering the research questions listed in Table 4. Our three top-level research questions address the (general) learning experience, the usefulness of the approach in terms of improving the understanding of the role of empirical studies, and the perception concerning the pros and cons of the approach presented.
Data Collection To collect the data, we (initially)
developed two online questionnaires for the SCM
course based on Google Forms [
Our questionnaire design allows for a 2-staged data collection that helps observing changing student perceptions and evaluating the courses over time. The questionnaires share a number of questions to allow for comparing courses, the implementations of our approach, and to qualitatively analyzing the student feedback. As a learning from the SCM data collection, the two ASE questionnaires put more emphasis on the single phases of the scientific workflow, e.g., by specifically asking for challenges and difficulties regarding the design of research instruments and conducting the data collection. Different to the questionnaires used in the SCM course, is the ASE questionnaires, students were asked to provide nicknames (to ensure anonymity), such that tracking individual students was possible to evaluate specific ratings and to evaluate such ratings over time.
Analysis Procedures All four questionnaires produce quantitative and qualitative data. For the quantitative analysis, we primarily use descriptive statistics to analyze the four measurements individually, over time per course, and for analyzing both courses. Furthermore, due to the questionnaire’s evolution, for the ASE course we could conduct additional inferential statistical analyses, e.g., hypothesis testing and correlation analysis.
To qualitatively analyze the data, we used the freetext answers provided by the students. For the ASE course, an analysis of general learning and learning outcomes was performed using the questions for the expected learning outcomes, and the questions about the learning regarding the mini-project topics and the way of conducting empirical research (Appendix; variables MP6–MP8). An overall analysis of the course as such (in both courses) was performed using the questions for the courses’ appropriateness, the lectures and exercises, and the perceived relation to practice (Appendix; variables GC2–GC4; interpreted as school grades). The coding of the feedback into categories was jointly performed by the two authors.
Validity Procedures To mitigate threats and to
ensure the validity of the instrument, we reused a
questionnaire design that was already applied to other
courses and that received an external quality
assurance [
Demographics In total, 68 students were enrolled in the SCM course of which 39 students participated 2Appendix: https://kuhrmann.files.wordpress.com/2018/ 11/appendix-draft.pdf in the initial evaluation, and 38 in the final evaluation. In the ASE course, 33 students were enrolled, and 29 students participated in both evaluations. From the 29 ASE-students, 27 provided a nickname that was used in the subject-based analyses. Table 5 gives an overview of the general course evaluation. Students of both courses perceived the course complexity, speed and volume as moderate and see a good relation of the course to practice.
This section presents the findings of the evaluation of our proposed course-integrated mini-project approach. The following sections present the findings according to the research questions described in Table 4.
With RQ1, we aim to study if course-integrated miniprojects help students understand the value of a structured scientific work approach. To better structure the findings, we defined three sub-questions (Table 4), which we discuss in the following.
RQ 1.1: Support for a better learning This subquestion is addressed by the answers to the statement: “The mini-projects improve the learning experience”, which was quantitatively analyzed. 20% 30% 40% 50% 60% 70% 80% 90% 100%
Somewhat Agree Indifferent Somewhat Disagree Fully Disagree
Figure 5 shows the results (taken from the final evaluation) for both courses and shows that 95% of the SCM-students consider the mini-project approach contributing to an improved perceived learning experience (3% each rate the teaching format neutral or less effective than other teaching formats). For the ASE course, 59% consider the mini-project approach more effective, and 21% each rate it neutral or less effective. In summary, mini-projects contribute to an improved perceived learning experience, especially in the SCM setting, but also in ASE, where mini-projects were only one part of the course. RQ 1.2: Support for a better understanding of concepts A key to provide value to the students is to make software engineering concepts better/easier to understand. For this, students were asked to rate the statement: “The mini-projects helped me understanding concepts better”, i.e., whether or not the understanding of concepts of interest has been improved. In this context, an investigation of the role of empirical studies is provided in Sect. 4.2.2. Again, the majority of the students (92% for the SCM course and 69% for the ASE course; Figure 6) considers the mini-project approach advantageous for gaining a better understanding of software engineering concepts.
ASE SCM 6 13 14
5 22
10% Fully Agree 20% 30% 40% 50% 60% 70% 80% 90% 100%
Somewhat Agree Indifferent Somewhat Disagree Fully Disagree RQ 1.3: Perceived Learning The question for the perceived learning is answered using five statements (Appendix, MP2:4–MP2:8). In particular, we were interested to learn about the perceived impact to the later career (“The practiced scientific work approach will help me in my later career.”) and shareable expertise built in the course (“I built a specific exercise that I could share with other teams.”), and a retrospective rating of the group work in the mini-projects (looking back: “contributed to my learning expericence”, “team work [..] was good” and “collaboration [..] was good”).
Figure 7 shows the aggregated results for the perceived learnings. Approximately 63% of the SCM students and 59% of the ASE students think that the courses provide take-aways that will have a positive impact on their later careers. Concerning the shareable expertise, 53% of the SCM students state that they have obtained knowledge and expertise that they can share with others; 29% are indifferent. In the ASE course, even though the course has more “practical” elements, only 41% of the students think that they built a shareable expertise, but 48% are indifferent.
Concerning the general perceived learning experience and the teamwork within the project team, the vast majority of the students rate the courses as good and very good. However, the cross-team collaboration shows a different picture—notably in the SCM course in which interdisciplinary work was enforced by the course design. In the SCM course, 29% of the students considered the cross-team collaboration good to very good, but 55% rated the cross-team collaboration bad to very bad. Analyzing this phenomenon, we found the necessity for the different teams to interact with each other to obtain required knowledge from other lta re ASE r e in re lepH ca SCM e r sha ASE 2 o t e s it rep SCM x
E am iton ASE - ab te r s o rsoC llcoa SCM 3 teams by also trying to keep their own schedule the most disappointing aspect. On the other hand, we found an “understanding” for this kind of work, which reflects reality in interdisciplinary collaboration and, thus, students eventually considered this a significant learning. Considering the ASE course, we wanted to learn whether a similar behavior can be observed. As Figure 7 shows, cross-team collaboration is still considered a problem, even though the heterogeneity of the project teams and thus the need to collaborate was reduced.
An in-depth analysis of the perceived learnings of mini-projects was performed by qualitatively coding the responses of the free-form text questions (GC1: “What was your major take-home asset [..]?”, MP7: “What did you learn about the topic of your research project?” and MP8: “What did you learn about empirical research?”). For GC1, 26 students from the SCM course provided feedback. In the ASE course, 27 students provided feedback for MP7 and MP8. In total, we extracted 38 statements from the SCM course and 60 statements from the ASE course (for both questions). The students’ statements were categorized based on keywords, and the threshold for building a category was set to three references.
Table 6 provides the condensed qualitative feedback on the perceived learnings in eight categories. Summarized, topic specific learnings (e.g., application of DSLs or comments in programming languages) as well as learnings related to the application of empirical methods (e.g., formulation of research questions or application of specific empirical methods like surveys) were frequently mentioned. Also, data management (i.e., data collection, preparation and analysis Category Topic specific (i.e., mini-project topics) Empirical methods (e.g. experiments) Reporting findings (from studies) Importance, meaning (emp. research) Data management (in studies) Effort (to plan/conduct a study) Technical skills Soft skills Total
SCM
ASE of data) and reporting results of empirical studies are highlighted. Students experienced that conducting an empirical study causes effort and might be a complex endeavor (“It is hard to get results, which have a strong meaning”). On the other hand, students also built an understanding of the importance and the meaning of empirical research in software engineering (“The topic exists and is very useful if done correctly”). Finally, students hardly report learnings regarding technical skills (e.g., using LATEX or R). However, manifold learnings about soft skills (e.g., reviewing techniques) are reported, notably concerning teamwork and cross-team collaboration (see also Figure 7).
This research question aims at investigating if students built an understanding of the role of empirical studies. Specifically, if students consider empirical studies a valuable instrument to complement the technical software engineering activities in a beneficial way. RQ 2.1: Changed Attitude towards Empirical Studies To learn about the students’ understanding of the value of empirical studies, we asked the students whether their view on empirical studies has changed once they actively conducted an empirical study themselves (“I like the mini-project part”).
ASE SCM 5 9 10 9
2 23 3
10% Fully Agree 20% 30% 40% 50% 60% 70% 80% 90% 100%
Somewhat Agree Indifferent Somewhat Disagree Fully Disagree
Figure 8 shows that 84% of the SCM students changed their view on science and the value of empirical studies after conducting an empirical study themselves. The ASE course provides a different picture. Still 52% of the students changed their view, but almost 38% did not change their view on science and empirical studies. RQ 2.2: Challenges of Empirical Studies Besides the general perception of science and empirical studies, we are also interested in specific challenges. For this, we revised the questionnaire (see Appendix) and added questions that explicitly address the scientific work process.
Activity Definition of research questions Working with scientific literature Design of research instruments Implementation of instrument Data collection Performing data analysis Writing the report Results V = 80 V = 55.5 V = 68 V = 43 V = 135 V = 107 V = 106 p-value
To study the perceived challenges students face when implementing empirical studies, we asked the students to rate the different parts of the scientific work process (Appendix, variables MP5:1–MP5:7; see also Table 7). Furthermore, 27 students enrolled in the ASE course provided a nickname, which we used to investigate challenges over time by evaluating the initial and the final questionnaires. We performed a Wilcoxon signed-rank test to compare if there are significant differences between the initial perception of the scientific work process and the final one after the study has been performed. Table 7 shows the test results for the various parts of the work process.
The results show that only for the activity data collection there is a significant difference (p < 0:05). This indicates the perceived difficulties and challenges regarding the data collection changed for the participating students. A Spearman rank correlation coefficient for the initial and final feedback on the level of challenges regarding the data collection is low ( = 0.2775), which further indicates that there is only a weak positive correlation between between the data collection challenges perceived initially and the challenges perceived at the end. Figure 9 shows the uncorrelated perceived challenges regarding the different activities in the scientific work process (collected in the ASE course; initial and final questionnaire).
We conclude that the data collection is an activity in empirical studies for which challenges can be easily over- and underestimated. When teaching empirical software engineering it is therefore important to put special emphasis on the important role of data collection and its challenges to prevent students from overor underestimating the required effort.
The third research question aims to investigate the perceived advantages (“pros”) and disadvantages (“cons”) of the mini-project approach. For this, we qualitatively analyzed the students’ feedback by codthe tr Final 3 tng epo ii rW R Initial 3 a tgD iss Final 1 a in ly frrom anA Initial 1 4 e P ing the responses of the free-form text questions (GC2: “Up to 5 things that were good” and GC3: “Up to 5 things that were bad”).
In the SCM course data for GC2 and GC3 was collected in the initial and final questionnaire. For the ASE course, data was collected in the final evaluation only. Hence, for the data analysis presented in the paper at hand, we only consider the data collected in both final evaluations. In the SCM course, 29 students provided comments in the final evaluation. Respectively, 21 ASE students provided feedback on perceived dis-/advantages. In total, we extracted 72 pro- and 42 con-statements from the SCM feedback, and we extracted 54 pro- and 23 con-statements from the ASE feedback. Both feedback sets were categorized and analyzed based on keywords (qualitative coding), whereas the threshold for a category was set to three mentions. Table 8 provides the aggregated qualitative feedback on the perceived pros and cons of the mini-project approach in nine categories grouped by SCM, ASE, and in total.
General Perception In summary, the mini-project approach was seen very positive. For both courses SCM and ASE, the students identified considerably more pros than cons on the mini-projects. In both settings, i.e., SCM (a self-contained empirical software engineering course) and ASE (a part of a software , engineering course), the obtained research-related and technical skills were perceived very positive. The topics covered in the courses were positively evaluated as well; especially in the ASE course in which mini-projects were integrated as a part of a software engineering course and focused on current (hot) topics in software engineering. Feedback and knowledge transfer were especially highlighted and considered positive in the SCM course with its different types of collaborating teams (dedicated practice, method and service teams), in-depth introductions to empirical methods, and introductions and exercises in scientific reading and writing. Also, group work was generally considered positive, yet, the ASE course with its more uniform teams was perceived more advantageous. As already discussed in Sect. 4.2.2, the setting from the SCM course suffers from the teams’ heterogeneity and the necessity to establish a cross-team collaboration, which caused more effort for the project teams. Guest Lectures In both courses, guest lectures were given by external researchers. Considering the outcomes from Table 8, guest lectures in the ASE course were considered positive, whereas the guest lectures in the SCM course received a more indifferent rating (with a slight tendency towards a negative evaluation). We argue that this perception is related to the selection of the speakers rather than the course setting, yet, this remains subject to further investigation. Course Organization From the organizational perspective, the courses were perceived indifferent and a relatively high number of positive and negative comments was provided. On the positive side, students highlighted the organization and structure in general (“Organization was good”), and, in particular, also the course assessment mode (“Fair grading system”) and the sequence of activities (“The mini-projects were structured well—good planning of when to do the work, when to make a presentation and when to turn in the paper”). On the negative side, the overall flow was mentioned (“Things started to slow down way too much after the first 5 lectures”) as well as the speed of the lecture (“A little slow lectures”) and the general scheduling and coordination of the lecture with other obligations (“During examination time, time overlap with studying”).
Effort Finally, the effort caused by the courses was perceived negative in both settings. Feedback for the SCM course says “The volume does not fit well a 5 ECTS point course” and, respectively, for the ASE course “Sometimes a single task was too big”. This feedback reflects a side-effect of project work that typically causes more effort than closed tasks—or even “listen-only” classes. However, such comments motivate a revision of the projects, e.g., splitting tasks into smaller units to better support continuous work and to reduce the perceived effort in a short time frame, but still keep the learning effect of performing empirical research as we received it also from the SCM comments “in my opinion learning the scientific method without working with the methods it’s only knowing about the methods, not learning them.” 4.3
We discuss issues, which may have threatened the construct, internal and external validity as well as measures to mitigate them.
Construct Validity The construct validity might be
threatened by the two different implementations of
the approach presented and the instrument used for
its evaluation. Although the two courses differed with
respect to the applied integration strategy for
miniprojects, for both courses, we used the same yet
tailored questionnaire and combined the responses. To
increase construct validity, we developed the
questionnaire from an external source [
Due to the overall setup, the questionnaires differed between the courses. Hence, some analyses like the individual-based investigation of challenges over time (RQ 2.2) were possible in the ASE course only. Furthermore, analyses regarding perceived learnings of the students had to be performed using different questions (SCM: GC1, ASE: MP7 and MP8; see Appendix), which was handled using a multi-staged coding process that resulted in common categories.
Internal Validity The internal validity might be threatened by the rather low number of participants and the participants’ self-reporting, which both might affect the relationship between course-integrated miniprojects and the investigated effects on learning and understanding of concepts and the role of empirical studies in software engineering. To mitigate these threats, we studied the integration of mini-projects in two settings with an acceptable number of participants (SCM: 39 and ASE: 29). We also introduced the questionnaire to the students to minimize the risk of misinterpretation.
To triangulate the results of quantitative analysis and to investigate the relationship between courseintegrated mini-projects and their effects holistically, we also applied qualitative analysis to analyze the responses of the participating students.
External Validity The external validity might be threatened by the issue of the rather low number of settings in which the course was performed and evaluated. However, we implemented and evaluated the course for each of the two course integration strategies proposed in Section 3.3, i.e., self-contained empirical software engineering course and integration in software engineering course. Two researchers from two different institutions were involved in preparing, conducting and evaluating the courses. Furthermore, we combined quantitative and qualitative data to get a broader view on teaching empirical software engineering with course-integrated mini-projects.
The participants’ self-reporting might also affect the generalizability of the results. To mitigate this threat, we introduced the questionnaire to the students to minimize the risk of misinterpretation. Since the paper at hand is an initial study, we consider further in-depth analyses, e.g., of course artifacts like final reports, and replications of the courses as well as the transfer of the approach presented and its evaluation as future work. 5
In this paper we presented and evaluated an approach to teach empirical software engineering with courseintegrated mini-projects. In such mini-projects students conduct small empirical studies in collaborating teams within a software engineering course or a research methods course. We illustrated the approach by presenting two integration strategies: first a self-contained course on empirical software engineering given at the University of Southern Denmark and second as part of an advanced software engineering course given at the University of Innsbruck. Both implementations were complemented by a study that showed a positive learning experience and an improved understanding of (empirical) software engineering concepts.
More than half of the participating students state as a perceived learning of the course that empirical studies are helpful for their later careers (systematic and evidence-driven work). In addition, a statistical analysis revealed that especially the data collection activities are underestimated by the students, which allows for future improvements of university courses. We consider this aspect critical not only for empirical software engineering in particular, but also due to the increased importance of Data Science, Machine Learning and Artificial Intelligence courses or even programs in general. In all these setting, effective and efficient data collection and preparation for further analysis is essential. Hence, we encourage other teachers to put special emphasis on all data-related activities, not only the data analysis.
In summary, students provided an overall positive qualitative feedback on the course-integrated miniprojects as well as on the skills achieved and their relevance. This motivates to further explore and disseminate the presented approach. However, on the downside, the students pointed out the overall flow of the courses and the perceived high effort to perform mini-projects, which requires a further refinement of the courses, e.g., by splitting it into smaller tasks of uniform granularity. In future, we will therefore revise our approach accordingly and further disseminate it. We also plan to perform further in-depth analyses of course artifacts, e.g., the students’ final reports, as well as replications of the courses.
Finally, this paper presents an approach that has been implemented twice and it also provides initial data. To get further insights and to improve the data available, we cordially invite other teachers to adapt and integrate our approach into their courses and to share their experiences. [19] M. Kuhrmann and J. Münch. Enhancing software engineering education through experimentation: An experience report. In 2018 IEEE International Conference on Engineering, Technology and Innovation (ICE/ITMC), pages 1–9, June 2018.