- An empirical software engineering (ESE) PhD thesis has some special features, which makes it slightly different from a thesis any other different field. One of the differences between the two is the intensive use of empirical studies in an ESE dissertation. This talk starts by giving students advice on what makes a good ESE PhD thesis in the form of a list of do's and don'ts. The keynote later discusses what different empirical studies can be used (surveys, case studies and experiments). Finally, it focuses on one specific type of empirical study: controlled experiments. Experimentation is a risky business, and software engineering (SE) has some special features, leading to some experimentation issues being conceived of differently than in other disciplines. Some advice is given on how to analyse SE experiments.
I. WHAT IS AN EMPIRICAL SOFTWARE ENGINEERING THESIS?
A few years ago, PhD theses did not use to include empirical validation. However, times are changing, and, nowadays, any PhD thesis must include at least some kind of empirical validation.
This does not mean, however, that any PhD thesis with an empirical component is an empirical software engineering (ESE) thesis. The key characteristic of ESE theses is that they have a major empirical component.
There are different types of ESE PhD theses. There is no formal typology, but, if we look at the ESE PhD theses written over the last 20 years, two main types stand out: 1) theses gathering knowledge about a specific topic by means of empirical studies, and 2) theses proposing methodological advances in ESE.
Examples of such theses are (in chronological order):
Seaman [
Shull [
Thelin [
Copyright © 2015 for this paper by its authors. Copying permitted for private and academic purposes.
Carver [
The thesis may, in some cases, take in methodological
aspects. For example, Seaman addresses the problem of how to
analyse qualitative data. Shull tackles the problem of
synthesizing the results of the different studies run. In other
cases, techniques that have not been used in SE before are
applied. For example, Carver uses grounded theory [
Examples of such theses are (in chronological order):
Daly [
Ciolkowski [
Solari [
Gómez [
In all cases, empirical studies are run (or used) as a means to develop and validate the research performed.
II. DO’S AND DON’TS
Irrespective of the thesis type, some general guidelines can be established around three key issues that a PhD thesis should address: tackled problem, research method used and publication.
the problem.
DO clearly specify what problem you are tackling.
DON´T think that you believing that the problem is important is enough.
DO objectively assess the importance of the problem so that you can establish that the problem exists and that it is important. There are several ways to do this. One way is by citing other authors that state that the problem is important. Another is by citing numbers taken from a reliable source (for example, “the testing process needs improvement as the process in place fails to detect X% of the defects before the software is delivered to the users”).
DON´T take it for granted that people will trust you when you say that the problem has not yet been solved. DO demonstrate that your work contributes to the advancement of the state of the art/practice (systematic literature reviews might be helpful here).
• • • • • • • • • • • • •
DON´T postpone publication until you have finished your PhD thesis. Although this was the standard approach years ago, it does not work like that anymore. Some universities require you to have published at least a conference or journal paper before the defence of your thesis.
DO try to publish results as early as possible (the state of the art could be a good choice). Of course, this does not mean that you should publish non-conclusive results.
DON´T think that publishing will be a waste of time that might be better spent on advancing in your research.
DO consider other options for writing your thesis. Some universities accept PhD thesis formats other than the traditional dissertation. For example, each thesis chapter is styled as a paper. This will speed up the process of writing your thesis and will be an incentive for publishing.
There are different types of empirical studies [
A survey is a method for collecting information from or
about people to describe, compare or explain their
knowledge, attitudes and behaviour [
A case study is an empirical study that draws on
multiple sources of evidence to investigate one instance
(or a small number of instances) of a contemporary SE
phenomenon within its real-life context, especially
when the boundary between phenomenon and context
cannot be clearly specified [
A controlled experiment (or simply experiment) [
DON´T start working until you have sketched your research method. This will save you from wasting time. DO explain and properly justify the research method that you have chosen (remember that ESE PhD thesis have a very strong empirical component; therefore, your research method should be empirical).
DO choose the empirical study that best fits your research.
According to Pfleeger [
Using one-tailed tests implies predicting the direction of the effect. One-tailed tests are more powerful than two-tailed tests (we need a smaller test statistic to find a significant result). However, if the result of a one-tailed test is in the opposite direction to what you expected, you cannot reject the null hypothesis, and you will have to disregard the result.
Data analysis is driven by the experimental design. Issues such as the scale used to measure the treatments and dependent variables, the number of factors and whether the experiment has a between- or within–subjects design, will determine the particular data analysis technique to be applied.
However, the choice of data analysis technique and/or statistical model is sometimes not straightforward. Complex designs may require the addition of some extra factors (and possibly interactions) to the statistical model. Take, for example, designs with blocking variables; the blocking variables and their interactions with treatments have to be added as factors to the statistical model. Another example are crossover designs; the order in which subjects apply treatments (sequences) and the times at which each treatment is applied (periods) have to be added to the analysis as factors.
Parametric tests are more powerful than non-parametric tests and are capable of analysing several factors and their interactions. But the data do not always meet the parametric tests assumptions (typically normality and/or homogeneity of variances). However, data transformation and robust tests are an alternative to non-parametric tests.
Statistical significance measures whether the observed effect is the result of treatments or sampling error. It gives no indication of how big the difference in treatments is. For relatively large sample sizes, even very small differences in treatments may be statistically significant. If we want to know whether the differences between treatments are large enough to be of practical importance, we need a measure of effect size.
There are different types of effect size measures.
A priori power analysis is used before the experiment is run to calculate the minimum sample size required for detecting an effect of a given size. Of course, a bigger sample size will be needed to detect small effects than medium or large effects.
Post-hoc power analysis determines the power of a given study assuming that the effect size of the sample is equal to the population. While the utility of a priori power analysis is universally accepted, the usefulness of post-hoc power analysis is controversial (it is a function of the statistical significance).
Research funded by the Spanish Ministry of Economy and Competitiveness research grant TIN2014-60490-P.