Background. Having a correct, although approximate, knowledge of software development productivity is clearly important. In some environments, the belief that software enhancement projects are characterized by higher productivity than new software development has emerged. Aim. We want to understand whether the mentioned belief is rooted on solid bases or is due to some cognitive biases. Method. An empirical study was performed, analyzing the data from a large dataset that collects data from real-life projects. Several statistical methods were used to evaluate the unitary cost (i.e., the cost per Function Point) of enhancement projects and new developments. Results. Our analyses show that-contrary to some popular beliefs-software enhancement costs more than new software development, at least for projects greater than 300 Function Points. Conclusions. Project managers and other stakeholders interested in the actual cost of software should reject ill-based evaluations that the productivity of software enhancement is greater than new software development. More generally, objective evaluations based on the analysis of representative data should be preferred to evaluations affected by cognitive biases.
Software cost models are relevant to the market since they influence budget allocation on individual projects, tenders, contracts and finally they affect the quality of customer-supplier relationships. Since the number of factors that may influence, at various levels of impact, the productivity in delivering software systems is huge, it is unavoidable to make some assumptions and to focus on a subset of variables considered as significant. Assumptions may be based on direct personal experiences, on common sense or on empirical evidence derived from collected data. Sometimes the experts that build software cost models are induced to introduce in models some biases that may affect the quality of correlation among the considered variables. This is done in absence of awareness due to the cognitive biases effect.
Specifically, we have observed the tendency to consider the enhancement of existing software as less demanding–in terms of total effort–than the development of new software. Since such beliefs can have quite relevant consequences (e.g., setting unrealistic prices for software enhancement contracts), it is of great importance that we show if the belief is rooted on solid bases, or it is affected by cognitive biases.
In this paper, we apply the empirical method of testing the assumptions using adequate data to address the problem of evaluating the actual effort of software enhancement, especially compared to the effort of developing new software. In this way, we contribute to reduce or inhibit the impact of cognitive biases on software cost models.
The paper is organized as follows. Section 2 presents the motivations of the work and highlights the importance of getting objective evaluations of the productivity of software enhancement. Section 3 describes the empirical study through which we derive objective effort evaluations. In Section 4 we discuss the threats to the validity of the empirical study. Section 5 accounts for related work. Finally, Section 6 draws some conclusions and outlines future work. 2
Cognitive sciences have shown that even the finest expert may be affected by
cognitive biases [
Prejudice is a pre-defined cause-effect relationship, based on hyper-generalizations, that does not need a test, validation, adaptation in order to be considered true by its “owner”.
Confirmation bias is the tendency to pay undue attention to sources that confirm our existing beliefs while ignoring sources that challenge our beliefs. Once we have elaborated a theory, we tend to look for confirmations instead of unexplained facts.
Overconfidence bias is the tendency to overestimate one’s skills and abilities. This may lead to ignore some factors in the model only because we do not think that we may personally be affected by that factor.
Group-thinking is a psychological phenomenon that occurs within a group of people in which the desire for harmony or conformity in the group results in an irrational or dysfunctional decision-making outcome, by inhibiting critical thinking to avoid conflicts. This may happen even to people that do not know one each other, but that belong to a community ruled by opinion leaders and recognized experts. It is very difficult for anybody to swim against the flow.
Availability bias is a tendency to allow information that is easier to recall unduly influence preconceptions or judgments. We use in models the parameters that are the easiest to be measured, regardless of the relevance they really have.
Finally, the framing effect is the tendency to react differently to situations that are fundamentally identical but presented (or framed) differently.
All these cognitive biases may induce to build a model that may be poorly representative of reality. The only way to know if this has happened is to “de-bias” the reasoning with specific approaches possibly supported by empirical data.
The wrong definition of a software cost model may be favored by situation in which the outcome of a production process is influenced by two or more factors that may affect the result in opposite ways and we are wrong in the identification of the resultant of the vector composition.
In this paper, we will focus on the productivity associated to a functional enhancement project compared to a new development project. If we believe that adding, changing and deleting a certain amount of functionalities in an existent application is much easier than adding the same amount of functionalities in a new application (due to reuse, fundamentally), we must expect a higher productivity in enhancement maintenance. On the contrary, if we consider the difficulty of adding, changing and deleting functionalities that are not well documented, were written with old technologies, by not particularly skilled people in an unmanaged environment, then we will expect a lower productivity with respect to new development. Which one is true or more often true in the market? The right thing to do is to collect data and derive statistical driven inference. The most frequent thing that has been done in the past is the application of the cognitive biases illustrated before.
The initial idea that changing and deleting functionalities should be associated to a
lower functional measure, if compared with the operation of adding new functions, was
supported by a NESMA (Netherlands Software Metrics Users Association) document
[
The specific situation may have, and actually had, a huge impact on markets and outsourcing contracts (in the order of millions of euro). Consider that the “discount approach” for enhancement initiatives generates less than half the revenues with respect to the neutral assumption (same cost as development by scratch) and less than 35% of the opposite assumption (cost of enhancement is 150% of development by scratch).
A cost model for contractual goals should be built according to sound articulated
approaches and empirically derived knowledge [
In this paper we present an empirical study–based on data from real-life projects–to determine objectively (although possibly approximatively) the real relation between the cost of enhancing existing software and the cost of developing new software. 3 3.1
We analyzed data from the ISBSG dataset [
The descriptive statistics of the analyzed datasets are given in Table 1. Specifically, Table 1 illustrates the characteristics of the New developments and Enhancements datasets. For each dataset, statistics concerning the size expressed in UFP, the effort expressed in PH (person hours) and the number of person hours required per function point are given.
First, we performed a very simple comparison of the effort per FP required by New development and enhancement projects.
Fig. 1 shows that the quantity of effort per FP required by enhancement projects is
generally greater: both the mean and the median are greater than new developments'.
Similarly, several enhancement projects required a very large amount of effort per FP,
as shown in the left part of Fig. 1. The main statistics concerning the required effort per
FP are given in Table 2. It appears that Enhancement projects require more effort per
FP than new development projects. This fact was tested by means of the Wilcoxon rank
sum test [
Another simple evaluation of the effort per FP required for New developments and enhancement projects can be obtained via lowess curves of effort vs. size. Lowess (locally weighted scatterplot smoothing) is a nonparametric method for fitting a smooth curve between two variables; in this nonparametric method, the linearity assumptions of conventional regression methods are relaxed.
Several project managers could consider the observations reported above to be sufficient to conclude that–except for small projects–the productivity of New developments is greater than the productivity of Enhancement projects. However, we would be more comfortable if we could provide evidence based on proper model, which provide a statistically sound synthesis of the relationship that links effort and size. To this end, we proceeded to build model of effort as a function of size.
All effort models were derived via ordinary least square (OLS) regression. We used
Cook’s distance to identify possible influential observations. Data points with Cook’s
distance greater than 4/n (n being the cardinality of the training set) were considered
for removal as suggested by Kitchenham and Mendes [
New Development
Effort = 9 Size 292 (34%)
Enhancement Effort = 11.48 Size
1172 (40%)
We started building linear models. The obtained models and their characteristics are given in Table 3.
The OLS linear regression models described in Table 3 provide some interesting indications, in that 1) they confirm that Enhancement projects are more expensive than New development projects; 2) the obtained coefficients are very close to the median Effort/Size values given in Table 2. However, both models do not conform to the OLS regression constraints, having not normally distributed residuals. In addition, they were built by discarding as outliers a large fraction of the projects.
Therefore, we looked for non-linear OLS regression models. Specifically, we
performed logarithmic transformations of both the independent and dependent variables.
This choice was suggested by both the desire to be compliant with earlier research (for
instance, Boehm’s COCOMO [
Since no really adequate OLS regression models could be achieved, we adopted analogy-based estimation (AbE). With AbE, the effort required by a project is estimated based on the effort that was required by "similar" projects.
Given a project P, we selected the projects that contribute to estimate the development or maintenance effort for P as follows: 1) Let sp=0.02. 2) Let NP be the set of projects such that p ∈ NP iff (1-sp) size(P) ≤ size(p) ≤ (1+sp) size(P) 3) If |NP| ≥ 7, let the estimated effort for P be the median of the efforts of the projects belonging to NP.
4) Otherwise, increase sp by 0.01 and go back to step 2).
Before proceeding to apply AbE, we determined the size range in which there are enough data points to support AbE. The histograms in Fig. 4 show that–as could be expected–there are few large Enhancement projects. Specifically, above 800 UFP projects are few and sparse, hence it is difficult to find "similar" projects for AbE. Therefore, in the rest of this study we consider only projects having size not greater than 800 UFP. Noticeably, for New development there are no problems of the type discussed above: there are many projects having size greater than 800 UFP, actually, there are many projects up to 2000 UFP. Nonetheless, since we are mainly interested in comparison between Enhancement and New development projects, we need to study both types of projects in the same size range.
To evaluate whether AbE estimates are worth considering, we have to verify that
they provide better performance than baseline models. To this end, Shepperd and
MacDonell [
Shepperd and MacDonell also proposed that the accuracy of a given estimation method be measured via the Mean Absolute Residual (MAR), i.e., the mean of the absolute values of errors, where errors are computed as actuals minus estimates.
A random effort estimation for a project is obtained by picking at random the actual effort of any of the other projects. Of course, in this way there are n−1 possible estimates for every project; therefore, to compute the MAR of the random model we need to average all these possible values. Shepperd and MacDonell suggest to make a large number of random estimates (typically 1000), and then compute the mean MAR. Shepperd and MacDonell observed also that the value of the 5% quantile of the random estimate MARs can be interpreted like α for conventional statistical inference. Accordingly, the MAR of a proposed model should be compared with the 5% quantile of the random estimate MARs, to make us reasonably sure that the model is actually more accurate than the random estimation.
We also used “constant” models as baseline, proposed, among others, by Lavazza
and Morasca [
The models’ estimation errors are given in Table 5 and visualized via a boxplot in Fig. 5, where outliers are not show, for readability. In Fig. 5, the orange diamonds represent the means, i.e., the MARs. The dashed line is the MAR of the mean constant model, the dotted line is the MAR of the median constant model, the continuous line is the 5% quantile of the random estimate MARs. It can be observed that the MARs of AbE estimates are smaller than the baselines' MARs, hence AbE is an improvement over baseline estimation methods. This fact is also confirmed by Wilcoxon sign rank test. 1940 2664 2267 3427
Fig. 6 show Effort estimates, in comparison with actual effort values. Understanding the trend of estimated effort vs size (hence of effort per FP) from Fig. 6 is not easy. Thus, we use again the lowess curves to provide a more readable representation of the estimated effort vs. size. Fig. 7 shows such lowess curves for New developments and Enhancement, respectively. It can be observed that, just like in Fig. 2, the slope of the New development curve decreases around 300 FP, while the Enhancement curve appears approximately straight.
Fig. 8 compares the lowess curves given in Fig. 7, highlighting that the effort per FP required by Enhancement projects is greater than the effort per FP required by New developments. Fig. 8 is remarkably similar to Fig. 2 and Fig. 3: this fact reinforces the observation given above. In the previous sections, we evaluated the effort per UFP in several ways: using simple statistics, building regression models, and via analogy-based estimation. In all cases we got clear indications that–at least when the project size is larger than 300 UFP, the effort required by Enhancement projects is larger than the effort required by New developments.
However, large variations of the effort per UFP were observed in the ISBSG dataset: Table 2 shows that the standard deviation is larger than the mean for both New developments and Enhancements; for the latter it is almost twice the mean. Therefore, practitioner should be careful in using the productivity values presented in this paper: they should take into account some variability. 4
Internal validity of the study could be affected by the way measures were obtained. We mitigated this threat by carefully selecting the measures that are classified as most reliable in the ISBSG dataset.
The models presented in Section 3 do not qualify as proper regression model, since most of them have not normally distributed residuals (and sometimes other problems as well). Nonetheless, together with the other findings given in Section 3, they provide reasonably reliable indications, at least qualitatively.
Concerning the generalizability of the proposed results, we analyzed the largest public available dataset, (see the descriptive statistics in Table 1). Although we cannot claim that our findings are generally valid, the fact that they are based on a large dataset, which collects data from many different software development organizations, supports the hypothesis that our findings are representative of many software projects.
It can be noticed that we built effort models based only on size, while usually effort models account for multiple factors that are believed to affect development or maintenance effort. In this paper, we limited the investigation to size based models to get straightforward indications concerning the amount of effort needed in relation to the size of the project. In this respect, it is worth noting that currently many public administrations and private organizations, worldwide, adopt contractual cost models that are based on the size of the software to be delivered as the only independent variable. Although such practice has evident limits, it is widely used. This paper provides results that help applying the mentioned practice based on objective empirical knowledge, thus avoiding macroscopic mistakes, like assuming that enhancement cost less–on a unitary basis–than new development.
Another possible threat comes from the possibility that the observed differences are due to the managerial choices concerning new development and enhancement projects. For instance, managers could assign more skilled people to new development projects and less skilled people to enhancement projects: this would explain the observed differences, at least partly.
Finally, we have to notice that the available dataset contains little data concerning enhancement projects larger than 800 FP. Accordingly, we limited the study to projects not greater than 800 FP. We cannot make any claim concerning larger projects. 5
Approaches to effort prediction are generally grouped into three general categories:
expert judgement, algorithmic models, and analogy [
Estimation by analogy predicts the effort of the target project using information from
former similar project at the system level and sub-system level. The primary steps in
this method consist of: 1) choosing the right analogy, usually measured via Euclidean
distance, 2) investigating similarities and differences, 3) examining analogy quality, 4)
providing the estimation [
The key factor of successful EbA method is finding the appropriate right analogy.
The simple method is using a fixed number of analogies starting from k=1 and increase
this number until non further improvement on the accuracy can be obtained [
In fact, the great majority of software effort estimation methods proposed in the literature use software size or size dependent elements as a major explanatory variable. Size measures are considered as the most influential predictors for estimation. Several functional size measures have been defined, including Function Points, Use Case Points, NESMA, FiSMA, Mark II FP, COSMIC FP, SiFP and many others. In this paper we investigate the relationship of effort required between enhancement projects and new development projects, using only FP as a size measure. Investigations using other measures is an objective for future work.
Concerning the study of productivity of software enhancement projects, this is
definitely not a new research field. Back in 1993, Abran and Robillard performed an
empirical study based on the analysis of 21 projects [
Having a–possibly approximate–quantitative knowledge of the unitary cost of software projects in terms of effort per function point is very important for several purposes. Specifically, when considering the possibility of making a bid, having a reasonably good idea of the cost of the project is essential.
Relatively little research was performed concerning the unitary cost of enhancement projects, at least in comparison with the huge amount of work performed on investigating the cost of new software development. The relatively small amount of empirical data is favoring ill-based guessing. In some environment, the false believe that software enhancement activity cost less than new development–on a unitary basis–is spreading.
In this paper, we looked at what reliable and objective knowledge we can derive from real project data.
Not all the results we presented here are perfectly reliable from a statistical point of view. Nonetheless, the consistency of results we obtained via different analysis techniques seems to indicate that the indications we derived are–at least qualitatively–correct and reliable.
In summary, we found that: ˗ Enhancement projects have a unitary cost that is generally greater than new development projects. ˗ Specifically, the unitary cost of enhancements and new developments is similar for projects up to around 300 FP, while for larger projects the unitary cost of enhancements is greater. ˗ As shown in Fig. 6, the unitary cost is largely variable, even for projects having approximately the same size. Therefore, the data presented here must be regarded as indicating tendencies, but are not necessarily valid for all projects.
From the cognitive bias point of view, our empirical study showed that, based on the data available in a very reputed dataset (namely, the ISBSG dataset), the assumption that productivity is higher for functional enhancement projects than for new development projects is not supported by evidence. Instead, the opposite is true, except for fairly small projects. We may thus state that–most likely–many huge contracts have been undervalued for years because of an apparently reasonable assumption, not confirmed by empirical data.
Future work include, among other activities, looking for factors that let us select project classes characterized by small variations in unitary cost, and experimenting with different techniques for building effort models.
This work has been partially supported by the “Fondo di ricerca d’Ateneo” of the Università degli Studi dell’Insubria.