Software Estimation has been a complex topic across the IT Industry. For successful delivery of any software solution, it is important that it should be standing on strong foundation of estimation. One of the important aspects of project effort estimation is that it should enable competitive pricing of the projects without compromising on quality and schedule of the solution. There are several sizing techniques and estimation approaches prevalent in the industry today to carry out project effort estimations. However, there is no technique or approach which can be termed as “On size fits all”. Each one of the approaches has its own merits and demerits and hence, it's a challenge for project managers to identify which one is best suitable for their project. In this paper we attempt to study and analyze 2 prominent Estimation Sizing techniques (approaches) and evaluate them on their critical success factors. Generally, the most common factors which establish usefulness of any Estimation Sizing technique are 3R&T, i.e.: Out of these 4 factors, here we have chosen to study and analyze Repeatability and Reproducibility factors for 2 prominent estimation sizing techniques viz: 1. Efforts Estimates derived using Relative Sizing 2. Efforts Estimates derived using Absolute Sizing.
1.1.
The biggest challenge that any Project Manager delivering a software solution, faces is how to arrive at the ‘Right’ effort estimates that will satisfy all the stakeholders of the project i.e., Client, Internal Management, Staff, and the project itself. There is a lot of work done in this complex estimation topic right from the era of heavyweight Function Points until today’s lightweight Story Points. There are various Estimation Sizing techniques developed; but in most of the instances these various approaches leave PMs undecided on which approach to choose for a given project. Because each of these techniques have their own merits and demerits. They are fit for various unique situations.
These sizing techniques have evolved over a period. In the initial days, Function Points (FP) found their space in mission critical projects. Eventually as the IT industry grew at an exponential speed, and Agile software delivery came to mainstream, FP lost its popularity and usefulness owing to not able to match the speed of delivery. As we know FP is a very detailed approach that requires meticulous study and input of solution scope.
Many IT organizations invested in Estimation & Measurement topic to derive their own proprietary sizing approaches which are kind of golden mean between the two extremes of absolute and relative sizing methods. But this bouquet of techniques often leaves PMs undecided over which one to select for a given project 1.2.
How can one determine if a particular sizing approach is fit for given situation? Are there any markers which can rate these? Analysis shows that there could be many parameters which are most useful to categorize and rate a particular sizing technique. After interviewing practitioners on which success criteria, they would like to see in any sizing technique; we found that below are the 4 major parameters that PMs voted for:
Out of these first one, “Reliability” is an intangible and subjective. It can be measured only based on perceptions of SMEs. For other 3, there is a scope to measure these by experiments. We selected 2 out of these 3 measurable indicators to study and rate 2 of the most prevalent Estimation Sizing techniques. And these are 1. Reproducibility 2. Repeatability These 2 indicators were evaluated for 2 prominent Sizing techniques: 1. Absolute Sizing 2. Relative Sizing
Why we focused on these 2 is because with popularity of Agile delivery method, community is tending towards using quick and handy method of Relative Sizing e.g., Story Point Estimation. But are the results produced using this technique are repeatable and reproducible?
We performed various tests by involving estimating volunteers. They carried out a set of different estimates with the combination of different input scope and at different time intervals. The outcome of efforts produced in each of these scenarios were compared and a Hypothesis test was conducted to determine which of these 2 techniques scores high on Repeatability and Reproducibility. 2.1.
A repeatability test is an experiment performed to evaluate how repeatable your results are under a set of similar conditions.
In the context of Software Estimation and Measurement, a repeatability is consistency in estimates derived when a same user performs estimates for a given scope using given approach over the period.
A good repeatability in the estimates derived determines the quality of the product. Following steps were followed to carry out repeatability test 1. We collected actual data (scope) for 4 different data centric projects. 2. Assigned each scope to 4 different users 3. Each user was asked to enter the scope assigned to them into both Story Point Estimator (Relative Sizing Tool) and e-GREAT™ (Absolute Sizing Tool) and note down the efforts derived.
4. Users were asked to repeat 3rd step for period of 4 days with same scope.
Based on above data collected, here is the hypothesis proposed: “Results produced using Absolute Sizing Approach are more Repeatable than the results produced by using Relative sizing approach.”
Efforts in Person Days Chart 1: Repeatability Hypothesis
Reproducibility denotes the consistency in estimates derived when multiple users perform estimates for same scope and same approach over the period. 2.4.
Following steps were followed to carry out reproducibility test 1. We collected actual data (scope) for 4 different data centric projects. 2. Assigned Scope for Project 1 to 4 different users on day 1, Project 2 on day 2 and so on for 4 days. 3. All users were asked to enter the scope assigned to them into both Story Point Estimator (Relative Sizing Tool) and e-GREAT™ (Absolute Sizing Tool) and note down the efforts derived on the same day at the same time 4. Users were asked to repeat 3rd step for period of 4 days for different scope.
Based on above data collected, here is the hypothesis proposed: “Results produced using Absolute Sizing are more Reproducible than the results produced by using Relative sizing approach.”
Efforts in Person Days Chart 2: Reproducibility Hypothesis
With above analysis, we concluded that Effort Estimates Derived using Absolute Sizing are more reproducible and repeatable as compared to effort estimates derived using Relative Sizing.