=Paper=
{{Paper
|id=Vol-3272/short1
|storemode=property
|title=Comparative Analysis of Estimation Sizing Approaches to Determine Their Suitability
|pdfUrl=https://ceur-ws.org/Vol-3272/IWSM-MENSURA22_short1.pdf
|volume=Vol-3272
|authors=Shashank Patil,Rashmi Sardesai,Carl Bideau,Yashowardhan Sowale
|dblpUrl=https://dblp.org/rec/conf/iwsm/PatilSBS22
}}
==Comparative Analysis of Estimation Sizing Approaches to Determine Their Suitability==
https://ceur-ws.org/Vol-3272/IWSM-MENSURA22_short1.pdf
Comparative Analysis of Estimation Sizing Approaches to
Determine Their Suitability
Shashank Patil 1 , Rashmi Sardesai,2, Carl Bideau 3 and Yashowardhan Sowale 4
1
Capgemini Technology Services India private limited, Mumbai, India
2
Capgemini Technology Services India private limited, Mumbai, India
3
Capgemini UK plc, Guernsey, British Isles
4
Capgemini Technology Services India private limited, Pune, India
Abstract
Software Estimation has been a complex topic across the IT Industry. For successful delivery
of any software solution, it is important that it should be standing on strong foundation of
estimation. One of the important aspects of project effort estimation is that it should enable
competitive pricing of the projects without compromising on quality and schedule of the
solution. There are several sizing techniques and estimation approaches prevalent in the
industry today to carry out project effort estimations. However, there is no technique or
approach which can be termed as “On size fits all”. Each one of the approaches has its own
merits and demerits and hence, it’s a challenge for project managers to identify which one is
best suitable for their project.
In this paper we attempt to study and analyze 2 prominent Estimation Sizing techniques
(approaches) and evaluate them on their critical success factors. Generally, the most common
factors which establish usefulness of any Estimation Sizing technique are 3R&T, i.e.:
1. Reliability,
2. Repeatability
3. Reproducibility and
4. Turnaround Time
Out of these 4 factors, here we have chosen to study and analyze Repeatability and
Reproducibility factors for 2 prominent estimation sizing techniques viz:
1. Efforts Estimates derived using Relative Sizing
2. Efforts Estimates derived using Absolute Sizing.
Keywords 1
Software Estimation, Agile Estimation, Story Point Sizing, Detailed Sizing, Repeatability,
Reproducibility, Empirical Study, Absolute Sizing, Relative Sizing
Introduction
1.1. Many Approaches, which one to choose.
The biggest challenge that any Project Manager delivering a software solution, faces is how to arrive
at the ‘Right’ effort estimates that will satisfy all the stakeholders of the project i.e., Client, Internal
Management, Staff, and the project itself. There is a lot of work done in this complex estimation topic
right from the era of heavyweight Function Points until today’s lightweight Story Points. There are
various Estimation Sizing techniques developed; but in most of the instances these various approaches
leave PMs undecided on which approach to choose for a given project. Because each of these techniques
have their own merits and demerits. They are fit for various unique situations.
Comparative Analysis of Estimation Sizing Approaches to Determine Their Suitability, July 31, 2022, Mumbai, India
EMAIL:shashank.patil@capgemini.com;rashmi.sardesai@capgemini.com;carl.bideau@capgemini.com;
yashowardhan.sowale@capgemini.com
© 2020 Copyright for this paper by its authors.
Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR Workshop Proceedings (CEUR-WS.org)
These sizing techniques have evolved over a period. In the initial days, Function Points (FP) found
their space in mission critical projects. Eventually as the IT industry grew at an exponential speed, and
Agile software delivery came to mainstream, FP lost its popularity and usefulness owing to not able to
match the speed of delivery. As we know FP is a very detailed approach that requires meticulous study
and input of solution scope.
Many IT organizations invested in Estimation & Measurement topic to derive their own proprietary
sizing approaches which are kind of golden mean between the two extremes of absolute and relative
sizing methods. But this bouquet of techniques often leaves PMs undecided over which one to select
for a given project
1.2. Are there any Indicators?
How can one determine if a particular sizing approach is fit for given situation? Are there any
markers which can rate these? Analysis shows that there could be many parameters which are most
useful to categorize and rate a particular sizing technique. After interviewing practitioners on which
success criteria, they would like to see in any sizing technique; we found that below are the 4 major
parameters that PMs voted for:
1. Reliability,
2. Repeatability
3. Reproducibility and
4. Turnaround Time
Out of these first one, “Reliability” is an intangible and subjective. It can be measured only based
on perceptions of SMEs. For other 3, there is a scope to measure these by experiments. We selected 2
out of these 3 measurable indicators to study and rate 2 of the most prevalent Estimation Sizing
techniques. And these are
1. Reproducibility
2. Repeatability
These 2 indicators were evaluated for 2 prominent Sizing techniques:
1. Absolute Sizing
2. Relative Sizing
Why we focused on these 2 is because with popularity of Agile delivery method, community is
tending towards using quick and handy method of Relative Sizing e.g., Story Point Estimation. But are
the results produced using this technique are repeatable and reproducible?
We performed various tests by involving estimating volunteers. They carried out a set of different
estimates with the combination of different input scope and at different time intervals. The outcome of
efforts produced in each of these scenarios were compared and a Hypothesis test was conducted to
determine which of these 2 techniques scores high on Repeatability and Reproducibility.
INDICATORS EXPLAINED & PROCEDURE OF ANALYSIS
2.1. What is Repeatability?
A repeatability test is an experiment performed to evaluate how repeatable your results are under a
set of similar conditions.
In the context of Software Estimation and Measurement, a repeatability is consistency in estimates
derived when a same user performs estimates for a given scope using given approach over the period.
A good repeatability in the estimates derived determines the quality of the product.
2.2. Procedure of Analysis
Following steps were followed to carry out repeatability test
1. We collected actual data (scope) for 4 different data centric projects.
2. Assigned each scope to 4 different users
3. Each user was asked to enter the scope assigned to them into both Story Point Estimator
(Relative Sizing Tool) and e-GREAT™ (Absolute Sizing Tool) and note down the efforts
derived.
4. Users were asked to repeat 3rd step for period of 4 days with same scope.
2.2.1. Repeatability Hypothesis:
Based on above data collected, here is the hypothesis proposed:
“Results produced using Absolute Sizing Approach are more Repeatable than the results
produced by using Relative sizing approach.”
Repeatability Test
Efforts in Person Days
Relative Sizing Estimating Tool Absolute Sizing Estimating Tool
150 156
240 150
150 150
385 156
150 270
377 295
385 200
240 270
240 307
150 300
150 250
240 300
625 400
385 350
625 450
385 300
Chart 1: Repeatability Hypothesis
2.3. What is Reproducibility?
Reproducibility denotes the consistency in estimates derived when multiple users perform
estimates for same scope and same approach over the period.
2.4. Procedure of Analysis
Following steps were followed to carry out reproducibility test
1. We collected actual data (scope) for 4 different data centric projects.
2. Assigned Scope for Project 1 to 4 different users on day 1, Project 2 on day 2 and so on for 4
days.
3. All users were asked to enter the scope assigned to them into both Story Point Estimator (Relative
Sizing Tool) and e-GREAT™ (Absolute Sizing Tool) and note down the efforts derived on the
same day at the same time
4. Users were asked to repeat 3rd step for period of 4 days for different scope.
2.4.1. Reproducibility Hypothesis
Based on above data collected, here is the hypothesis proposed:
“Results produced using Absolute Sizing are more Reproducible than the results produced by
using Relative sizing approach.”
Reproducibility Test
Efforts in Person Days
Relative Sizing Estimating Tool Absolute Sizing Estimating Tool
95 270
377 295
385 200
240 270
150 156
240 150
60 150
385 156
240 307
150 300
150 250
240 300
625 400
385 350
625 320
385 300
Chart 2: Reproducibility Hypothesis
Results and Conclusion
With above analysis, we concluded that Effort Estimates Derived using Absolute Sizing are more
reproducible and repeatable as compared to effort estimates derived using Relative Sizing.
References
[1] ISO Budgets website: https://www.isobudgets.com/how-to-perform-a-repeatability-test/
[2] Pedro Gustavo Torres: Story Points Explained: The What, Why, and How
https://www.scrumalliance.org/community/member-
articles/1338#:~:text=Story%20points%20can%20be%20described,long%20a%20task%20will%
20take.