Comparative Analysis of Estimation Sizing Approaches to Determine Their Suitability Shashank Patil 1 , Rashmi Sardesai,2, Carl Bideau 3 and Yashowardhan Sowale 4 1 Capgemini Technology Services India private limited, Mumbai, India 2 Capgemini Technology Services India private limited, Mumbai, India 3 Capgemini UK plc, Guernsey, British Isles 4 Capgemini Technology Services India private limited, Pune, India Abstract Software Estimation has been a complex topic across the IT Industry. For successful delivery of any software solution, it is important that it should be standing on strong foundation of estimation. One of the important aspects of project effort estimation is that it should enable competitive pricing of the projects without compromising on quality and schedule of the solution. There are several sizing techniques and estimation approaches prevalent in the industry today to carry out project effort estimations. However, there is no technique or approach which can be termed as “On size fits all”. Each one of the approaches has its own merits and demerits and hence, it’s a challenge for project managers to identify which one is best suitable for their project. In this paper we attempt to study and analyze 2 prominent Estimation Sizing techniques (approaches) and evaluate them on their critical success factors. Generally, the most common factors which establish usefulness of any Estimation Sizing technique are 3R&T, i.e.: 1. Reliability, 2. Repeatability 3. Reproducibility and 4. Turnaround Time Out of these 4 factors, here we have chosen to study and analyze Repeatability and Reproducibility factors for 2 prominent estimation sizing techniques viz: 1. Efforts Estimates derived using Relative Sizing 2. Efforts Estimates derived using Absolute Sizing. Keywords 1 Software Estimation, Agile Estimation, Story Point Sizing, Detailed Sizing, Repeatability, Reproducibility, Empirical Study, Absolute Sizing, Relative Sizing Introduction 1.1. Many Approaches, which one to choose. The biggest challenge that any Project Manager delivering a software solution, faces is how to arrive at the ‘Right’ effort estimates that will satisfy all the stakeholders of the project i.e., Client, Internal Management, Staff, and the project itself. There is a lot of work done in this complex estimation topic right from the era of heavyweight Function Points until today’s lightweight Story Points. There are various Estimation Sizing techniques developed; but in most of the instances these various approaches leave PMs undecided on which approach to choose for a given project. Because each of these techniques have their own merits and demerits. They are fit for various unique situations. Comparative Analysis of Estimation Sizing Approaches to Determine Their Suitability, July 31, 2022, Mumbai, India EMAIL:shashank.patil@capgemini.com;rashmi.sardesai@capgemini.com;carl.bideau@capgemini.com; yashowardhan.sowale@capgemini.com © 2020 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings (CEUR-WS.org) These sizing techniques have evolved over a period. In the initial days, Function Points (FP) found their space in mission critical projects. Eventually as the IT industry grew at an exponential speed, and Agile software delivery came to mainstream, FP lost its popularity and usefulness owing to not able to match the speed of delivery. As we know FP is a very detailed approach that requires meticulous study and input of solution scope. Many IT organizations invested in Estimation & Measurement topic to derive their own proprietary sizing approaches which are kind of golden mean between the two extremes of absolute and relative sizing methods. But this bouquet of techniques often leaves PMs undecided over which one to select for a given project 1.2. Are there any Indicators? How can one determine if a particular sizing approach is fit for given situation? Are there any markers which can rate these? Analysis shows that there could be many parameters which are most useful to categorize and rate a particular sizing technique. After interviewing practitioners on which success criteria, they would like to see in any sizing technique; we found that below are the 4 major parameters that PMs voted for: 1. Reliability, 2. Repeatability 3. Reproducibility and 4. Turnaround Time Out of these first one, “Reliability” is an intangible and subjective. It can be measured only based on perceptions of SMEs. For other 3, there is a scope to measure these by experiments. We selected 2 out of these 3 measurable indicators to study and rate 2 of the most prevalent Estimation Sizing techniques. And these are 1. Reproducibility 2. Repeatability These 2 indicators were evaluated for 2 prominent Sizing techniques: 1. Absolute Sizing 2. Relative Sizing Why we focused on these 2 is because with popularity of Agile delivery method, community is tending towards using quick and handy method of Relative Sizing e.g., Story Point Estimation. But are the results produced using this technique are repeatable and reproducible? We performed various tests by involving estimating volunteers. They carried out a set of different estimates with the combination of different input scope and at different time intervals. The outcome of efforts produced in each of these scenarios were compared and a Hypothesis test was conducted to determine which of these 2 techniques scores high on Repeatability and Reproducibility. INDICATORS EXPLAINED & PROCEDURE OF ANALYSIS 2.1. What is Repeatability? A repeatability test is an experiment performed to evaluate how repeatable your results are under a set of similar conditions. In the context of Software Estimation and Measurement, a repeatability is consistency in estimates derived when a same user performs estimates for a given scope using given approach over the period. A good repeatability in the estimates derived determines the quality of the product. 2.2. Procedure of Analysis Following steps were followed to carry out repeatability test 1. We collected actual data (scope) for 4 different data centric projects. 2. Assigned each scope to 4 different users 3. Each user was asked to enter the scope assigned to them into both Story Point Estimator (Relative Sizing Tool) and e-GREAT™ (Absolute Sizing Tool) and note down the efforts derived. 4. Users were asked to repeat 3rd step for period of 4 days with same scope. 2.2.1. Repeatability Hypothesis: Based on above data collected, here is the hypothesis proposed: “Results produced using Absolute Sizing Approach are more Repeatable than the results produced by using Relative sizing approach.” Repeatability Test Efforts in Person Days Relative Sizing Estimating Tool Absolute Sizing Estimating Tool 150 156 240 150 150 150 385 156 150 270 377 295 385 200 240 270 240 307 150 300 150 250 240 300 625 400 385 350 625 450 385 300 Chart 1: Repeatability Hypothesis 2.3. What is Reproducibility? Reproducibility denotes the consistency in estimates derived when multiple users perform estimates for same scope and same approach over the period. 2.4. Procedure of Analysis Following steps were followed to carry out reproducibility test 1. We collected actual data (scope) for 4 different data centric projects. 2. Assigned Scope for Project 1 to 4 different users on day 1, Project 2 on day 2 and so on for 4 days. 3. All users were asked to enter the scope assigned to them into both Story Point Estimator (Relative Sizing Tool) and e-GREAT™ (Absolute Sizing Tool) and note down the efforts derived on the same day at the same time 4. Users were asked to repeat 3rd step for period of 4 days for different scope. 2.4.1. Reproducibility Hypothesis Based on above data collected, here is the hypothesis proposed: “Results produced using Absolute Sizing are more Reproducible than the results produced by using Relative sizing approach.” Reproducibility Test Efforts in Person Days Relative Sizing Estimating Tool Absolute Sizing Estimating Tool 95 270 377 295 385 200 240 270 150 156 240 150 60 150 385 156 240 307 150 300 150 250 240 300 625 400 385 350 625 320 385 300 Chart 2: Reproducibility Hypothesis Results and Conclusion With above analysis, we concluded that Effort Estimates Derived using Absolute Sizing are more reproducible and repeatable as compared to effort estimates derived using Relative Sizing. References [1] ISO Budgets website: https://www.isobudgets.com/how-to-perform-a-repeatability-test/ [2] Pedro Gustavo Torres: Story Points Explained: The What, Why, and How https://www.scrumalliance.org/community/member- articles/1338#:~:text=Story%20points%20can%20be%20described,long%20a%20task%20will% 20take.