In software development, the management of standardized metrics are not as frequent as it should be, which encourages the immaturity of software engineering. Currently, few companies use standards for the software functional size measurement (i.e. COSMIC); however, an increase in the adoption of this practice is emerging, derived from the need to have greater certainty, both in the estimates and in the management of their projects. A problem faced by companies that already use standardized metrics is knowing formally what proportion of improvement can be required of suppliers as they gain more experience as the time of the customersupplier relationship passes. This paper presents a proposal to determine the learning ratio of a supplier in order to request improvement of the productivity factor (PDR) with which the supplier has been worked in previous cycles through a real case study in the Mexican industry, using the learning curve theory.
It is known that the functionality of a software product can be measured using some functional size measurement method (FSMM ), for example, COSMIC
ISO/IEC 19761. The possibility of measuring size allows us to also measure productivity in software development.
When a software provider is contracted for various sequential periods to
develop products in the same or similar context, know-how is obtained about
the problem domain, the development context and the technologies utilized.
This can also be observed in individual projects, since for large projects higher
productivity is observed than for small projects[
After having hired a software provider for a certain number of periods, it is natural that the client wants to have an improvement factor that impacts a reduction in cost or increase in productivity related to a better knowledge of the context obtained through the time from the provider. However, there is di culty in de ning this factor formally for the projects in subsequent periods, since it must be well-founded and consistent value, otherwise, it could compromise the success of the projects.
The learning curve theory is a tool for to estimate the recurring costs in a
production process and is based on the common observation that, by repeating
a task, it can be completed in shorter periods of time or required less e ort to
be developed, allowing to determine the learning level in repetitive activities,
which serves to make estimates[
In the literature reviewed, this learning factor is found in di erent areas of
the industry, such as in microenterprises[
However, in the literature reviewed for the software development, a formal and well-founded way to estimate the productivity improvement factor in subsequent projects has not been identi ed; having a mechanism to do so is useful for companies that contract software developments for consecutive periods.
This article presents a case study to determine the degree of learning considering di erent periods in which a supplier worked under the same context by applying the learning curves. Using the PDR applied in previous periods by the supplier, the idea is to determine a reliable improvement factor through a formal mechanism such as the learning curves theory.
The article is organized in the following manner: Section 2 mention the background of the measurement of the functional size of software using COSMIC. Section 3 shows the theoretical bases of the learning curves and the di erent approaches for these theories. Section 4 is the case study of this paper. It shows the data and the calculations to use the estimating lot cost using unit theory approach to estimate the PDR, which is possible and achievable by the provider, and quality criteria for the model obtained. Finally, the conclusions are presented, and the PDR expected for the rst semester of the year 2020. 2
Functional size is the only current standard measurement of the software [
The COSMIC method was initially accepted by ISO/IEC JTC1 SC7 as an
International Standard in December 2002. The current version is ISO/IEC
19761 : 2011 Software Engineering - COSMIC - A Functional Size Measurement
Method (actually ISO/IEC 19761 )[
Learning curves analysis is a theory that allows us to estimate recurring costs
in production processes[
In the theory of learning curves, the dominant factor is the direct work required to complete the task or product. This is based on the observation that completing a task several times generates learning for the person who performs it, allowing them to nish this task in less time the next time it is made.
There are two predominant theories:
{ the Unitary Theory (UT )[
Both theories are based on the fact that; if a task is repeated several times,
the experience is gained, allowing the tasks to be performed in a shorter time
for the next iteration[
The main di erences between these theories depend on the data required to carry out the analysis; for context with considerable variations in costs or design, it is recommended to use the AAT since it allows reducing the estimation risk. On the other hand, UT is recommended to be used when the available data are accurate.
Considering this, for this paper, we will use the Unitary Theory, which de nes two types of approaches, estimating unit cost and estimating lot cost.
If there is learning in the production process, the cost of a 2k unit is equal to the cost of unit k times the slope of the learning curve (J. R. Crawford 1947). This means that for an 80% Learning Curve, there is a 20% cost reduction each time the number of units is doubled, for example, unit 2 is 80% the cost of unit 1 and the cost of Unit 4 is 80% the cost of Unit 2.
The following equation de nes the learning curves using Unit Theory: Y (x) = A xb
Where: { Y (x) is the cost of unit number x. { A is the cost of the rst unit. { x is the unit number x. { b is a constant which represents the slope of the learning curve.
To apply the equation 1, it is necessary to have the cost data for each unit produced. 3.2
However, because the cost of production for each unit is rarely reported, a viable option is to estimate the production cost in lots.
To estimate the production cost in lots, equation 1 is used; however, it is necessary to adjust the information of each lot. Considering that the analysis of learning curves requires knowing the unit number and the cost associated with each one, therefore these values will be represented by the lot midpoint (LMP ) and the average cost per unit within it (AUC ). The following operations are performed to calculate these values.
Calculating the exact LMP value is an iterative process, but it can be easily estimated using an approximation, as shown in equations 2, 3, and 4.
For the rst lot: For all subsequent lots:
If lotSize < 10; then LM P =
F + L + 2pF 4 lotSize
2 lotSize
3 L
Where: { F is the rst unit number in a lot. { L is the last unit number in a lot. (1) (2) (3) (4)
Since the initial and nal unit values for each lot are cumulative, we must nd the F and L values of each one in order. For example, for 3 lots of 50 pieces each, the rst lot goes from unit 1 to 50, the second from 51 to 100, and nally, the third goes from unit 101 to 150.
Finally, to calculate the AUC value of each lot, divide the number of units in the lot by the cost of developing it, as shown in equation 5:
AU C =
(5) 4 4.1
To conduct this case study in a real world context, the methodology proposed
by Runeson[
In this case study, the analysis will be performed on information from a contract between a Mexican government entity from the energy sector and a software provider. Due to con dentiality issues, the name of the entity will not be mentioned in this document.
Every certain period, approximately two years, this entity contracts software factories to develop the project it requires. The current provider for the software development service was contracted for two years, 2018 and 2019, and the contract was extended for the rst half of 2020.
To analyze the productivity that can be required of this factory for the rst
semester of 2020, the entity provided information on 21 projects developed in
the rst two years by the provider. This information contains, for each project
developed, the year and semester in which it was developed, identi er, required
e ort, and functional size, measured in CFP 3 (Table 2). The functional size
was acquired utilizing the EPCU approximation approach, as was de ned by
the Experts Guide for Early Software Sizing with COSMIC[
Derived from the fact that the data provided can group the projects by the semester in which they were developed, the unit theory approach using lot costs is considered in this case study. Each semester contains a batch of certain number of functional size units.
Two concepts widely known in the Operations Research theory will be used
to allow the modeling of the productivity of the provider: Productivity Factor
(PDR) and Productivity[
The PDR represents how many Work Hours [WH] are required to develop a functional size unit [CFP], and its units are given by [WH/CFP]. In contrast, Productivity represents how many [CFP] are implemented by [WH], and its units are given in [CFP/WH], which is the inverse of the PDR.
It is worth mentioning that the approach to determine the PDR in the rst
two semesters was not entirely correct, since the entity, due to poor advice,
de ned that all the projects that were developed in this period of time would
be estimated with a determined xed PDR in 34 [WH/CFP], which was de ned
using expert judgment, that is, without formal support derived from the analysis
of historical data as recommended by the best practices. For the second year two
semesters, 2019, the COSMIC methodology was correctly implemented using
the reference database of the Mexican Association of Software Metrics[
With the data in Table 2, the learning curve analysis will be performed using the UT to estimate cost per lot, considering each semester of development as one, so it is necessary to perform the LMP and AUC calculations as mentioned in section 3.2. Table 3 shows the data resulting from these calculations.
Figure 1 shows the curve generated using the data from Table 3, in the "Y" axis (AUC ), the average cost of each unit per lot is presented. In the "X" axis (LMP ), the midpoint of the lot was considered.
A transformation over the values of both axes is made using the Natural
Logarithm (ln) function to make the data more linear in order to perform a linear
regression[
Fig. 1. AUC(PDR) vs. LMP.
The values of the constants that de ne the equation of the line contained in gure 2 can be found by performing a linear regression using the data in Table 3, which results in the following equation:
Y (x) =
0:171x + 4:1581
Where: { Y is the AUC { x is the LMP
However, these results are found in units of natural logarithm (ln), since the equation that we really have is: ln(P DR) = 0:171 ln(LM P ) + 4:1581 (6) (7)
To transform back this equation, the exponential function needs to be used in each side; as a result, the following equation:
In this equation, we can see that the slope of the learning curve is: P DR = 63:951
LM P 0:171 2b = 2 0:171 = 0:887 = 88:7% (8) (9)
This is the equation that best models the production environments for the data set in Table 2. This shows that there is a learning curve of 88.7%; in consequence, there is a productivity increase of 11.3% in each time it is doubled the amount of CFP developed.
Now it is possible to solve the equation to know the average cost per unit of any future lot using its LMP value, which can be obtained once the rst and last unit are known.
As a validation exercise, the functional size to be developed during the contract extension was estimated. The average size of the projects developed in the four previous semesters was considered as the estimated value for the extended period. The value was 1997 [CFP ].
From this data, the values for lot 5 ( rst semester of the year 2020) are calculated, which are the rst and last unit, the lot midpoint (LMP ), and equation 8 is used to calculate the PDR of the lot. Finally, since PDR = [WH/CFP], the amount of [WH] is given by multiplying the PDR [WH/CFP] by the functional size in [CFP].
Table 4 shows in column 4 the PDR for each lot from the past periods, while column 5 shows the estimated value for that lot using equation 8. In the last line, there is a lot number as 5, this corresponds to the rst semester of 2020, in column 1 the average of [CFP] developed from the previous periods was considered as the last batch (1997 [CFP]) size, for this value the PDR estimated using the equation 8 also was calculated. In the last column (6) the Magnitude of Relative Error (MRE) was calculated considering the real value and the estimated value for the four initial lots, considering the real PDR (column 4) and the estimated PDR (column 5).
Using the column (5), the quality criteria for estimation was evaluated to analyze the robustness of the model, which are Mean Magnitude of Relative
Error (MMRE), MRE Standard Deviation (RMS) and the Prediction level at 25% (Pred 25%) shown in Table 5.
Based on the information in Table 5, it can be mentioned that there is an average relative error of 8.8%, with a standard deviation of 2.913, and all the points are within the 25% prediction level. Observing the MRE value from Table 5, and considering the prediction level quality criteria, all the points are within the prediction level Pred(9.8%). That is, all the estimations present a relative error equal or below to 9.8%; in this sense, we can expect that the estimate for period 5 of the PDR to be used is 13.44 [WH/CFP] 9.8%.
After graphing the values in Table 4, it is observed in gure 3 that the estimated values (green) for the known periods have the same behavior as the original values (blue), with an MMRE of 8.8% (Table 5).
So we have a model that reliably represents the learning curve de ned by equation 8, which has a learning rate of 88.7%. It is possible and achievable by the provider, since it was formally obtained using an analysis of Learning Curves and based on the observed productivity of previous development periods. 5
A problem faced by companies that have the same provider for several periods of time, is knowing formally what proportion of improvement could be expected because the know-how acquired during the service.
This paper presents a proposal to determine the learning ratio of a supplier in order to request an improvement rate of the productivity factor (PDR) based in the previous cycles through a real case study in the Mexican industry. The approach utilized to determine the degree of learning is the learning curve theory.
The analysis of Learning Curves for this case study has shown that there is learning throughout each period in which di erent projects were developed. This shows that productivity improves over time, showing a learning rate of 88.7%, representing a productivity increase of 11.3%.
With these results, we can estimate the e ort in [WH] to produce a [CFP ] unit, that is, the PDR for the next batch, once we have the estimated size of the software to be developed in that period.
In order to estimate the expected functional size to be developed in the lot of the rst semester of the year 2020, the average of the functional size from the lots referring to the years 2018 and 2019 was considered. The expected functional size for the rst semester is 1997 [CFP ], using the equation 8, we can expect a PDR of 13.44[WH/CFP] 9.8%.
The case study presented has only few data to analyze, so the future work is to look for the bigger software projects data sets to repeat the analysis and compare the results. 6
To analyze the productivity of a software development company, other variables
must be included, such as personnel turnover and production interruptions[