In this paper we provide an overview of the research conducted on three novel models for estimating effort and costs in software project implementation. All three models utilize different architectures of artificial neural networks (ANNs) constructed based on Taguchi's orthogonal vector plans. The idea behind the conducted research is to optimize these novel models to avoid experiment repetition and reduce training time when estimating software projects. The structure of the proposed models could be improved by employing different encoding functions and clustering techniques for input data, aiming to mitigate the heterogeneous nature observed in various sets of real projects. Additionally, studies suggest the homogenization of input values across projects, leading to higher reliability and accuracy in the obtained results. Optimization with the Taguchi method, coupled with increased coverage of a wide range of industrial projects, results in the efficient and successful completion of various software projects, bringing benefits to the modern software industry.
In the contemporary software industry, there is an increasing demand for rapid, high-quality, and
precise estimation of effort and costs prior to commencing software product development [1]. An
essential factor for successful software project development and risk reduction involves accurately
estimating the effort and costs associated with implementation. This process entails a meticulous
evaluation of essential resources, such as time, personnel, and materials, necessary for project
execution. The primary objective of effort and cost estimation lies in providing informed
decisionmaking support for planning, budgeting, resource allocation, and project management [2]. The precise
and reliable estimation of effort and costs facilitates efficient project management, mitigates risks, and
leads to successful outcomes. Statistics reveal that only one-third of projects are successfully completed
and implemented, nearly half exceed budget and implementation timelines, while around 20%
experience complete failure. Therefore, precise and comprehensive estimation of effort and costs is a
crucial determinant of software project success [
ANNs are artificial intelligence techniques constructed based on mathematical models and the
architecture of each ANN [
Table 1
Taguchi design No. of experiments FFD No. of experiments 4(23) 8(27) 12(211) 16(215) 9(34) 18(37)
In this paper, new enhanced approaches and constructed models will be analyzed, which utilize different ANN architectures, to improve the accuracy and efficiency of effort and cost estimation during software project implementation. What is innovative compared to previous models is the construction of different ANN architectures based on Taguchi's orthogonal array plans. This enables fast, precise, and efficient estimation of effort and costs presented through three different models, using three commonly employed approaches. The main objectives of these models are: the construction and identification of the best model; the selection of an optimal ANN architecture that quickly converges to minimal magnitude relative error; reduction of the number of experiments; and shortening the time required for software effort estimation through high convergence rates. The structure of the paper is as follows: Section 2 outlines prior software estimation approaches. Section 3 introduces three new improved models' methodology. Section 4 presents achieved results. Section 5 discusses these results. Concluding remarks are in Section 6.
As a result of insufficiently adequate processes in the past, a large number of software projects were
unsuccessful or not realized. Commonly used methods were similarity-based estimation, analysis and
synthesis method, expert knowledge-based estimation, and various parametric methods [
The Analysis/Synthesis method is an approach to estimating software project effort and cost that
involves dividing the project into smaller parts that are estimated separately. The advantage is that the
method breaks down the project into smaller parts that are easier to estimate, allowing for a more
detailed analysis and synthesis of effort and cost. However, this method requires more time as each part
needs to be thoroughly estimated. Additionally, the overall estimation may be less reliable [
Expert knowledge-based estimation is a rapid and simple method that relies on the experience of
experts who have worked on similar software projects. The advantage of this method lies in the rapid
estimation of effort and cost. However, the subjectivity of the experts can lead to variations in the
estimates. Consensus among experts can be challenging, so it is important to establish clear guidelines
and criteria for estimation. Validating the estimates through comparison with actual results from
previous projects can improve the accuracy of the estimates [
Parametric (algorithmic) methods rely on project metrics to construct an algorithm for determining
time and cost. This method has advantages in objectivity, speed, and simplicity. Objectivity is achieved
by using quantitative measures. Speed is achieved through automated calculation based on the
algorithm. Ease of use is reflected in the requirement for basic knowledge of project size measurement
and tracking historical data. However, drawbacks include the need for relevant and reliable historical
data and lower precision in cases of high variability or novel technological concepts [
The COCOMO2000 (Constructive Cost Model) is the most frequently employed model based on
the source lines of code [
The approach based on analyzing function points is utilized to estimate the size of software
functionality during development [
The approach of analyzing users and use cases is also utilized to estimate software effort. COBRA
(Cost Estimation, Benchmarking, and Risk Assessment) and UCP (Use Case Point Analysis) are the
most commonly used models within this approach [
motivation, stability requirements, team availability, and programming language
complexity. The UCP method combines system users and use cases, categorizing them based on their
interaction complexity and transaction volume, to determine the actual size using weighted factors. The
system size is defined using a four or six-dimensional vector representing user(s) and use case(s)
complexity [
In this section, three new models constructed using different ANN architectures based on Taguchi's
orthogonal vector plans will be presented [
Twenty-two parameters are the input variables of the COCOMO2000 model, divided into two groups: The first group consists of five parameters known as scale factors: PREC, FLEX, RESL, TEAM, PMAT. The second group consists of seventeen parameters known as effort multipliers: RELY, CPLX, DATA, RUSE, TIME, STOR, PVOL, ACAP, PCAP, PCON, APEX, PLEX, LTEX, TOOL, SCED, SITE, DOCU. Finally, the COCOMO2000 formula was obtained by formulas (1)-(7): = × [ ] × ∏1=71 , , = 2.94, = 0.91,
] × , = + 0.01 × ∑5=1 = ∏1=71 , [
] = 2.94 × [ = × (
= ,
) ,
= + 0.2 × 0.01 × ∑5=1
, = 3.67, = 0.28,
(1)
(2)
(3)
(4)
(5)
(6)
(7)
where A and B are basic calibration constants; KSLOC (thousands of lines of source code) is the size of
the software project; SFj is five scale factor; EMi is seventeen effort multiplier. Our experiments utilize
the COCOMO2000 Post Architecture model. This model combines the effort factors and multipliers to
calculate the required person-months [PM] for the implementation of a specific software project.
Variables E, PEMi, and KSLOC are used as input for four different ANN architectures constructed based
on Taguchi's orthogonal vector plans: ANN-L9, ANN-L18, ANN-L27, and ANN-L36 [
The first ANN architecture, in the COCOMO2000 approach, ANN-L9, is based on Taguchi's
orthogonal vector plan (L9) with four parameters ( , i=(1,4)) and three different levels (L1, L2, L3).
The experiments are conducted with 9 ANNs candidates labeled as ANN1,..., ANN9 Figure 1. The
second constructed ANN architecture, labeled as ANN-L18, is based on Taguchi's orthogonal vector
plan (L18) with eight parameters ( , i=(1,8)) and three different levels (L1, L2, L3). The experiments
are conducted with 18 candidate ANNs labeled as ANN1,..., ANN18 [
Unlike the previous COCOMO2000 method, which relies on three input parameters, COSMIC FFP
relies on four: Entry, Exit, Read, and Write. It can be concluded that the functional size of the system
represents the total number of all used messages. The system can be viewed as a four-dimensional
vector space representing the total number of messages in form of: input data, reported data, written
data, or data read from files. The advantage of this method is its technology independence and absence
of an upper limit for the value of the functional quantity. Therefore, there is no saturation as the
complexity of functionality can increase indefinitely depending on the number of messages in the
system [
Functional size is determined based on a four-dimensional vector denoted as FFP as follows (8): = ( , , , ) (8) whereFFP represents the total number of messages in the entire observed system and is calculated as the norm of the vector, formula (9):
⃗ = + + + (9) where E = Entry, X = Exit, W = Write, and R = Read.
The first constructed ANN architecture, in the COSMIC FFP approach, labeled as ANN-L12, is
based on Taguchi's orthogonal vector plan (L12) with 11 parameters ( , i=(1,11)) and two different
levels (L1, L2). The experiments are conducted with 12 ANNs candidates labeled as ANN1,..., ANN12
Figure 5. The second constructed ANN architecture, labeled as ANN-L36prim, is based on Taguchi's
orthogonal vector plan (L36prim) with 16 parameters ( , i=(1,16)) and combined two and three
different levels (L1, L2, L3). The experiments are conducted with 36 ANNs candidates labeled as
ANN1,..., ANN36 Figure 6. In Table 3, basic statistics is provided for the datasets used in all three phases
of the experiment: training, testing, and validation. The table includes the dataset names, the number of
projects in each dataset, as well as the minimum, maximum, mean, and standard deviation values
expressed in Functional Size (FS) [
The estimated value of the UCP method utilizes twenty-one parameters for estimation, of which thirteen parameters represent the technical characteristics of the system, while the remaining eight are environmental factors. It is calculated based on G. Karner's formulas. The representation of actual effort using the UCP approach is a six-dimensional vector, where its value is calculated as the norm of the vector as follows, formulas (10), (11):
UCP = (UAW, UUCW, UUCP, TCF, ECF, AUCP) (10) ⃗ = + + + + + (11) where UAW is the unadjusted actor weight, UUCW is the unadjusted use case weight, UUCP is calculated as UUCP = UUCW + UAW, TCF is the technical factor, ECF is the environmental factor, and AUCP is calculated as AUCP = UUCP x TCF x ECF. Representation of the actual effort using the UCP approach as a four-dimensional vector is achieved calculating the value, as the vector norm, by formulas (12), (13):
UCP = (UAW, UUCW, TCF, ECF) (12) ⃗ = + + + (13) where UUCP= UAW+ UUCW, and AUCP= UUCPxTXFxECF.
In both cases, the Real Effort is obtained as the norm of the UCP vector and represents the actual
functional size or the number of use case points. This method is currently widely used for effort
estimation, although it is not standardized within the ISO standard like the previous two methods. Four
input variables, UAW, UUCW, TCF, ECF, or six input variables, UAW, UUCW, UUCP, TCF, ECF,
AUCP, are used as inputs for two different ANN architectures constructed based on Taguchi's
orthogonal vector plans, namely ANN-L16 and ANN-L36prim [
The first constructed ANN architecture, denoted as ANN-L16 in the UCP approach, is based on
Taguchi's orthogonal vector plan (L16) with 15 parameters ( , i=(1,15)) and two different levels (L1,
L2). The experiments are performed with 16 ANN candidates labeled as ANN1,..., ANN16 Figure 7. The
second constructed ANN architecture, denoted as ANN-L36prim in the UCP approach, is based on
Taguchi's orthogonal vector plan (L36prim) with 23 parameters ( , i=(1,23)) and three different levels
(L1, L2, L3). The experiments are conducted with 36 ANN candidates labeled as ANN1, ..., ANN36
Figure 8. Table 4 provides basic statistical data on the datasets used in all three phases of the experiment:
training, testing, and validation. The table includes the dataset names, the number of projects in each
dataset, as well as the minimum, maximum, mean values, and standard deviation expressed in Real
Effort (RE) [
The algorithm of robust experimental design involves the following steps: Step 1. Input values:
COCOMO2000 approach has three input values: X1=E, X2=PEMi, and X3=KLOC for all four proposed architectures: ANN-L9, ANN-L18, ANN-L27, and ANN-L36;
COSMIC FFP approach has four input values: X1=Entry, X2=Exit, X3=Read and X4=Write for both proposed architectures: ANN-L12 and ANN-L36prim;
UCP approach for the first proposed architecture ANN-L16 has six input values: X1=UAW, X2=UUCW, X3=UUCP, X4=TCF, X5=ECF and X6=AUCP, while for the second proposed architecture ANN-L36prim there are four input values: X1=UAW, X2=UUCW, X3=TCF and X4=ECF. Step 3. The sigmoid function, as the activation function of the hidden layer was used (15): Example: Hidden and output layer functions for ANN-L36prim architecture (16)-(19): − 36
1 =
1+ −(Y1∙W13+Y2∙W14+Y3∙W15+1∙W16).
In the first proposed ANN-L16 architecture, an orthogonal vector plan of two levels L1 and L2, and the initial values of the weighting factors Wi that take the values from the interval [-1, 1], were used.
The second proposed architecture has an orthogonal vector plan of three levels and the initial values
of the weighting factors Wi that take the values from the interval [-1, 0, 1]. For each subsequent iteration,
new weight factor values must be calculated as follows (e.g., for ANN-L16 architecture) [
For each subsequent iteration, the interval [-1, 1] is divided depending on the cost effect function
as follows [
The set of input values of each dataset converges depending on the value of the cost effect function. Step 4. Defuzzification of the input values, formula (22): = ( + ( )) ∙ (
− ),
OA(ANNi) = Xi , where i = 16, i = 36.
Step 5. Different evaluation metrics are used to validate the obtained results, formulas (23)-(28): where MRE refers to Magnitude Relative Error and MAE refers to Mean Absolute Error.
For each of the experimental part in every iteration, the Gradient Descent is monitored with the condition GA<0.01, calculated as (28): Step 6. Pearson’s, Spearman’s and R2 c2oe<ff0ic.i0e1n,ts arℎe monitored during the experiment, formula (29): = 1 − = 1, … ,
Additionally, Prediction at 25%, 30%, and 50% is the percentage of the total number of ANNs that meet the GA criterion (30). PRED(k) = count(MRE) < 25%, PRED(k) = count(MRE) < 30%, PRED(k)=count(MRE)<50%, 1, 0, ℎ ≤ ,
.
where k=25, k= 30, and k = 50 [
After conducting numerous experiments with different ANN architectures within the three approaches
and utilizing various activation functions, the authors in [
ANN-L9 ANN-L18 ANN-L27 ANN-L36 ANN-L12
ANN-L16 ANN-L36prim ANN-L9 ANN-L12
ANN-L18 ANN-L36prim
ANN-L27 ANN-L16
ANN-L36 ANN-L36prim
In the new COCOMO2000 model, four different neural network architectures were used, along with five datasets divided into three clusters, sigmoid activation function, fuzzification method, and Taguchi method for effort and cost estimation. The experiments showed that this approach guarantees reliable and stable results, as confirmed by monitoring the MMRE values. The convergence speed of the
UCP
ANN-L9 ANN-L18 ANN-L27 ANN-L36 ANN-L12
PRED(30) (%) 35.1 28.3 31.2 34.2 50.5 56.0 100.0 100.0
PRED(50)
(%)
66.8
58.2
54.8
57.4
81.2
87.0
100.0
100.0
architecture depends on the cost function and project nature, and the number of iterations is less than
10. More complex architecture exhibits faster convergence, shorter iteration time, and minimal MMRE.
Fuzzification as an encoding function along with clustering of input data proposed in [
The new COSMIC FFP model in this study demonstrates that the application of two different ANN
architectures, based on Taguchi Orthogonal Arrays, further reduces the MMRE value. This model
belongs to user-functional requirement-based approaches with four input values. By using data input
clustering methods from the ISBSG dataset and fuzzification as encoding method, the different project
structures are successfully controlled and mitigated [
The new UCP model utilizes two different ANN architectures and four distinct datasets, a sigmoid
activation function, fuzzification method, and Taguchi method for software development effort and cost
estimation. This model yields significantly better results compared to the previous two, as evidenced
by the MMRE value and convergence rate of each architecture. Based on the three parts of the
experiment, it is concluded that the ANN-L16 architecture converges after the fourth iteration, resulting
in an MMRE value of only 7.5%, which is 35.8% better than the first COCOMO2000 model. The error
value of the UCP model is 21.3% lower than the second proposed COSMIC FFP model. Both
architectures of this model demonstrate 100% accuracy in predictions. The advantages of this model
include a lower number of iterations (4-6), simple architectures, high coverage of different effort values,
and the lowest MMRE value of 7.5%. A potential drawback is the need for finding new methods to
further reduce of the MMRE. This model can be used independently or in combination with the previous
two, depending on the company's historical data. Although not standardized, it is increasingly used in
the software industry for effort estimation for software project implementations [
The proposed, new models in the studies [
[1] A Guide to the Project Management Body of Knowledge (PMBOK Guide). Third Edition, Project
Management Institute, Inc. 2004. ISBN: 1-930699-45-X. [2] B. W. Boehm, C. Abts, and S. Chulani. Software development cost estimation approaches-A survey. Annals of software engineering, 10 (1): 177-205, 2000.