The need for successful software projects has been a major area of discourse amongst researchers and software developers in academia and software industry respectively. Failure of software projects has been tied to flawed estimation at the early stages of software development life cycle. Recently, soft computing techniques such as Fuzzy logic models has been seen as an alternative to handle uncertainties and vagueness of input parameters to the early software estimation models. In order to analyze the various conditions which affect estimation accuracy of fuzzy-based models, a sample of 93 COCOMO NASA projects was used to develop two groups of fuzzy models. One was the controlled group while the other was the experimental group varying in conditions of model structure, linguistic variables, parameters of input and output variables. A comparative analysis of the Mean Magnitude of Relative Error (MMRE) and Prediction accuracy Pred(l) evaluation criteria for the models was made and findings recorded. Results from the experiments show that the performance of a fuzzy-based software cost estimation model utilizing Takagi-Sugeno inference, Gaussian/Sigmoid membership function with more number of input variables and linguistics variables is more efficient. • Software and its engineering ➝Software system structures ➝Software system models ➝Model-driven software engineering • Computing methodology➝Artificial Intelligence Knowledge representation and reasoning➝Vagueness and fuzzy logic
Software engineering is the application of a systematic,
disciplined, quantifiable approach to the development, operation
and maintenance of software products [
In a recent update of Standish Group study (2012) conducted for
ComputerWorld, Standish examined 3,555 IT projects between
2003 and 2012 that had labour costs of at least $10 million and
found that only 6.4% of them were successful [
Estimation is a process which uses prediction systems and
intuition for resource and cost planning. It is controlled by cost
realism, which does not always insist on exactness but lays equal
emphasis on logic as much on the mathematical form of the
prediction system [
Software Cost Estimation (SCE) has been identified as one of the
most crucial activities in managing software projects required
during the early stages of software development life cycle [
The entire process of software cost estimation is generally
classified into two broad categories [
Measures provide a quantitative indication of amount, dimension,
capacity or size of a given attribute of a product. Metrics are a
quantitative measure of the degree to which a system, component
or process possesses a given attribute of a product. Measurement
is achieved as the result of the collection of a data point or more
data points. Software metrics can be defined as the continuous
application of measurement based techniques to software
development process and its products to supply meaningful and
timely management information, together with those techniques to
improve that process and its products [
Performance evaluation of estimates had always been a critical
stage in software development. Thus, proper and accurate the
evaluation criteria of software projects become, the better the
software cost estimation models. The mostly used software cost
estimation (SCE) criteria are listed in [
a. Magnitude of Relative Error (MRE): An MRE criterion is value error estimated for each of the projects compared to the actual. b. Mean Magnitude of Relative Error (MMRE): MMRE is used as the criteria error of the mean value of the project. c. Percentage Relative Error Deviation (PRED(l)): PRED(l) criterion is used in order to estimate the accuracy of the models.
MRE is evaluated as follows (for the i-th observation) | | MMRE is evaluated as follows (for n observations)
∑ Pred(l) is calculated as follows for n observations ( ) ∑ { (1) (2) (3) where n is the total number of projects and l is the number of projects with MRE less than or equal to l. However, a model estimate with low MMRE has a better performance index than one with high MMRE. A model with high Pred(l) has a better performance than one with a low Pred(l). A model with high VAF is better than one with a low VAF. Alternatively, a model showing low VARE has more accuracy than one with high VARE. Likewise a model having low MARE is preferable to one with high MARE.
Various fuzzy-based models for software cost estimation have
been proposed by researchers in [
Early algorithmic models which include COCOMO [
( ) ∑ ∏ where a = 2.94 and b = 0.9. Other algorithmic models include: Halstead Model Equation
Effort 5.2 * KLOC 1.5
Effort 5.5 0.73* KLOC 1.16 Effort 5.288* KLOC 1.04 (4) (5) (6) (7) Algorithmic models posses the following shortcomings; i. They make assumptions about the form of the prediction function. ii. They need to be adjusted or calibrated to local circumstances. iii. They are practically model-intractable. iv. They can’t handle the vagueness and uncertainty present in the input parameters to the model. v. The output from this set of models is almost crisp values which often lead to overconfidence in accuracy and precision of estimate. vi. Software cost estimation is a complex non-linear stochastic problem. These models can’t model nonlinear and complex problems.
Fuzzy logic is an extension of multi-valued logic that models
effectively how the human brain reasons. A systematic use of
fuzzy logic in software cost estimation is a necessity when the
available information is imprecise, incomplete or not totally
reliable [
A fuzzy model is a mapping between input and output spaces described using conditional propositions and inference engine. Fuzzy systems have two distinguishing features.
a. They implore non-linear mapping between input and
output vectors and can be accurately described using
mathematical formulae.
b. They are also knowledge-based driven expressed in
linguistic terms. These are known as intuitive systems.
However, humans can make inferences in an imprecise
environment and that is where fuzzy logic comes handy
[
The theory of fuzzy sets introduces a paradigm which extends the
concept of the crisp set, allowing objects to partially belong to a
set [
A linguistic variable can be defined as a variable that takes its value as terms defined by words in natural language. These terms are described by fuzzy sets defined in the universe of discourse in which the variable is defined. The number of linguistic variable was among the conditions that were varied during the experiments while others remain constant to analyze the effect on performance of estimate.
Fuzzy Inference System (FIS) is based on the generalization of the
classical inference rules to fuzzy logic. There are two basic FIS
[
It is formed by the following four components: A fuzzy rule base formed by a set of logical implications having the form described in Rule 1.
IF X1 is Aj,1 and ... and Xm is Aj,m THEN Y is Bj(Rule 1) where the propositions X1 is Aj,1,..., Xm is Aj,m; Y is Bj are all defined in the fuzzy sets Aj,1,..., Aj,m, Bj A fuzzy inference engine is defined using the sup-star operation. A fuzzifier operator and defuzzifier operator completes the system. b. Takagi-Sugeno fuzzy inference system
It is formed by the following components: A fuzzy rule base formed by a set of logical implications having the following form: IF f(X1 is Aj,1,...,Xm is Aj,m) then y = gj(x*1,...,x*n) (Rule 2) where x*1,...,x*n are crisp values on the input universes, X1 is Aj,1,...,Xm is Aj,m are fuzzy propositions defined on the same input universes by the fuzzy sets Aj,1,...,Aj,m, f(.) is a logical function, y is a crisp value on output universe and g(.) is a crisp function that implies the value of y when the inputs x*1,...,x*n satisfy the premise. An algorithm of reasoning which is a modified version of Mamdani FIS completes the system.
In summary, the Tagaki-Sugeno FIS method divides the input spaces in fuzzy subspaces and next builds a relation between input variables into each subspace. In the experiments, two groups of fuzzy models were used; one generated using the Mamdani and the other using the Takagi-Sugeno FIS type.
The only condition a membership function must really satisfy is that it must vary between 0 and 1. The function can be an arbitrary curve whose shape can be defined as a function that suits from the point of view of speed, efficiency and ease of use. The membership function maps an element, say X, to a membership value between 0 and 1. The eleven membership functions in fuzzy logic are derived from several basic functions namely; a. piece-wise linear functions b. Gaussian distribution function c. sigmoid curve d. quadratic and cubic polynomial curves These membership functions (MFs) include: trapezoid (trapmf), triangular (trimf), Gaussian (gaussmf), generalized bell (gbellmf), combination of Gaussian (gauss2mf), sigmoid (sigmf), pi-shaped (pimf), product of sigmoid (psigmf), difference between two sigmoid (dsigmf), s-shape (smf and z-shape (zmf). In this study, we carried out experiments using five membership functions (namely triangular, trapezoid, Gaussian, generalized bell and product of sigmoid) while keeping all other conditions constant.
It is the mapping from a fuzzy set (aggregate output fuzzy set), say S, defined on the universe of discourse, say U, to a crisp value x*ЄU. The defuzzifier’s main function is to ascertain an object x* that best represents the fuzzy set S. The aggregate of a fuzzy set includes a range of output values which must be defuzzified in order to achieve a single output value from the set. Five defuzzification techniques available include: smallest of maximum, largest of maximum, middle of maximum (the average of the maximum value of the output set), centroid and bisector.
The various experimental groups of models in this study are described below.
The controlled conditions for experiment 1 include: input and output variable, linguistic variable and value, membership function and defuzzification technique respectively. This experimental group utilized triangular membership, five linguistic variables (very low, low, nominal, high, very high), centroid defuzzification technique and fourteen effort drivers including Kilo line of code size (KLOC). The varied condition is in inference type as shown below.
a. Model with Mamdani inference system (Experiment model 1a) b. Model with Tagaki-Sugeno inference system (Experiment model 1b)
The controlled conditions for experiment 2 include number of input variables (fourteen effort drivers including Kilo line of code size (KLOC)), membership function (triangular), inference type (Mamdani inference) and five linguistic variables (very low, low, nominal, high, very high) respectively. The varied condition is in the defuzzification type as shown below.
a. Model with centroid defuzzification technique (Experiment model 2a) b. Model with bisector defuzzification technique (Experiment model 2b) c. Model with Middle of Maximum (MOM) defuzzification technique (Experiment model 2c) d. Model with Largest of Maximum (LOM) defuzzification technique (Experiment model 2d) e. Model with Smallest of Maximum (SOM) defuzzification technique (Experiment model 2e)
The defuzzification technique implored here is the middle of maximum (MOM). Membership function is the varied condition in this setup as described below.
a. Model with triangular membership function (Experiment model 3a) b. Model with trapezoid membership function (Experiment model 3b) c. Model with Gaussian membership function (Experiment model 3c) d. Model with generalized bell membership function (Experiment model 3d) e. Model with sigmoid membership function (Experiment model 3e)
In this setup, the controlled conditions for experiment 4 are product of sigmoid membership function, number of input variables (fourteen effort drivers including Kilo line of code size (KLOC)), inference type (Mamdani inference) and defuzzification technique (middle of maximum) respectively. The varied condition is linguistic variable. For experimental model 4a, five linguistic variables (very low, low, nominal, high, very high) are used while three linguistic variables (low, nominal, high) are used in experimental model 4b respectively.
a. Model with increased linguistic variables (Experiment model 4a) b. Model with reduced linguistic variables (Experiment model 4b)
All other conditions remained the same as in experiment 4a above except for the number of input variables to the model. For experiment 5b, ‘main memory constraint’ and ‘time constraint for CPU’ input variables were merged together prior to fuzzification because both had the same correlation coefficient respectively with effort. The same also was done for ‘analyst capability’ and ‘programmer capability’. The correlation coefficient between effort and these named drivers was approximately 0.35. Also ‘required software reliability’ and ‘process complexity’ were combined together because both had approximately same correlation (0.2) to development effort (output). Thus, the number of input variables in experiment model 5b reduced to twelve. a. Model with increased number of input variables (Experiment model 5a) b. Model with reduced number input variables (Experiment model 5b)
For this study, we made use of three evaluation criteria due to their widespread relevance in most related literature. They include a) Magnitude of Relative Error (MRE) b) Mean Magnitude of Relative Error (MMRE) c) Prediction accuracy criteria (PRED) There expressions are explained in equations 1 to 3.
Below shows the results from the validation of the experimental models using one third of the project data. From Table 4, the MMRE values for model 3a, 3b, 3c, 3d, 3e were 0.2623, 0.3176, 0.2549, 0.1894 and 0.2 respectively. The prediction accuracy were 48.38%, 54.83%, 70.96%, 67.74% and 67.74% respectively.
MMRE for Model 1 a
b
Experimental model 1 Pred(30) for Model 2 a b c d
e
Experimental model 2 MMRE for Model 2 a b c d
e
Experimental model 2 4.1
Pred(30) for Model 1 a
b
Experimental model 1 a b c d
e
Experimental model 3 MMRE for Model 3 a b c d
e Experimental model 3
Pred(30) for Model 4 MMRE for Model 4 a
b
Experimental model 4 MMRE for Model 5 a
b Experimental model 5 a
b
Experimental model 5 From figures 1 and 2, it is observed that in experiment 1 models, inference type had a great effect on performance of the fuzzy model. The model with Takagi-Sugeno inference performed better than the model with Mamdani inference in terms of mean magnitude relative error (0.0820 against 0.4958) and prediction accuracy (70.96% against 67.74%) respectively. Figures 3 and 4 shows that, no one choice of defuzzification technique enjoyed absolute preference in both evaluation criteria. While the models with centroid (67.74%) and bisector (67.74%) defuzzification technique show better predictive accuracy than the models with largest of maximum (54.83%), middle of maximum (48.38%), and smallest of maximum (38.7%) respectively. Alternatively, the models with smallest of maximum (0.1215) and middle of maximum (0.2623) defuzzification technique shows better results of mean magnitude relative error than models with bisector (0.4233), centroid (0.4958) and largest of maximum (0.6377) defuzzification technique respectively.
In terms of prediction accuracy from figure 5, the model with Gaussian membership function (70.96%) shows promising results than models with generalized bell (67.74%), product of sigmoid (67.74%), trapezoid (54.83%) and triangular (48.38%) membership functions respectively. From figure 6, it could be observed that generalized bell membership function shows better results of MMRE (0.1894) than product of sigmoid (0.2), Gaussian (0.2549), triangular (0.2623) and trapezoid (0.3176) membership functions respectively.
Another important observation from experiment 4 is that the number of linguistic variable used in a fuzzy model has effect on its performance. As seen in figures 7 and 8, the model with more linguistic variables (five) performed better in terms of MMRE (0.2) and prediction accuracy (67.74%) than the one with lesser (three) linguistic variable of MMRE of 0.8291 and prediction accuracy of 29.03%.
Also from figures 9 and 10, the model with increased number of input variables to the fuzzy model (fifteen) outperformed the one with lesser input variables (twelve) in terms of lower MMRE and better prediction accuracy respectively.
This paper explored the various factors that enhance high performance of fuzzy-based models for software development cost estimation. From the comparative analysis of fuzzy models used in the experiments, it could be observed that using TagakiSugeno inference type is preferred than Mamdani inference type for a fuzzy-based software cost estimation model. Also, increasing the number of linguistic variables and input variables to the fuzzy model has positive results on performance. Generalized bell, sigmoid and Gaussian membership functions performs better than triangular and trapezoid membership functions for this area of software engineering application.
The choice of a suitable defuzzification technique depends on whether smaller or larger projects are being modelled. While centroid and bisector defuzzification technique would favour modelling of medium software projects effort, smallest of maximum and largest of maximum defuzzification technique would show favourable results for small projects and large projects respectively. It is unlikely to achieve a fuzzy model which can give 100% Pred(30) but by suitably adjusting the values of the parameters in fuzzy inference system (FIS), estimated effort could be optimized.
Future research will involve a comparative study of fuzzy models using data from in-house software projects and also investigating the performance of fuzzy models with customized membership functions.
Special thanks to lecturers and postgraduate students of Computer Science department, Federal University of Technology Akure, Nigeria for their various inputs and suggestions to the success of this work, most especially Dr. (Mrs.) B. A. Ojokoh.