Successful prediction of defects at an early stage is one of the main goals of software quality assurance. Having an indicator of the severity of defect occurrence may bring further bene t to allocation of testing resources. This paper is a part of a project aimed at identifying the role of di erent software metrics in order to improve the software quality assurance activities. In a preliminary case study we analyzed the relationship between the software metrics and defects using fundamentally di erent approaches for feature selection. Our case study showed that some metrics do not indicate defect occurrence, several of them exhibit moderate level of correlation, and the choice of the appropriate metrics is biased by the choice of feature selection technique. The next step would be to integrate the nding from di erent approaches and various datasets to develop a hybrid method for a precise de nition of software metrics and their threshold levels that are good indicators of defect occurrence.
9:2 learning techniques. The problem that occurs is that different code smell detection tools, as well as metrics tools/programs, give significantly different results, resulting in different perception of code smells [Paiva et al. 2017].
This paper is a continuous effort on analyzing code smells and their impact on software quality [Gradisˇnik et al. 2019b; Gradisˇnik et al. 2019a]. The aim of this paper is to address the variability of object-oriented software metrics’ importance for detecting the problematic code by investigating the relationship between faults and code smells, i.e. software components with technical debt. The relationship is analyzed on 5 subsequent releases of Eclipse JDT open source project written in Java by using four different approaches: A vast number of collected and stored attributes or features is nowadays presenting more disadvantages than advantages. One of the biggest reasons for that statement lies in the problem of building a suitable classifier based on the whole set of features, and related significant time complexity [Brezocˇnik et al. 2018]. To overcome such problems, researchers often employ some feature selection methods.
Too many independent variables can also have negative effects on model’s fault-proneness prediction, making the model more dependent on the data set currently in use and therefore less general [Briand et al. 2000]. Selecting the appropriate measures to be used in the model requires a strategy of minimizing the number of independent variables in the model. This paper investigates the usage of static code attributes as independent variables and univariate forward and backward step-wise selection principles in choosing the appropriate ones. By finding the subset of the most relevant measures, we might also gain deeper understanding of the relations between software metrics and defect occurrence. The model performances are evaluated using widely used performance measures such as accuracy, sensitivity and false alarm rate [Zhou and Leung 2006; Guo et al. 2004]. A defect prediction model should identify as many fault prone modules as possible while avoiding false alarms [Lessmann et al. 2008].
Some studies analyzed the relationship between software metrics and defects and found that certain metrics do exhibit strong association. An early study by Shatnawi et al. [Shatnawi 2010] and a recent study by Arar et al. [Arar and Ayan 2016] analyzed several object-oriented metrics like coupling between objects (’CBO’), response for class (’RFC’), weighted method per class (’WMC’), depth of inheritance tree (’DIT’) and number of children (’NOC’) . The results of their analysis showed that ’CBO’, ’WMC’ and ’RFC’ metrics are the ones that have stronger association to software defects that the remaining ’DIT’ and ’NOC’ metrics. The replicated study by Arar et al. [Arar and Ayan 2016] included also other metrics like lack of cohesion between objects (’LCOM’), maximum and average cyclomatic complexity (’MAXCC’, ’AVCC’) and lines of code (’LOC’) and showed that these metrics may also prove to be significant for successful prediction models. However, the object-oriented programs can hold several types of software complexity [Seront et al. 2005] and hence, our project goal is to examine a larger number of software metrics aiming to find the appropriate Java code smell identification metrics.
The goal to find relationship between static code attributes and defect occurrence in the program code. Four different techniques are described in the following subsections.
Correlation analysis is a standard statistical approach to establish whether there exists a relationship between two variables. As the values within the variables are usually distributed normally, Pearson’s correlation is the standard technique. However, in software engineering this is rarely the case, so the Spearman’s correlation, a non-parametric variant, is used to express correlation [D’Agostino and Pearson 1973]. Spearman’s correlation coefficient (rs) measures the power of a monotonic relationship between paired data in a sample s, that consists of n observations, using the following expression [Myers and Well 1991]:
Parameter di presents the difference between the ranks of two variables at each observation i, yielding the values in range 1 rs 1 with interpretation similar to Pearson’s correlation [Lehamn et al. 2005]. Positive and negative values indicate correlation, whereas values close to zero indicate absence of correlation. The more distant the value is from zero, the stronger the correlation is, and it is interpreted according to the five levels of correlation strength, as represented in Table I [Evans 1996].
Algorithm BPSO+C4.5 [Brezocˇnik, Lucija 2017] uses a Binary Particle Swarm Optimization algorithm (BPSO) [Kennedy 1997] as a basis and extends it with Feature Selection mechanism. Since particles in the BPSO+C4.5 algorithm are moving in a binary search space, their initialization is done with binary values 0 and 1. Each particle represents a feasible solution in an n-dimensional search space. For example, a particle 1001, presents 4-dimensional space where only the first and the last features are included.
The particles are moving through iterations following two extremes. First one is individual extreme pBesti, which comprises the best-obtained position of each particle i in the n iterations pBesti = (pBesti1; pBesti2; : : : ; pBestin). The second extreme is called global extreme gBest which comprises the best-found solution so far in the entire swarm gBest = (gBest1; gBest2; : : : ; gBestn).
The movement of the i-th particle from old to new position in the search space is controlled by velocity vector Vi = [vi1; vi2; : : : ; vin] and by position vector Xi = [xi1; xi2; : : : ; xin]. Velocity is calculated according to the equation 2 and consists of three main parts: vidnew = ! vidold + c1r1(pBestidold xidold) + c2r2(gBestdold xidold) (1) where ! is the inertia weight, c1 and c2 are positive constants and r1 and r2 are two random functions in the range [0,1]. Inertia component ! vidold is responsible for controlling the velocity of each particle, cognitive component c1r1(pBestidold xidold) or particle memory tends to direct particles to their personal best positions, and the social component c2r2(gBestdold xidold) which tends to direct particles to the globally best position found so far in the entire swarm. After the velocity update, position adjustment is carried out with equation 3: (3) (4) xindew = (1; if 1 + e 1vindew > U (0; 1)
0; otherwise where e is the base of the natural logarithm and U (0; 1) is the uniform distribution.
Logistic regression is a statistical modeling method used to estimate the probability of occurrence of an event, referred to as the dependent variable. Depending on the number of independent variables that are fitted to a logistic function, there are multivariate and univariate models, the later containing only one variable. Logistic regression is suitable for binomial dependent variables and hence it is an appropriate model for software defect prediction. In comparison to other related statistical techniques it is more flexible and robust [Hamadicharef et al. 2008; Tabachnick et al. 2007]. It does not assume normal distribution or equal variance within independent variables, nor linear relationship between the dependent and independent variable. These characteristics make logistic regression a commonly used classification algorithm.
The logistic regression models the probabilities of two different classes, ensuring the sum of probabilities equals 1 using the following equation [Hastie et al. 2009]:
P (X) =
e 0+ 1 X 1 + e 0+ 1 X where 0 is the free coefficient and 1 is the regression slope coefficients for dependent variable X. In case of software defect prediction, X represents a software metric and P (X) is the fault-proneness probability. The coefficients 0 and 1 define the curvature of the non-linear logistic regression output curve and they are estimated by the maximum likelihood procedure.
A likelihood ratio chi-square test is usually used to assess the statistical significance of each independent variable in the model. The null-hypothesis is: there is no relationship between the logistic regression model and the dependent variable, i.e. the true coefficients are zero. Finally, if the level of statistical significance p-value is below 0.05, the dependent variable is considered to be significant for prediction. In univariate logistic regression model, each software metric is estimated separately and this is another way to find the ”appropriate indicators of software quality”.
Bender method is a threshold derivation method based on the univariate logistic prediction model [Arar and Ayan 2016], graphically presented in Figure 1. The method itself consists of three phases: sampling the data according to the stratified 10-fold cross-validation, training the univariate logistic regression model, and computing the threshold level for all the metrics which are statistically significant for prediction in the logistic model. The fourth phase is the testing phase in which threshold-based defect prediction models are built and their effectiveness is evaluated in terms of the geometric mean (GM) accuracy measure. In these models, an instance is declared fault-prone should a metric level go beyond the calculated threshold. majority class percentage
p(0)
Data 10 fold cross-validation 9 folds
Train 1 fold
Test
GM Results
The univariate logistic regression model described in previous subsection is used in the same way in this method. After performing the analysis of statistical significance, the free coefficient 0 and the regression slope coefficient 1 are passed along with the the base probability p(0) to compute the threshold level. As shown in figure 1, the percentage of majority class is used as the base probability [Arar and Ayan 2016], thus taking into account the probability that a randomly selected instance belongs to the majority class. The value of an acceptable risk level, i.e. the threshold value for each metric T HR is computed using the following equation [Bender 1999]:
The effectiveness of calculated threshold levels for each metric is evaluated in the last phase. A simple decision tree is built using the obtained threshold levels to classify the software modules from the testing dataset according to the following equation:
T HR =
(ln( 1 1 1 p0 p0 )
0); Yi = f aulty non f aulty if Xi > T HR if Xi T HR ; where i represents the i th software module in the testing dataset, Xi represents the actual value of software metric and Yi represents the prediction output. After comparing the prediction results against the known values of defect proneness in the testing set, GM is calculated as:
GM = pT P R T N R where true positive rate (TPR) and true negative rate (TNR) represent the accuracy of positive (faulty) and negative (non-faulty) class, respectively.
T P
T P + F N
This section presents preliminary results of a case study used to compare the four fundamentally different approaches which we used for scoring the importance of software metrics for defect prediction. Each of these four approaches has been used to perform some sort of feature selection, i.e. to find the important indicators of software quality. Only the BPSO method is designed to perform pure feature selection based on wrapper method, whereas other methods are used for different purposes. The aim of this comparison is to estimate their degree of overlapping and open new research questions about this issue.
The datasets used in this case study are five consecutive releases of Eclipse JDT open source project (2.0, 2.1, 3.0, 3.1 and 3.2), systematically collected by the BuCo Analyzer tool [Mausa and Grbac 2016]. The datasets contain 50 different software metrics and the number of defects for each java source code file in the project. The full list of metrics and their descriptions may be found in [Mausˇa and Grbac 2017]. The software metrics that are computed describe static attributes like size, complexity, various object-oriented principles, design characteristics, programming style and more. The datasets are also available on-line for the whole research community1.
Table II presents full list of metrics from software defect prediction datasets and how well they scored in four different techniques used to analyze their importance for defect prediction. The correlation analysis computed the Spearman correlation coefficients (rs), the BPSO algorithm returned the percentage of including a metric (Rate), Logistic regression returned the percante of finding a metric significant for prediction of defects (Rate) and Bender Method yielded the geometric mean accuracy value of how successful a metric’s threshold is for finding defects (GM). There is no rule on how to interpret the inclusion rate given by BPSO and Logistic Regression, nor GM value of threshold values calculated by Bender method. In order to have a common baseline of comparison, the standard interpretation levels of correlation analysis, given in Table I, were used for all four approaches. This kind of interpretation seams understandable for interpreting the Rate given by BPSO and Logistic regression and it is in accordance with the interpretation of GM values obtained by Bender method. The interpretation is given in the right sub-column of each technique.
There are only a few metrics for which all techniques agreed about the level of their relationship with defects. For example, LCOM and cohesion (COH) exhibit low or very low relationship while efferent coupling (FOUT) and specialization index (SIX) exhibit low or very low relationship for all techniques besides the BPSO. On the other hand, a greater number of metrics have been found to have moderate or strong relationship, like: AVCC, HBUG, HEFF, UWCS, RFC, LMC, HVOL, EXT, TCC, NCO and CCOMHLTH. The dissagreement between the techniques is present for metrics like INST, PACK, CBO, MI, CCML, NLOC, F IN, R R and HIER where correlation analysis and BPSO indicate lower level whilst Logistic regression and Bender method indicate stronger levels. The disagreement in the opposite direction is present for metrics like S R, NSUB, where correlation analysis and BPSO found a stronger level of relationship than the other techniques. This means that the choice of the appropriate software metrics is biased by the choice of feature selection. 1http://www.seiplab.riteh.uniri.hr/?page id=834
Further analysis of the obtained results have shown that the relationship of certain metrics and defects exhibits uniform level of strength within different datasets. The correlations analysis is a more strict technique, rarely indicating strong and never indicating very strong relationship. Hence, the moderate level of relationship indicated by the correlation analysis has a greater weight that the same level of relationship indicated by the other techniques. The only two metrics that have strong relationship according to the correlation analysis are McCabe complexity (MVG), header comments (HCLOC) and depth of inheritance tree (DIT). It is interesting to notice that HCLOC and DIT exhibit such relationship when analyzed by BPSO and correlation analysis, but opposite levels of strength when analyzed by Logistic regression or Bender method and the agreement of all four techniques is achieved only for MVG.
Unlike the correlation analysis, the uniform Logistic regression technique is a less strict one and it indicated very strong level of relationship for most metrics. The Bender method, which is based upon the Logistic regression, and BPSO are moderately strict techniques. The distribution of the levels of relationship found by these two techniques is closer to a normal distribution than for the other two techniques.
The validity of this small scale case study is strongly affected by the choice of data. This is a first preliminary study, so it is based only on a sample of data from five consecutive releases of an highly object-oriented open source project from Eclipse community. The datasets are chosen from big and complex software projects to resemble as much as possible the industrial projects to cope with construct validity. The external validity is clearly threatened and general conclusions cannot be drawn yet. The aim of this case study was more to open new research questions and motivate the research direction of future work. Projects from different background, like different communities, development methodologies or written in different programming language need to be included to obtain more general conclusions.
The four chosen techniques which were used to analyze the relationship between software metrics and defects are a threat to internal validity. There exists a number of other techniques of similar purpose, but these are chosen because correlation analysis and logistic regression are the well known and widely used ones, Bender method is a rarely used technique with completely different aim (finding threshold levels) and BPSO is a novel hybrid technique for feature selection. At this stage, the conclusions about the relationship levels between analyzed metrics and defects still lack a precise explanation, hence threatening the conclusion validity and the their causality is unknown and open to speculations. That is why we believe this project to be an important one, and giving precise answers to our research objectives may bring significant improvement in understanding the role of different software metrics and improving the software quality assurance activities.
The algorithms that implement the proposed methodology have been applied to 50 different software metrics that are present in the datasets. The moderate level of relationship is an indication enough that a metric may improve the classification of defective software modules. The metrics for which at least one technique found a stronger level of relationship is thus a potential indicator of code smell. However, this may be misleading since different techniques find different metrics to be important. Hence, it is important to continue this research direction and find stronger evidence to precisely defining software metrics and their threshold levels that are good indicators of code smell. If such metrics are to be found, it would be possible to pay more attention to them in the whole life cycle of software development and reduce the possibility of code smell turning into defects.