The knowledge about the software metrics, which serve as quality indicators, is vital for the e cient allocation of resources in quality assurance activities. Recent studies showed that some software metrics exhibit threshold e ects and can be used for software defect prediction. Our goal was to analyze if the threshold derivation process could be used to improve a standard classi cation models for software defect prediction, rather than to search for universal threshold values. We proposed two classi cation models based on Bender method for threshold derivation to test this idea, named Threshold Naive Bayes and Threshold Voting. Threshold Naive Bayes is a probabilistic classi er based on Naive Bayes and improved by threshold derivation. Threshold Voting is a simple type of ensemble classi er which is based solely on threshold derivation. The proposed models were tested in a case study based on datasets from subsequent releases of large open source projects and compared against the standard Naive Bayes classi er in terms of geometric mean (GM) between true positive and true negative rate. The results of our case study showed that the Threshold Naive Bayes classi er performs better than the other two when compared in terms of GM. Hence, this study has shown that threshold derivation process for software metrics may be used to improve the performance of standard classi ers in software defect prediction. Future research will analyze its e ectiveness in general classi cation purposes and test on other types of data.
11:2 ferent projects and do not represent a reliable predictor for software defects when used alone. However, within the same project and project release, a stable threshold level can be found. Thus, we concluded that rather than searching for universal threshold values, threshold values should be used in a different way to aid software quality assurance. In this paper we investigate that idea and our main research question (RQ) is: —RQ: Does the threshold derivation process have the potential to improve a standard classifier for the defect prediction purpose?
We proposed two novel algorithms based on the threshold derivation process, namely Threshold
Naive Bayes (TNB) and Threshold Voting (TV). The proposed TNB algorithm enhances the standard
naive Bayes classifier using the calculated levels of thresholds for SDP, while TV is based on the
percentage of metrics which exceeded the threshold level. We used the naive Bayes model as a base
for our TNB as it is one of the most common classification techniques successfully used in various
application domains [Z
In this paper, we present a case study which compares our two proposed algorithms, TNB, TV with
the standard Naive Bayes (NB) in terms of geometric mean (GM) between true positive rate and true
negative rate. Our algorithms use the ensemble of many metrics predictions for prediction of the fault
prone code, rather than using single metric prediction, therefore they provide more accurate results
for SDP. The algorithms are tested on 10 different dataset versions using 10 fold cross-validation to
overcome t
The rest of the paper is organized in the following way: related work that motivated this study is presented in Section 2; the research methodology is described in Section 3; the details of our case study are given in Section 4; the results are presented and discussed in Section 5; and the paper is concluded in Section 6.
Many different machine learning models have been used for fault proneness prediction. A study that
benchmarked different classification models for SDP showed that the logistical regression model and
naive Bayes based model acquired good re
Moreover, recent studies focused more on software metrics threshold levels using ROC curves and
Alaves r
The methodology for the construction of threshold derivation process based classifiers is schematically presented in Fig. 1. The threshold derivation is presented in the upper part, while the lower part presents our two proposed algorithms. The initial step of presented methodology is to divide the data in 10 parts of equal size and class distribution for 10-fold cross-validation using stratified sampling method. These parts are then used as follows: 7 parts are used for training, 2 for validation and 1 part for testing purposes. Two classifiers based on the threshold derivation method are proposed by this methodology, named Threshold Voting (TV) and Threshold naive Bayes (TNB). The confusion matrices are calculated for each algorithm for performance comparison. Further details about the proposed methodology are described in the next few subsections.
The threshold is derived by using the univariate binary logistic regression model, a classification technique in which one dependent variable can assume only one of two possible cases. The probability for the occurrence of each case is defined by the logistical regression equation:
P (X) =
e 0+ 1X
1 + e 0+ 1X
where P(X) is the case probability, X represents the metric value, and 0 and 1 are the logistic
regression coefficients. The coefficients are calculated for every metric separately. The logistic regression
coefficients along with the percentage of the majority class p0 are used by the Bender’s method for
th
V ARL =
(ln( 1 p0 )
0)
1 1 p0
where 0 and 1 are logistical regression coefficients and p0 is the base probability. The ratio of the
faulty modules is used as the base prob
Afterwards, the algorithm iterates through the validation data part and compares the calculated threshold levels with real metric values. If the metric value exceeds the threshold level for a given instance, the individual metric prediction classifies that instance as fault-prone and otherwise as non fault-prone. This step is repeated for every metric and the results are many individual software defect predictions based on metric values. Our proposed classification algorithms, TNB and TV, are trained based on the mentioned individual metrics predictions.
3.2.1 Standard naive Bayes Classifier . The Naive Bayes Classifier is a probabilistic multivariate classifier based on the Bayes theorem. In our case, the multiple attributes correspond to different software metrics and software modules, respectively. Using the conditional probability P(EjH), we can calculate the probability of an event E using its prior knowledge P(H):
Naive Bayes classifier calculates these probabilities for every attribute. This step trains the naive Bayes model which is afterward used for prediction. The mentioned model is trained on the raw data which is a combination of the training and validation data part. After the conditional probabilities are calculated they are used in Naive Bayesian equation:
P(H jE) =
P(E jH) * P(H)
P(E) hbayes = argmaxP (H)P (EjH)
n = argmaxP (H) Y P (aijH) i=1 (1) (2) (3) (4) to calculate the posterior probability (MAP estimation) for each class. The class with the highest posterior probability is the outcome of prediction which in our case predicts the fault-prone code or non fault-prone code. These predictions are being made on the testing data part which is also used for other two algorithms.
3.2.2 Threshold naive Bayes. Our proposed classification algorithm TNB is based on the naive Bayes Classifier described in the previous subsection. The difference is that it is being trained on individual metric predictions mentioned in the subsection 3.1, rather than on the raw metrics data as explained in previous subsection 3.2.1
The training of the TNB model is based on the individual predictions, i.e. calculated threshold levels from the validation data part. These individual metric predictions are evaluated using the evidences of the fault prone code in validation data part which derives 4 different conditional probabilities for every metric (attribute): —probability that the metric predicted that there will not be a defect in code under condition that the defect actually did not happen P (Xi = 0 j Y = 0), —probability that the metric predicted that there will not be a defect in code under condition that the defect actually happened P (Xi = 0 j Y = 1), —probability that the metric predicted that there will be a defect in code under condition that the defect actually happened P (Xi = 1 j Y = 1) and —probability that the metric predicted that there will be a defect in code under condition that the defect actually did not happen P (Xi = 1 j Y = 0), where Xi represents the individual metric prediction of a fault prone code and Y represents the evidence of a fault prone code which is stored in the last column of the validation dataset. These probabilities correspond to the number of cases in which a certain condition (eg. Xi=1, Y=1) happened. Therefore, by calculating the percentage for each of the four mentioned conditions and repeating this step for every significant metric, a model similar to NB is trained. Consequently, the mentioned validation data part is actually used as the second training part for our model.
Finally, the TNB predicts every instance of testing data part using these calculated conditional probabilities and the calculated threshold levels as fault-prone or non fault-prone. The prediction process starts by comparing the real metric values with the calculated metrics threshold levels which generates the individual metric predictions for the testing data part. This step is similar to the step where conditional probabilities are being calculated. The difference is that these individual metrics predictions are now being used in next step along with calculated conditional probabilities to finally predict the class of an instance. It is done by multiplying trained conditional probabilities for every metric which correspond to the P (aijH) from the naive Bayes equation (4), iteratively through metrics in one row. The individual metrics predictions correspond to the ai and the percentage of a class correspond to P(H). The result of these steps are two posterior probabilities (MAP estimations) for every file (posterior probability that the instance is class 1 and posterior probability that the instance is class 2). These probabilities are then compared mutually and the decision is made based on which of the two is more likely.
3.2.3 Threshold voting. Our second proposed classification algorithm, named Threshold Voting also relies on individual metrics predictions based on the calculated threshold levels. Threshold levels are calculated based on training data part and the validation data part combined into one dataset as it does not need the validation part. This step trains the TV classifier model. Afterwards, these calculated threshold levels are being compared with the real metric values from the testing data part which generates the individual metric predictions. Finally, it calculates the percentage of the individual metrics predictions which predicted class 1 and the percentage of the individual metrics predictions which predicted class 2. The final classification is based on majority vote for these two percentages, i.e. the higher percentage predicts the instance as fault-prone or non fault-prone.
Matlab 2016 environment was used to develop the algorithms and to perform this case study. The details of the study will be explained in the next few subsections.
The case study is based on empirical SDP datasets from 5 subsequent releases of two open source
projects of Eclipse community, JDT and PDE. The chosen datasets are carefully collected, structured
in a matrix form and formated as csv (comma-separated values) files and open to public use [M
The case study is based on 10 fold cross-validation, so the datasets are divided into 10 parts using the stratified sampling technique. We combined 7 parts of the dataset as the training set, 2 parts as the validation set, and 1 part as the testing part. This process is repeated in 10 iterations, changing the order of parts used for training, validation and testing in every iteration.
After the algorithms have been trained and tested, we evaluated their performance of defect prediction. Since the stratified sampling has been used to divide the data, the percentage of the two classes is kept within all the data parts, therefore all the classifiers are fairly compared with the base class probability (p0) being provided for all three classifiers. Comparing the predicted class with actual class value for each java file within the testing part, we computed the elements of confusion matrix. Confusion matrix consists of four possible outcomes: true positive (TP) and true negative (TN) for correct classification and false positive(FP) and false negative(FN) for incorrect classification.
Using the four possible outcomes, we computed evaluation measures standard for SDP: true positive
rate (TPR), true negative rate (TNR) and geometric mean (GM) using Equations 5, 6 and 7. TPR and
TNR represent the accuracy of the positive and the negative class of data, respectively. GM represent
the geometric mean between TPR and TNR and it usually used to evaluate the performance of binary
classification in presence of unbalanced datasets [Japkowicz a
T P R = T N R =
T P T P + F N
T N
T N + F P GM = pT P R T N R (5) (6) (7)
We analyzed the software defect prediction performance of the three classifiers using the real data and we computed the confusion matrices. The arithmetic mean of TPR, TNR and GM values for every dataset release is presented in Table II. Boxplots of the GM results is given in Figure 2 for a more thorough representation. TNB results are colored green, TV results are colored red and Standard NB results are colored black and dataset releases are separated with the vertical line as every release was tested separately.
The GM results have shown that the TNB classifier performs better than the NB classifier for all dataset releases. For JDT project releases the prediction improvement over the NB is in range from 11% to 22%, while the improvement for PDE releases is in range from 5% to 10%. The TV also performed better than the NB for most releases, between 1% - 14% for JDT and between 2% - 8% for PDE. The overall improvement in terms of GM is the result of a higher TPR for our two proposed algorithms, which is very important for the good SDP. The NB has very high TNR because it predicts most instances as non fault prone but in the same time it gives very low prediction accuracy of the fault prone instances, which is not very useful for the quality insurance. Our proposed algorithms have slightly lower TNR than the NB but substantially higher TPR which gives better overall results. Finally, our TNB classifier also performs better than our second proposed algorithm, TV. When compared mutually the improvement of TNB over TV is between 1% - 11%.
The goal of this study was to analyze if the threshold derivation process could be used to improve the standard classification techniques for software defect prediction. We proposed two novel algorithms 11:8 based on threshold derivation of software metrics, namely Threshold Naive Bayes - TNB and Threshold Voting - TV. The proposed algorithms were compared against the standard naive Bayes classifier, which inspired us to develop the TNB classifier.
The main contribution of this study is the new approach and methodology for improvement of the standard classification techniques using the threshold derivation. We hope that our approach will serve as guidelines for future related research in the SDP research area. The conclusions we obtained with this study are: —Threshold derivation method may be used for improving the performance of standard classification models for SDP. —The improvement of the proposed TNB classifier over the standard NB in terms of GM ranges from 3% to 22%, and it gives the best results for SDP overall. —The improvement of the proposed TV classifier over the standard NB in terms of GM ranges from 1% to 14%. —The standard, unmodified naive Bayes algorithm does not give very good result for SDP, because it has a very low TPR.
Conclusion validity of this research is strong because we used standard techniques for sampling and statistical analysis. The used methodology is explained in details and the case study is based on publicly available datasets so the whole process is easy to replicate. However, our conclusions are based on a small scale case study and this limits their external validity. This study should be extended on a larger scale for more general conclusions and our future work will replicate it on additional SDP datasets.