A classification model which is designed to predict technical conditions of the construction object must not make the mistakes in categorizing “unfit” for normal operation objects as “fit” as such mistakes could lead to the accidents with a wide range of severities. So, to guarantee the mode's results are correct, the scientists are doing model's evaluation by the metrics, which high score is the indicator of model's ability does not make the mistakes. However, on the question which metrics to be use to assess the model is not received a single answer as the researches' conclusions often contradict each other while recommending the metric. The goal of this study is to propose the approach on how to select the metric to assess a binary classification model for predicting the technical conditions of the construction object. To meet the goal in the research Matthews Correlation Coefficient formula and F-measure were described using maximized Youden index, which value is possible to obtain when model doesn't make the mistakes when predicting negative instance. The results of this work will provide the scientist the decision's support method which recommendation depends on the optimal cut-off point on the ROC curve, so it improves the accuracy of the received evaluation score. .
Machine learning methods, specifically classification algorithms, are used to automate the
process of the construction objects’ technical inspection. Multiple classification model are built
to put the objects depending on its’ technical condition into one of four categories: "1" - normal;
"2" - satisfactory; "3" - unfit for normal operation; "4" - emergency [
Recently, are submitted the studies which are showing that F-measure may not accurately
reflect the model's true ability to classify objects as expected, and it is recommended to use the
Matthews method for the evaluation instead [
In row 1 of Table 2, is seen that the model incorrectly classified only one object, so model’s score is high (>0.9) by all metrics (refer to Table 3). In row 2 of Table 2 - the model has a weak predictions in positive class PPV=100/(100+1000)=0.09 because 1000 “unfit” objects were treated as “fit” but model’s ability to predict correctly negative is still high so NPV=30,000/(30,000+20)=0.99. As the result, the model score by normalized MCC is low but still may be accepted, however F-measure indicates the model’s failed. In row 3 of Table 2, the model incorrectly classified 10,000 “fit” objects as “unfit”, i.e. the model has a low ability to predict negative class as a result NPV=9/(9+10,000)=0.0 is low, how model’s has a good prediction in positive class PPV = 9000/(9000+1)=1. As a result, F-measure score is high, but normalized MCC score dropped.
Since an effective binary classification model for prediction a construction object’ technical
conditions may make a mistake when “fit” object is considered as “unfit” (like 3) but must not
consider “unfit” as “fit” (line 2), so value of F-measure is important and could not be ignored as
recommended in [
In current paper, will be proposed an approach to evaluate and recommend which nMCC or F-measure score to use while evaluation a binary classification model for predicting the technical conditions state of the construction object.
The relations between MCC and F-measure in researches [
Youden Index is frequently used to identify an optimal point of ROC curve [
TP TP + FN −
F P F P +TN
In equation (1), TP; FP; FN; TN are cell values of a confusion matrix, which values are used to calculate marginal values in Table 1, so that Pj= TP+ FP – is the number of positive instances which were predicted by model; Nj= FN+TN is the number of negative instances which were predicted by model; Pi=TP+FN is the number of positive instances in dataset; Ni=FP+TN is the number of negative instances in dataset.
At the same time the maximum Youden score can be achieved also when FP is zero. In this case, according to Table 1 TP=Pj and (1) is rewritten as (2).
Youde nfp−¿0= PP ij , (2) As both equations (1) and (2) express the maximum value of Youden then they both equally identify the optimal point of the ROC curve, so in our proposed approach we take equation (2) and express MCC and F-measure using Youdenfp→0.
MCC equation (3) when FP is zero is rewritten as equation (4) according to which MCC is a function of two arguments: maximized Youden and a ration of the negatives in dataset to predicted negatives. (3) (5) (6) 2 P j = 2 P j =
P j+ P i 1+ 2
1 Youdenfp→0 MCC fp−¿ 0 = √ P j⋅PNj⋅i⋅NPii⋅N j =√ PP i⋅j⋅NNji =√ PP ij⋅√ NNij =√Youdenfp−¿ 0⋅√ NNi
F-measure equation (5) when FP is zero is rewritten as equation (6) according to which Fmeasure is a function of maximized Youden.
Having formally defined MCC and F-measure viamaximized Youden gave us possibility to obtain objective differences in metric scores and use them in the proposed approach to select the most appropriate metric (Figure 2).
The algorithm includes 3 blocks: 1. “Define and Calculate measures”; 2. “Assess Significance of Differences”; 3. “Rank Metrics Based on Significance”. In the 1st block are executed calculations of the metrics and their differences compared to Youden are are saved. In the 2nd block – the differences saved as a result of execution of block 1 one by one is compared with the defined in block 1 the maximum expected difference, which is denoted as “threshold”. When the difference is less then the “threshold” it is marked as “notSignificant”. In last block all differences which are marked by flag “notSignificant” will be ranged and the metric with the smallest difference value will proposed by the algorithm. In case, when the algorithm did find any metric with flag “notSignificant” it will stop with no suggested metric to use.
In the current research the formal descriptions of the metrics: MCC and F-measure were described as the functions of Youden index, which is calculated on the confusion matrix when a binary model doesn’t make a mistake in predicting negative instances.
The obtained definitions gave us the possibility to calculate unbiased differences in the metric pair scores instead of using the differences which are received when ROC curve cut-off is equal to 0.5, which is not obviously optimal and may never be selected by data scientists to get model’s predictions.
The defined calculations are included in the proposed algorithm to support the scientists to select a metric in order to evaluate a binary classification model for predicting the technical conditions state of the construction object.
The novelty of the proposed approach compared to recommendations from [
Further work will be to extend the proposed algorithm by including more measures, such as balanced accuracy, diagnostic odds ratio (DOR) and others.