Introduction

Machine learning methods, specifically classification algorithms, are used to automate the process of the construction objects' technical inspection. Multiple classification model are built to put the objects depending on its' technical condition into one of four categories: "1" -normal; "2" -satisfactory; "3" -unfit for normal operation; "4" -emergency [1][2]. The ability of the classification model to determine correctly the technical conditions of the construction object is evaluated by metrics [3][4][5][6]. There are four metrics which are commonly accepted among the scientists, which formulas are the functions of the cells' values of a confusion matrix Table 1 (in case of supervise learning) or a matching matrix in the case of unsupervised learning:

1. Sensitivity (SEN) and Specificity (SPE) -the metrics to assess the model's ability to correctly identify positive and negative cases, respectively. 2. Positive Predictive Value (PPV) and Negative Predictive Value (NPV) -the metrics assess the model's error for positive and negative cases, respectively. Recently, are submitted the studies which are showing that F-measure may not accurately reflect the model's true ability to classify objects as expected, and it is recommended to use the Matthews method for the evaluation instead [7][8]. However, such recommendation could not be used to evaluate model for prediction a construction object technical condition. Let's consider the confusion matrices of a binary classification model, create to categorize objects: "positive"the objects fit for normal operations; "negative" -the objects unfit for normal operation. In Table 2 is considered 3 possible results of a binary classification and the notation to read Table 2 is:

1. True Positives (TP) -the number of "fit" objects whose conditions are correctly identified by the model. 2. True Negatives (TN) -the number of "unfit" objects whose conditions are correctly identified by the model. 3. False Positives (FP) -the number of "unfit" objects which are incorrectly identified by the model as "fit". 4. False Negatives (FN) -the number of "fit" objects which are incorrectly identified by the model as "unfit". In row 1 of Table 2, is seen that the model incorrectly classified only one object, so model's score is high (>0.9) by all metrics (refer to Table 3). In row 2 of Table 2 -the model has a weak predictions in positive class PPV=100/(100+1000)=0.09 because 1000 "unfit" objects were treated as "fit" but model's ability to predict correctly negative is still high so NPV=30,000/(30,000+20)=0.99. As the result, the model score by normalized MCC is low but still may be accepted, however F-measure indicates the model's failed. In row 3 of Table 2, the model incorrectly classified 10,000 "fit" objects as "unfit", i.e. the model has a low ability to predict negative class as a result NPV=9/(9+10,000)=0.0 is low, how model's has a good prediction in positive class PPV = 9000/(9000+1)=1. As a result, F-measure score is high, but normalized MCC score dropped. Since an effective binary classification model for prediction a construction object' technical conditions may make a mistake when "fit" object is considered as "unfit" (like 3) but must not consider "unfit" as "fit" (line 2), so value of F-measure is important and could not be ignored as recommended in [7][8] as in line 2 F-measure is much lower than MCC's value.

In current paper, will be proposed an approach to evaluate and recommend which nMCC or F-measure score to use while evaluation a binary classification model for predicting the technical conditions state of the construction object.

Literature Review

The relations between MCC and F-measure in researches [7][8] were studied with the values of TP, TN, FP, FN from the confusion matrix which is received when threshold equal to 0.5. However, cut-off threshold τ=0.5 is not obviously optimal and may never be selected by data scientists to get model's predictions for the given dataset. Therefore, the recorded differences in the scores of the evaluation metrics may not happen when the selected threshold corresponds to optimal cut-off point on ROC curve or difference's scale may be small. On Figure 1 is illustrated how the difference in the scores of normalized MCC and F-measure is changing depending on the ROC's curve threshold, where curves ware constructed with the result of 4 machine learning classification algorithms GaussianNB; Random Forest Classifier; Logistic Regression; Support Vector Classifier learned on the same data set and scaled by method specified in [9]. In our approach, instead of using the constant threshold of ROC curve to construct a confusion metric and get the values of normalized MCC and F-measure we will find an optimal point of ROC curve and use it to design a decision support algorithm.

Research Materials

Youden Index is frequently used to identify an optimal point of ROC curve [10][11]. When both SEN and SPE are given the same priority then Youden is calculated as the maximum difference between TPR and FPR (1).

Youden=max τ {SEN +SPE−1 }=max τ {TPR+1−FPR−1}=max τ {TPR−FPR }=max τ { TP TP+ FN − F P F P+TN } ,(1)

In equation ( 1), TP; FP; FN; TN are cell values of a confusion matrix, which values are used to calculate marginal values in Table 1, so that P j = TP+ FP -is the number of positive instances which were predicted by model; N j = FN+TN is the number of negative instances which were predicted by model; P i =TP+FN is the number of positive instances in dataset; N i =FP+TN is the number of negative instances in dataset.

At the same time the maximum Youden score can be achieved also when FP is zero. In this case, according to Table 1 TP=P j and (1) is rewritten as (2).

Youde n fp−¿ 0 = P j P i ,(2)

As both equations ( 1) and ( 2) express the maximum value of Youden then they both equally identify the optimal point of the ROC curve, so in our proposed approach we take equation (2) and express MCC and F-measure using Youden fp → 0 . MCC equation (3) when FP is zero is rewritten as equation ( 4) according to which MCC is a function of two arguments: maximized Youden and a ration of the negatives in dataset to predicted negatives.

MCC= TP⋅TN−FP⋅FN√ (TP+FP )(TP+FN )(TN +FP )(TN +FN ) (3) MCC fp−¿ 0 = P j ⋅N i √ P j ⋅N i ⋅P i ⋅N j = √ P j ⋅N i P i ⋅N j = √ P j P i ⋅ √ N i N j = √ Youden fp−¿ 0 ⋅ √ N i N j (4)

F-measure equation ( 5) when FP is zero is rewritten as equation ( 6) according to which Fmeasure is a function of maximized Youden.

F 1 = 2TP 2TP+ F P+ FN (5) F 1fp−¿ 0 = 2 P j 2 P j + P i −P j = 2 P j P j + P i = 2 1+ 1 Youden fp → 0 (6)

Having formally defined MCC and F-measure via maximized Youden gave us possibility to obtain objective differences in metric scores and use them in the proposed approach to select the most appropriate metric (Figure 2).

The algorithm includes 3 blocks: 1. "Define and Calculate measures"; 2. "Assess Significance of Differences"; 3. "Rank Metrics Based on Significance". In the 1 st block are executed calculations of the metrics and their differences compared to Youden are are saved. In the 2 nd block -the differences saved as a result of execution of block 1 one by one is compared with the defined in block 1 the maximum expected difference, which is denoted as "threshold". When the difference is less then the "threshold" it is marked as "notSignificant". In last block all differences which are marked by flag "notSignificant" will be ranged and the metric with the smallest difference value will proposed by the algorithm. In case, when the algorithm did find any metric with flag "notSignificant" it will stop with no suggested metric to use.

Conclusions

In the current research the formal descriptions of the metrics: MCC and F-measure were described as the functions of Youden index, which is calculated on the confusion matrix when a binary model doesn't make a mistake in predicting negative instances.

The obtained definitions gave us the possibility to calculate unbiased differences in the metric pair scores instead of using the differences which are received when ROC curve cut-off is equal to 0.5, which is not obviously optimal and may never be selected by data scientists to get model's predictions.

The defined calculations are included in the proposed algorithm to support the scientists to select a metric in order to evaluate a binary classification model for predicting the technical conditions state of the construction object.

The novelty of the proposed approach compared to recommendations from [8][9] is that it recommends the metric considering optimal cut-off point of the ROC curve, so its recommendation depends on the leaning algorithm's results, so it improves the accuracy of model's evaluation. Further work will be to extend the proposed algorithm by including more measures, such as balanced accuracy, diagnostic odds ratio (DOR) and others.