The basic data in the problem of the prediction of technical object's state of health based on the known indicators of its operation are the known results of the object state estimation by information about previous service. The problem may be solved using the machine learning methods, it reduces to binary classification of states of the object. The research was conducted in the Matlab environment, ten various basic methods of machine learning were used: naive Bayes classifier, neural networks, bagging of decision trees and others. In order to improve quality of healthy state identification, it has been suggested that aggregated methods combining several basic classifiers should be used. This paper addresses the issue of selection of the best aggregated classifier. The effectiveness of such approach has been confirmed by numerous tests of real-world objects.
It is possible to forecast the state of the technical object using various methods. The realistic
simulation using the time-series system is the most commonly used approach [
For example, the engine performance is diagnosed by reference to the fuel consumption rate, the
gas temperature, the noise and vibration level, the exhaust gas composition, the clearance between the
cylinder and the piston, the clearance between crankshaft necks and bearings and some other
indicators [
Basic data are a priori information about the state of the object according to the results of the
previous exploitation: upon the given values of controlled indicators the technical system is operating
one or the faulty one. It is assumed that there is some unknown dependence between indicators of the
object functioning and its states. Based on basic data it is necessary to restore this dependence, i.e. to
plot an algorithm, capable of providing a fairly valid answer about the state of the object for the given
set of indicators of its functioning. It is a task for the computer-aided learning or the learning from
examples (with a tutor). Binary classification, i.e. the object state division in two categories [
To assess the quality of the plotted algorithm in the context of the opportunity to forecast, the original sample is divided in two disjoint subsets. The first subset is the learning sample itself for handling the task of learning (which, usually, comes to assessing parameters of the model of the appropriate algorithm). The second subset is the control (or test) sample which is not used for learning. This part of the sample is used estimate the forecast error which characterizes the quality of learning. When using the cross-validation, the sample is divided in N parts (in practice, usually, N = 5 or N = 10). In this case, N – 1 parts are used for learning and the rest for control. All possible options are sorted out successively.
Methods of the computer-aided learning are actively used in all kinds of activities. Many different
approaches to the classification are used. For example, classical statistical methods (Bayesian
classifiers, the discriminative analysis, the logistic regression), methods specially focused on the
computer-aided learning (the support vector machine, neural networks), compositional methods
(bagging, boosting) and etc. The question at issue that it is impossible to determine which method
from selected ones will provide the best solution of the task. That is why many different methods or
their combinations are usually used. Decision to apply is made based on findings of the research of the
quality functional for the control sample. In works [
The purpose of this study is to plot selection algorithms of the best aggregated classifier.
The most widely known indicator that can be used for the quality assessment of the binary classification is the proportion of correct answers in the control sample, Q N where Q is the number of correctly classified objects from the control sample and N is the overall control sample size. The opposite characteristic which is the proportion (or the percentage) of errors in the control sample is used more often.
Sometimes the error dispersion (the mean square deviations of the operating state true probability in the r-test P(Yr ) from its forecasted value Pˆ ( X r ) ) is used to assess the quality of the classification: σ 2 = 1 ∑l (P(Yr ) − Pˆ ( X r ))2.
N r =1
If classes are unbalanced (when there are much more operating states than faulty ones), the
proportion of errors cannot be used for the reliable quality assessment of the classification [
, , , P = R = tp + fp
tp tp + fn where tp is the number of properly classified operating states, fp is the number of misclassified operating states, fn is the number of misclassified faulty states. Based on these two indicators the uniform criterion can be formed.
F = 2PR
P + R – it is called the harmonic average of the accuracy and the completeness (F-measure): the closer is the value of F to one, the higher is the quality of the classification.
Area under the ROC-curve (receiver operating characteristics): AUC (area under the curve) can
also be selected as the quality functional [
As an illustration of the numerical study we considered the water treatment system. We had the results of 348 tests upon eight quality indicators of the drinking water treatment. The system was faulty in 47 cases (when even one water quality indicator was beyond the limits). Whereas the division of basic data in the learning sample and the control sample was carried out randomly, we repeated tests 50 times.
We used the Matlab-package for tests. In Table 1 there are averaged values of the F-criterion and the area under the ROC-curve AUC for those five methods of the computer-aided learning where these values were maximum. Estimates suggest that the correlation between these two indicators was nonsignificant at the significance level 0.05. If the F-criterion is the same for selected classifiers, AUC values can be used for selection of the best classification method.
It is apparent that the decision tree bagging showed the best results in the considered example. Fcriterion discrepancy between the best and the worst (0.801 for the RUSBoost method) results was 8.7%, AUC – 21.5%.
The aggregated approach was suggested for handling tasks of the credit scoring [
We will use the exhaustive enumeration of sets from H base methods. Then, for example, if H = 2, we will get three sets: two basic ones and one aggregated; if Н = 3, there will be 7 sets: three basic ones, three aggregated ones, by two basic ones and one aggregated of all three basic methods. It is not too difficult to see that in the general case the number of sets is equal to 2Н – 1. To make the consistent decision about the operating state of the object based on solutions from certain classification methods, we will consider the aggregation of results on the average value, on the median line and using the voting procedure.
Suppose PˆK ( X r ) is the probability that r-object is the operating one determined with the aid of the К-basic method, K = 1,..., H . In mean that when aggregating on the average value: H ∑ PˆК ( X r )
H where PˆАК ср ( X r ) is the probability that r-object is the operating one.
ˆ
PАК ср ( X r ) = K =1
When aggregating on the median line, firstly, it is necessary to range the line with the results of basic methods in the set. If the number of basic methods is odd, the probability that r-object is the operating one will be:
PАК мед ( X r ) = PˆH +1 ( X r ). ˆ 2
If the number of basic methods is even, the relevant probability will be calculated as the half-sum of the median value results.
The result of the aggregated classification method on the voting procedure is the average value of of basic methods results determining the operating state of the object with the probability, for example, not lower than 0.1 ( PˆК ( Х r ) ≥ 0,1 ). Otherwise the probability that r-object is the operating one is considered as zero. PˆК is the probability that r-object is the operating one at base values of the object Xr functioning. As can be seen from the above, values of classification probabilities lower than 0.1 are treated as 0 and the rest are treated as 1. Aggregated classification models are plotted using these very values.
In this case, as mentioned above, the division of basic data in the learning sample and the control sample is carried out randomly. That’s why structures of aggregated classifiers turn out to be different. The question that has to be answered is what structure to select for making the final decision about the operating state of the object.
As before, we repeated tests 50 times. The sample volume was one and the same (25%) using all eight functioning indicators. Corresponding results of the F-criterion for five options of every aggregation type are shown in Table 2.
For example, the entry in the first line GrB+DTB+AB means that the aggregate of three basic classifiers including the gradient boosting (GrB), the decision tree bagging (DTB) and the AdaBoost boosting method was the best aggregation option on the average value when using the F-criterion in this experiment. The number of classifiers included in the aggregate (Table 2) fluctuates from two to six. In the general case it can include all basic classifiers.
Firstly, let us remark that any aggregated method on the F-criterion turned out to be better than any basic one. Secondly, values of the F- criterion for aggregated methods are not widely diverging. And finally, it is worth paying attention to the fact that the best of basic methods (the decision tree bagging) is included into the structure of all aggregated classifiers.
It is of interest to study the distribution pattern of the F-criterion values. As far as the aggregation using the voting procedure is concerned, we applied the following sequence of steps. We used the Statistica system to plot the normal probability curve. Then we transferred this curve to the value distribution histogram (Fig. 2) of this criterion. To check the normality, we used the Shapiro-Wilk criterion recommended for small samples (up to 50 tests). It is apparent that the distribution can be considered as the normal one when the significance level is 0.05. Similar results were obtained for other classifiers (both basic and aggregated) as well.
The distribution normality gives an opportunity to use the standard approach for checking the hypothesis that in the given example the aggregation does lead to the improvement of the diagnostics quality.
We checked the null hypothesis for the equality of F-criterion average values when aggregating and when using basic classification methods (in comparison with data from the decision tree bagging being taken into consideration as the best basic method). As an alternative, we considered the hypothesis for excessing of the average value when aggregating. Firstly, we compared dispersions of two samples upon the Fisher criterion (the difference turned out to be statistically non-significant). Then we tried the Student criterion with similar dispersions. It was concluded that the null hypothesis should be rejected: the average value of the F-criterion when aggregating is higher than when using basic classifiers.
As has already been noted, values of the F-criterion in Table 2 do not much differ. We checked the hypothesis that the increase in the number of basic classifiers (more than two) in the aggregate structure will non-significantly influence the value of the F-criterion. We divided the whole sample in two subsets. Data on aggregates consisting of only two components will be included in the first subset. All rest values will be included in the second subset.
Checking of the hypothesis for the equality of average values in these subsets shows its validity: the average value of the F-criterion does not change when expanding the number of basic classifiers in the aggregate structure.
A consequence of the above result is the fact that it is possible to reduce dramatically the time required for the calculation. Instead of enumerating all aggregation options when searching for the maximum value of the F-criterion (three aggregation methods and 11 basic classification methods used in the Matlab package, 3*(211 - 1) = 6141 options); it will be enough to enumerate only options including two basic methods (3*11!/2!9! = 165).
It is necessary to take into consideration one more circumstance. During all tests the aggregate included the best basic method (the decision tree bagging). Taking this fact into consideration gives an opportunity to scale back the number of options being enumerated by 30.
However, it is necessary to bear in mind that the given results are obtained in tests of only one technical object. Nevertheless, this experiment shows that the suggested approach shall be approbated for the diagnostics of any other system being studied.
To assess the operating state of the object it is recommended to select the simplest aggregated classifier with the sufficiently great value of the F-criterion. In the given example this classifier is the aggregation on the average value for the decision tree bagging and AdaBoost, or the decision tree bagging together with the gradient boosting (except for sufficiently great values of the F-criterion, these combinations can be more often found in Table 2).
The considered approach was also used (except for the water treatment system) when assessing the faulty state of the hydroelectric installation on the vibration level and the technological process of the mechanical processing when it showed similar results.
This test was carried out with the financial support from the Russian Foundation for Basic Research (RFBR) and the Government of Ulyanovsk region, the project 18-48-730001.