- Diversity is the basis of approaches using multiple classification systems. Solution sets are formed in ensembles of models. Many models give poorly distinguishable results. The use of strongly correlated model results in ensembles significantly reduces their effectiveness. Therefore, here the main influence is exerted by the variety of decisions made by the models. The value of each model decreases with an increase in the group of models. Accordingly, it is necessary to reduce the contribution of an individual model to the solution of the model. An approach based on clustering is proposed, according to which the influence of an individual model is inversely proportional to the volumes of aggregated groups. With this approach, the influence of an individual solution of the model, which differs from others, is significantly increased. Aggregation of groups is made in direct proportion to the correlation of decisions. Moreover, the aggregation of groups of models is performed according to the hierarchical structure of the ensemble. The solutions of strongly correlated groups of models are replaced by a single cluster solution. This solution at the next level can be grouped with other closest groups of models. Due to this architecture, the level of influence of a single solution of the model is increased. The main advantage of the proposed approach is the determination of the structure of the ensemble depending on the correlation of model decisions. Clusterization of decisions for features of similarity enhances the role of diversity and allows leveling out the error of an individual decision at a local level and to provide acceptable global indicators of cluster efficiency. Advantage of the proposed approach is the possibility of building an ensemble based on the properties of the correlation parameters of the models.
The basis for the use of group classification methods such as ensembles is the diversity of solutions of the models. For successful use, models must provide diverse and at the same time accurate solutions. Each model complements its solution with other models. This underlies the application of the ensemble. However, obtaining a variety of solutions is difficult, since the models are often trained on the same data, and they are based on similar mathematical approaches. The consequence of this is similar results, which have a strong correlation. Variety is also necessary for using the accuracy of solutions since combining less accurate models often gives better results. Supplementing informativeness with models is effective with a low correlation of decisions. Correlation is one of the most important indicators of the need to use the model in determining group decisions. And it can also serve as a criterion for determining the need to use a model in an ensemble. Therefore, the influence of each model should be determined depending on the correlation of decisions on the overall result of the ensemble. Consideration of the peculiarities of the model should be displayed in the architecture of the ensemble. At the same time, the model should improve the outcome of the overall solution. 2
Mutual addition of a solution in order to obtain an objective assessment in the form of a general solution is a fundamental principle on which the methods of group determination of solutions are based, as an example of ensembles of models [1]. A variety of methods are used to combine group models with data manipulation, with enhanced features. There are various modifications of bugging [2], boosting [3], stacking [4] as the area of the most well-known ensembles.
Despite the fact that the concept of diversity is intuitive, attempts to develop a system for measuring diversity are quite extensive. An example is [5, 6, 7]. The definition of a measure of diversity would allow the development of an approach based on that measure. However, the variety of factors affecting the effectiveness of the application of known solutions is very wide, requires extensive experience in use and does not always give the necessary result.
Another approach is to reduce the number of models that make up an ensemble. Pruning can reduce the redundancy of models in a group since models can also worse the outcome of ensemble predictions. At the same time, the computational complexity of the combination of models still remains significantly at the modern level of technology. Known a pruning method based on forward selection [8], ensemble pruning based on objection maximization [9] and others. This allows optimizing the calculations to achieve the expected result.
The most interesting in our opinion are the clustering-based [10, 11] methods. The rationale for this is the fact that the correlation of solutions contributes to the grouping of objects into certain agglomerations. Moreover, the difference in solutions within the agglomeration is relatively small. This may indicate that the correlation between the solutions of these models is high. And accordingly, they can be reduced by the required size. The correlation of models is an important factor, as it can be assumed that correlation is the inverse of diversity. By decreasing correlation, it is possible to increase the variety of solutions of ensemble models.
The selection of ensemble models is also important. The difference in models can often be determined based on the results of their predictions. The informative features of the data on which the model is based also play an important role. To expand the diversity of models, approaches using human intellectual abilities can be used. This allows building machine models based on the mental models of humans [12, 13, 14].
Methods of increasing the diversity of models and their effective use in ensembles is an important area of research. In this paper, we consider the basics of building ensembles based on the correlation of models and determine the effectiveness of the result based on the estimation of the prediction error. 3
The solution of the ensemble is a set of individual predictions of classifiers. In accordance with this, it is necessary to determine how the prediction of an individual classifier can affect the general decision. 1. The classifier may have the wrong solution. However, with the vast majority of correct decisions, the effect of an incorrect decision may generally be of little importance. 2. An incorrect solution to each of the classifiers introduces noise into the overall solution, thereby adversely affecting the quality of general solutions as a noise source. A noise source can be understood as errors of one of the classifiers in a significant percentage bias. Also, the noise source can be a set of errors of all classifiers that appear relativity to each prediction element. 3. The correct decision of an individual classifier can be refuted by a combination of incorrect decisions of a group of classifiers. And the presence in many solutions of the right one will not affect the general wrong decision. 4. A strong correlation of incorrect decisions of classifiers may have a dominant effect on the result of the work of the ensemble, despite the presence of correct predictions of other classifiers. 5. In the general case, a strong correlation of solutions for all data elements leads to worse results, since there is a greater probability that classifiers receive less information from the data and have less overall value within the training sample in comparison with the total data set. For arithmetic means, this is based on the dispersion formula for the sums of correlated random variables. In this case, the question arises of the influence of individual correct predictions of classifiers on the general incorrect decision of the ensemble.
Of primary interest is the definition of the influence of an individual classifier solution on the overall incorrect decision. The obvious influence of the majority on the decisions of the group, in this case, is erroneous. The interconnection of decisions indicates the incorrectness of the general approach for making decisions regarding a particular data element. This means that this element has a negligible content of general informativeness, on which the decisions of the majority were based. At the same time, if another decision was made on it by any classifier, there is another informativeness that was not determined by the majority. This is significant in the event of a majority group error.
The positive correlation of group decisions has an undesirable effect. The presence of a negative correlation may indicate the presence of an alternative opinion. This aspect indicates the variability of predictions. A prerequisite is the presence of a positive relationship between the prediction of the ensemble and the expected result. 4
The presence of a strong positive correlation between the individual decisions of the models within the ensemble facilitates their aggregation into groups. The decision of each group model is strongly correlated with the decisions of other members of this group. With a large measure of generalization, it is possible to formulate a general decision of the group, which to one degree or another will represent the solution of each model. Since differences in model decisions are insignificant within the group, the generalizing ability of aggregation will be significant within the ensemble. This allowing to divide the aggregate of models in the ensemble into aggregations on the basis of strongly positively correlated groups and representing clusters. An individual decision of a model within a cluster is of insignificant value, and it can be replaced by a generalized solution - a cluster decision. A cluster decision represents a solution to the models that form its solutions, and each individual model delegates its opinion to the cluster. Further subsequent aggregation of cluster decisions forms the ensemble solution. Such a process of delegating a decision to a higher-order level allows creating a hierarchical structure for the formation of the ensemble decision. Using this approach, the set of highly correlated solutions is replaced by a single cluster solution. The influence of an individual element on the ensemble solution decreases under conditions of strong correlation with other elements, and as a result of this, is determined by its location in any cluster. The larger the group size, the less influence the model has on the ensemble decision.
The formation of groups allowing to gradually reduce the variance in the ensemble. The variance of the group is replaced by a bias of a higher hierarchical order. A hierarchical structure of the ensemble forms the conditions for the separation of decisions by levels of locality, and the delegation process gradually globalizes local solutions. An important consequence of this process is that global solutions may be wrong at the local level, and local solutions may differ from the global one. The globalization of local solutions through delegation through the hierarchical structure of the ensemble improves the bias-variance tradeoff. The cluster decision is formed on the basis of an unbiased estimate. In this case, the dispersion of the cluster at the highest level of the hierarchy is not taken into account, and in fact, the distribution within the group is converted into a solution of the cluster, which has a certain bias at the next level. This allowed creating conditions for heterogeneous accounting for model predictions in the ensemble solution. The participation of the model in the global solution is made dependent on the correlation strength of the solution with relation to other models. This translates into a general rule for the structure of the hierarchy: the more general information contained in the model, the less its participation in global prediction. It is also, the correctness of a local solution may be weakly correlated with a global solution.
Consider a simple example in a one-dimensional space, which is shown in Figure 1. This example demonstrates a general unbiased hierarchical estimate. Group I consists of six strongly positively correlated models. Group II consists of one model. However, the participation of this one model is high in relation to the general decision of the groups. The figure shows the hierarchy of the second level and the unbiased estimation of the highest level. Aggregation of model predictions and location relative to the expected target In the general case, group II can increase the bias and therefore worsen the result. As an example, an asymmetric arrangement of groups I and II relative to the expected result (target). This is a consequence of increasing the value of a weakly correlated model. An important condition for this approach is the presence of a hierarchy of a higher level. In our case, this is a level three hierarchy. At the third level of the hierarchy (and this is the level of the ensemble), the model should have acceptable indicators of unbiased estimation. Only in this case, an increase in local bias can indicate that the model takes into account some information content that other models could not determine. Moreover, information content is exclusively local in nature and weakly correlates with the general. Using models with a biased global estimate introduces uncertainty in which we cannot determine whether the prediction of the model at the local level is an outlier or the model was able to determine the hidden local information content of the data. Thus, under the condition of a global unbiased assessment of the model, the hierarchical structure of the ensemble makes it possible to strengthen the latent information content of the data. And in the general case, it tells us that it is necessary to use some form of cascading classification as a consequence of the appearance of uncertainty and the process of strengthening local information content in relation to the global one. 5
Let us examine how the presence of model correlation affects the error of ensemble decisions. The ensemble makes decisions using the hierarchical structure of delegation. The model is accepted into the ensemble if it is overall unbiased. We will use the normal distribution of model predictions. For the simplest hierarchy structure with symmetrical distribution, the number of groups is two. This is also consistent with a minimum number of alternative solutions.
We take the relation between the predictions of two cluster groups CL1 = C1, C2 ,..., Cn1 and CL2 = C1, C2 ,..., Cn2 , n1, n2 N . Let the prediction Err (C L1 ) and Err (C L2 ) errors of two clusters (groups) of models. We accept the presence of cluster decision errors in general. Within each cluster, both variability of predictions and the presence of a strong positive correlation are possible. The studied variables Err (C L1 ) and Err (C L2 ) are both distributed normally and have large unbiased dispersion CL of cluster predictions. Group predictions are opposite signP(CLi (x)) signP(CLk (x)) , x X , i k 1,2 Groups are correlated with a certain Pearson coefficient CL . The combination of the two groups should lead to the expected result with a minimum ensemble prediction error min Err (E ) . Based on the opposite of the predictions of the groups of models, the ensemble prediction error
Err (E ) =Err (CLi ) + (1 − )Err (CLk ) , i k 1,2, 0,1 Accordingly, the variance of the error
Var(Err(E)) = 2 C2Li + (1 − )2 C2Lk + 2 (1 − )Cov(Err(CLi ), Err(CLk ))
, i k 1,2
Cov(Err (CL )) = CL CLi CLk , i k 1,2 Dispersion of ensemble error
Var(Err(E )) = 2 C2Li + (1 − )2 C2Lk + 2 (1 − ) CL CLi CLk , i k 1,2
The parameter 0,1 is a nondeterministic quantity and allows one to study the
variance of the ensemble error at the extremum
Var(Err(E ))
= 0
Solution (
(2 − 1) CLi CLk
The minimum coefficient value CL , based on equation (
C2Lk
C2Li + C2Lk
, i k 1,2
Provided C2Li = C2Lk , i k 1,2 we get the parameter value
We solve (
CLk CL CLi −
CLk C2Li + C2Lk − 2 CL CLi CLk
, i k 1,2
= −
2 ( CL −1)
2 2 (1− CL )
, i k 1,2
(
System (
We use equation (
lim − i →0
CLk = 1 , i k 1,2 CLi − CLk Provided CLi → 0 value → 1 .
Based on (16), bias from the median value of the parameter is accompanied by an increase in intergroup correlation. In the general case, the bias of the dispersed estimate is manifested in the correlation dependence. Thus, it can be assumed that the presence of a correlation between the clusters is the cause of the bias.
The dispersion of solutions of a cluster (group) has a dependence on the internal dispersion of solutions of the models forming this cluster = f ( ). Solutions within the cluster are also correlated.
Consider a situation in which i → 0 value → 1 . The general correlation of the ensemble is dependent on the internal correlation of the groups. If the correlation of one of the clusters tends to zero, then it can be assumed that the intragroup correlation value of the other group is largely represented in the general correlation at → 1 . 6
To verify the effectiveness of using the hierarchical structure, it is necessary to conduct experimental studies. It is necessary to form a data set for classification, and sets of correlated predictions of solutions. The most optimal from the point of view of understanding the effectiveness will be synthetic tests that provide the necessary parameters for research management.
Based on the generated normal distribution data using a linear relationship, variations of the model predictions are generated. This set of solutions is used to obtain correlated solutions.
To generate a correlation of random predictions, we use the Cholesky decomposition. If Corr is a correlation matrix, the Cholesky expansion has the form
Here X is the base distribution, Y is the correlated distribution.
When generating a correlation of two random variables
LLT = Corr
LX = Y 1 L =
0 1 − 2 Here is the Pearson correlation coefficient. (16) (17) (18) (19) Thus, using the set of Pearson correlation coefficient 1, 2 ,..., n , i = 1,..., n , sets of correlated model predictions S are formed. Clustering is a big area of data analysis methods that are used to obtain information about the data structure. This allows determining the aggregation of data and the identification of these groups. The data in each cluster are most similar in terms of the measure of similarity of the correlation distance, as one of the criteria. Cluster analysis will be made based on the decisions made by the models regarding the data element. In our case, clustering is one of the stages of classifications. Of the various clustering methods, we will use the simplest one in order to simplify the understanding of the proposed approach. The method should use the definitions of grouping data into a predetermined number of clusters. In experimental studies, we will use two clusters. For this purpose, the k-means algorithm is chosen. This is an iterative algorithm that splits data into disjoint groups. Predictions P = p1, p2 ,..., pn , pi R d , i = 1,..., n , within a group are most similar to the breakdown criteria, and clusters k N are most distant from one another. In terms of usability, the K-means approach defines the centers i , i = 1,..., k of cluster sets Si , i = 1,..., k , k S = P , Si S j = , i j, i, j k . Here t - iteration index.
Algoritm: K-means set i , i = 1,..., k ; t 1 while
i 1, k : it i(t−1) p P : p S it if p − it 2
2 p − j(t−1) ,j 1, k i : it = pSit t++ p
Sit K-means strives to create clusters in which the fewer the variations, the more uniform the data points in one cluster.
The cluster center is a common solution of the formed group. The dispersion of prediction of a group of models is leveled and embodied in a point solution with a possible bias at the next hierarchy level. 6.2
Experiment Results As the base we use a linear model, presented in the form y = xT b + (20) Here b are the model parameters, is the random error of the model. Alternative model predictions will be obtained by generating correlated sets with relation to the distribution of the basic random error.
Set 1 Set 2
Set 4 Fig. 2. Distributions of predictions based on a linear model with error modeling and a hierarchical cluster structure of an ensemble of models on given sets of correlations Using a visual representation of the averaged cluster solutions, one can observe a local amplification of weakly correlated model predictions. Moreover, the general statistical estimates of the distribution of ensemble predictions remain in acceptable values. Compared to the random error of the linear model, the hierarchical structure of the ensemble gives a significant advantage from the point of view of classification.
We determine the effect of the size of the set of models in the ensemble on the change in the results of the ensemble. To do this, we fix the distribution of the generated data based on the seed parameter, and change the set of the correlation parameter.
Examples of sets of randomly generated normal distribution data visually demonstrate clustering using. Cluster decisions are generalized by the centers of these clusters, the values of which are determined using the K-means algorithm.
Cluster decisions are passed to the next level of the hierarchy. These experimental studies applied two levels of ensemble hierarchy. When making an ensemble decision, cluster predictions are averaged.
Set 1
Set 3 Fig. 3. Change of the ensemble predictions based on the structure of the set of correlation parameter on a fixed data distribution Variation of the predictions of ensemble models makes it possible to reduce the influence of a random prediction error on the ensemble result. This confirms the effectiveness of the use of group decision-making methods. At the same time, a prediction is divided into localization levels depending on the architecture of the ensemble construction. This indicates the fact that local model errors practically do not affect the global characteristics of the ensemble. The hierarchical structure allows enhancing the influence of the model with relative deviations from the agglomeration groups. Thus, the redundancy of the models is leveled by reducing their influence. 7
Building an ensemble on a hierarchy of clusters, based on the correlation of model decisions, has several advantages. ─ The hierarchy of clusters creates localization of predictions at the localization level with a constant global estimation. ─ The proposed ensemble structure allows managing decisions based on the level of decision. ─ The introduction of decision levels allows creating a control mechanism for the influence of local decisions on the global result. This allows to locally change decision parameters without affecting global prediction. ─ A controlled mechanism of globalization of local predictions is being created. ─ The influence of model predictions is ranked depending on the correlation properties of recognized information content. ─ Correlation of the solution is the main factor forming clusters. ─ The role of model predictions with a strong correlation of the solution decreases. ─ The influence of the identified low information content on the global ensemble solution is increasing.