The problem of quality model creation for models based on decision trees and forests is considered. The set of indicators characterizing properties of decision trees and forests is proposed. It allows to quantitatively evaluate such properties as diversity, equivalence, retraining, confidence in a decisionmaking, hierarchy, equifinality, generalization, nonlinearity, robustness, homogeneity, sensitivity to the input signals, plasticity, variability adaptability, symmetry, asymmetry, emergence (integrity), interpretability (logical transparency), learnability, and autonomy as for individually considered single tree model, as for an ensemble of tree models (forest).
Automation of decision-making in applied tasks, as a rule, requires the construction of a decision-making model. To solve the problem of decision-making model constructing on the precedents, a wide class of computational intelligence methods has been proposed, including neural networks [1-5], neuro-fuzzy networks [6-9], decision and regression trees [10-17], forests of decision trees [18-22] and etc.
Usually, the quality of such models is characterized by the error function [1, 2]. As a result model is selected from several alternative obtained models, which has the smallest error. Note that for each class of methods, even for the same given training sample of observations, it is possible to obtain a wide range of different models with acceptable accuracy. At the same time, achieving the maximum accuracy (the smallest error) does not guarantee a high level of customer properties of the model.
Earlier in [23-26] author has proposed a set of indicators, applicable to models based on neural and neuro-fuzzy networks. However, most of these indicators are not applicable to the models based on decision trees and forests due to their paradigm difference from network models paradigm. Therefore, it is necessary to develop a quality model for decision trees and forests providing comparability of its indicators with the quality indicators of models based on neural and neuro-fuzzy networks proposed earlier in [2325].
The properties of models can be affected not only by the structural parameters, but also by the properties of the training sample [27-31]. Therefore, it is necessary to take into account information about the properties of the sample when determining the quantitative indicators characterizing the properties of models.
The aim of this work was to create a quality model for models based on decision trees and forests as a set of quality indicators. 2
Let we have a training sample of observations in the form <x, y>, where x = {xs}, xs={xsj}, y={ys}, s = 1, 2, ..., S, j = 1, 2, ...., N, xs is a set of input values (descriptive) features of s-th sample instance, xsj is a value of j-th input feature of s-th sample instance, y is an output feature value for the s-th instance in a sample, S is a number of instances in the training sample, N is a number of input (descriptive) features characterizing the instances of training sample.
Then the problem of model building for the dependence y=f(w, x) on a sample of observations <x, y> based on the decision tree tree consists in identifying the model structure f and the values of its parameters w that provide an acceptable value of the given quality functional of the model F(x, y, f, w) [17].
The problem of model building based on a forest of decision trees, in turn, can be represented as a problem of obtaining a set of models forest={treet}, treet =ft(wt, x), that provide an acceptable value of a given quality functional F(x, y, {ft},{wt}), where t is a number of a tree in the forest, t = 1, 2, ..., T, T is a number of trees in the forest, f is a model structure of a t-th tree , wt is a set of model parameter values of a t-th tree [20].
It is obviously, that the problem of the quality functional creation of models based on decision trees and forests requires the determination of a set of indicators {Ii} that quantitatively describers the properties of the models. 3
Primary model characteristics Along with the sample parameters described above, we will use such notation for the characteristics of the samples: <xtest, ytest> is a test sample, Stest is a number of instances in the test sample; N, N' are, respectively, the number of signs in the original set and in the reduced set of features; fmax , fmin are, respectively, the maximum and minimum boundary values of a model output, y max , y min are, respectively, the maximum and minimum boundary values of output feature.
The basic properties characterizing the model structure are defined as: M is a number of levels in a tree, Nn is a number of nodes in the model, N is a number of nodes in the -th layer of a tree, N nmax is a maximum possible number of nodes in a model, fi is an i-th node function, min is a smallest possible change of the real number, taking into account the bit grid of the computer, o ( j) is a complexity of the j-th node, which can be defined similarly to [250] in units of elementary operations of addition and multiplication, ia*ut is a characteristic of autonomy of a formation of i-th element of a model structure ( ia*ut = 0, if the inclusion (or exclusion) of i-th node to the model determined only by the human; ia*ut = 1, if the inclusion (or exclusion) of i-th node to the model is automatically defined by the training method; ia*ut = 0.5, if the inclusion (or exclusion) of i-th node to the model can be determined by the human or training method), np (i) is a characteristics of plasticity of i-th node of the tree, which is equal to the number of possible states of a node i (for leaf nodes containing singleton (not containing functions) np (i) = 1, for the nodes with functions the np (i) should be taken equal to the number of different functions that may contained in the node, for the rest nodes of the tree we should take np (i) as a number of branches of the node), wij is a connection of i-th and j-th nodes ( wij = 0, if i-th and j-th nodes are not connected, and wij = 1, if i-th and j-th nodes are connected).
Let introduce the notation for the description of a model parameters: wmax , wmin are, accordingly, the maximum and minimum possible values of a model parameters, Nw is a number of adjustable parameters of model node, N max is a maximum possible w number of adjustable parameters of a model nodes, wim,jax , wim,jin are, respectively, the maximum and minimum possible values of j-th parameter in i-th node, w is a smallest possible change in weights taking into account the size of the computer bit grid, wj is a j-th model parameter, Nw(i) is a number of parameters of i-th model node, wi, j is a minimum possible change of j-th parameter of i-th node taking into account the bit grid size of a computer, sp(i) is a characteristics of a plasticity of parameters of i-th node ( sp(i) = 0 if the node has no adjustable parameters, otherwise
Nw (i) set: sp (i) round((wim,jax wim,jin ) / wi, j ) ),
j1
We define the notation for describing the functioning of a model as: Etr, Еtest are, respectively, model errors for the training and test samples, E(w) is a model error at a set of weights w.
The following notation will be used for the model training method parameters: N met is a the number of training method parameters, Nmauett is a number of parameters of a training method, which values of are determined automatically without a human intervention, aut (wi ) is a characteristic of autonomy of i-th node parameter values setting ( aut (wi ) = 0, if parameter values set only by a human; aut (wi ) = 1, if the values of the node parameters are determined automatically by the training method; aut (wi ) = 0.5, if the values of the node parameters can be determined by the human and method).
For tree models in a forest we denote: w f ,max , w f ,min , accordingly, maximum and minimum possible values of parameters of a forest model, Nwf is a number of parameters of a forest model without taking into account the number of parameters of its trees, T max is a maximum possible number of trees in the forest.
If necessary to distinguish the use of indicators for the decision tree we will use notations in the form of I t or I tree (here I is an indicator, t is a tree number, tree is a tree symbol), and for the forest we will use notation of the form of I forest (here I is an indicator, forest is a forest symbol). 4
Diversity is defined by a number of different states of a system. In accordance with the law of "Requisite variety" of W.R. Ashby, creating of a system able to decide a problem, which has certain known diversity (complexity), it is necessary to provide for a system an even greater diversity (knowledge of solving methods) than the diversity of the problem being addressed, or to ensure the ability of the system to create this diversity within itself (it would have methodology, could have developed a methodic or proposed new methods for solving the problem) [32].
The absolute indicator of the limiting diversity of the synthesized model based on
the decision tree tree, by analogy with [24], is defined as (
Idiv (tree) round wmax wmin Nw Nnmax
w i1 (np (i)).
Idiv (tree, x, y ) Idiv (tree) ;
– in relation to the population universe as (
The absolute indicator of the limiting diversity of the synthesized model based on
the forest of decision trees forest is defined as (
I div ( forest ) round w f ,max w f ,min Nwf T max
w t1 (I div (treet )).
The greater the value of limiting diversity, the wider the range of models we can obtained on its basis.
By analogy with [24], we define the diversity indicators of the tree model tree and
forest model forest:
– in relation to the training sample as (
I div ( X ,Y )
I div ( X ,Y )
(
The more the value of Idiv(tree,<x, y>) for a single tree, and the value of Idiv(forest,<x, y>) for the forest, the more will be the model potential for approximating the relationship represented by the sample. The smaller the value of the corresponding indicator for the model at an acceptable level of error E , the better the approximation of the sample is.
The more the value of Idiv(tree, X, Y) for a single tree, and the value of Idiv(forest, X, Y) for a forest, the more the model will be able to solve the given problem. However, if the relevant indicator is greater than one, or is near to one, then the model is too excessive for the problem solving.
The equivalence of models is determined as follows: two models are equivalent if they have the same sets of answers (they respond equally to the same input stimuli) [33].
The equivalence coefficient of trained models based on decision trees t1 and t2 for
the sample <x, y> we defined as (
1 S
Ieq (t1, t2 ) exp S s1 ft1 (xs ) ft2 (xs )2 .
The equivalence coefficient of trained forests models can be determined similarly to the above, replacing the calculated tree outputs with the corresponding forest outputs.
The values of the equivalence indicator will be in the range from zero to one: the more similar the responses of the models with the same input influences, the greater the value of the equivalence coefficient.
Retraining of a model for the training sample x relatively to the test sample
<xtest, ytest> <x, y> may be defined as (in the given form with substitutions) [24]:
– for classification problems as (
1 f (xs ) y s S x1 1 Stest
1 f (xtsest ) ytsest ;
Stest x1
– for the evaluation problems as (
1 f (xs ) ys 1 S S x1 1 Stest
1 f (xtsest) ytsest .
Stest x1 where is an error threshold.
Since the error threshold for an instance in practice cannot always be set, as well as
for greater universality and uniformity in solving various problems we define it as
(
exp S s1
f (xs ) ys max ( ys ) min ( ys ) s1,2,...,S s1,2,...,S 2 1 Stest
exp Stest s1
f (xtsest ) ytsest.
max ( ytsest ) s1,2,...,S
s . min ( ytest ) s1,2,...,S 2
These indicators can be used not only for a model based on a single tree, but also for a forest of decision trees, using as f the output value determined by the ensemble of forest trees.
The higher the value of the retraining indicator, the worse the approximating properties of the model for data that did not used in the training.
Confidence in a decision-making is a subjective assessment by the model of the made decision [34].
Regarding the value at the model output for the instance xs fed to its inputs, we
determine the confidence indicator of the model in the made decision (
It is possible to estimate the coordinates of the cluster centers for leaf nodes on the basis of a training sample:
– for the decision tree constructed on the basis of the training sample, determine the belonging of the training sample instances to leaf nodes;
– for every q-th leaf node uq form a cluster Cq {Cqj} of instances fallen into this node, q = 1, 2, ..., Q , where Q is a number of leaf nodes (clusters);
– for each j-th feature as the coordinate of the cluster center take the arithmetic
average of the coordinates of the instances of the corresponding cluster according to the
corresponding feature (
Indicators of subjective confidence of model based on a decision tree will take values in the range from zero to one: the higher the value, the closer the properties of a recognized instance to formed cluster center templates, and the more the model is confident in the made decision.
The confidence of the forest of decision trees for the instance xs is defined as (
Icfeorrtest (xs ) Icfeorrtest (xs , k ), k 1, 2, ..., K ,
k Icfeorrtest ( xs , k ) I t
cert | ft (xs ) k,t 1, 2, ...,T , t ft is an estimated value of the model output of t th tree, I forest is an indicator of cert confidence of forest models, , are symbols of operators, defining the confik t dence of the forest in the decision for the k-th class and the t-th tree, respectively. As such operators it is possible to use the minimum, maximum, arithmetic mean of the set of arguments.
The indicator of averaged confidence of forest for sample x is defined as (
– determine the function u( xs ) that maps the recognized instance xs to the node number of the tree into which it fell.
The average confidence of the decision tree for a sample x is defined as (
Icert (xs ).
S s1 Icfeorrtest (x) 1 S
Icfeorrtest (xs ).
S s1
Indicators of subjective confidence of the forest will take values in the range from zero to one: the higher the indicator value, the more the trees ensemble confident in made decision.
The hierarchical organization of the structure, the integrity and crushability of elements allows to build models of complex objects from simpler ones; the work of the hierarchical structure requires that the information element in each hierarchical level behave as a whole, but when moving from level to level it must be fragmented, and when moving from the upper hierarchical level to the lower, this fragmentation corresponds to the allocation of its constituent elements, and when moving from the lower level to the top, it corresponds to the inclusion of a certain part of this element in a more complex object [33].
The hierarchy of the model based on the decision tree is defined by analogy with
[25] as (
Ih
M 2 N
1 M (M 1)Nn
, M 1, Nn 1.
The greater the Ih value , the greater the number of hierarchical levels in the model with respect to maximum possible number of levels for a given number of nodes Nn.
Estimate the maximum possible number of levels. Since the maximum number of
levels in the tree will be at the minimum number of outcomes from nodes, then at
each level there should be at least one node with two outcomes, and the rest of the
nodes should be leafy. Moreover, the greatest number of levels will be achieved for
the tree, where only one node at each level (except for the last) has two outcomes, and
the rest are leafy and contain only two nodes at each level. Thus, for the deepest tree,
the number of nodes of the highest level is 1 (root), for the lower level is 2 (leaves),
for the remaining layers is 2, i.e. 1+2(M–1) = Nn . From here we get: M =0.5(Nn–1)+1.
Therefore, for the decision tree we get (
Ih
M 2 N
1 0,25Nn3 Nn2 0,75Nn , M 1, Nn 1.
The hierarchy of the model based on the forest of decision trees is defined as (
t1,2,...,T
The elasticity for a function y(x) on the variable xj in [35] is defined as (
x j where y y(x j x j ) y(x j ) , xj > 0, y > 0.
y(x j )
The relative elasticity indicator on the variable xj of approximating function y=f(x)
realized by the model at output y trained on a training sample <x, y> is defined
similarly to [24] as (
El x j ( y )
1 S ~x js ~f( ~xjs x j ) ~f( ~xjs ) ,
2S s1 x j ~f( ~xjs ) 2
(
The larger the value of elasticity indicator, the more elastic is the model. This indicator applies both to a single tree model and for a forest model.
Equifinality is a regularity of functioning and development of the system, characterizing its ultimate capabilities [36].
Relative equifinality of a model based on a decision tree tree defined similarly to
[24] as (
Ieqf (tree, x, y ) NNmwax NNmnax exp 1 S ( f (w, xs ) y s )2 .
w n S s1
(
Relative equifinality will receive the largest value (top-limited by one) for those models which have reached the maximum possible size and the number of parameters during synthesis as well as the smallest error (bottom limited by zero) in the learning process.
For the forest of decision trees, we determine the relative equifinality indicator as
(
T T Ieqf ( forest , x, y ) T max Ieqf (treet , x, y ).
t1
Generalization is the model’s ability to integrate partial data to determine patterns and prolongate results, that is, after training based on the training set, to give answers for test sample instances similar to the training sample but not included in it [37, 38].
The generalization indicator of the decision tree for the training <x, y> and test
<xtest, ytest> samples is determined by analogy with [37, 38] as (
( f (x p*) ytpest)2 N(xsj xjptest)2 IG 1 exp 1 Stest j1 Stest p1 N(ys ytpest)2 (yis yiptest)2 0, s arg min N(xtj xjptest)2, j = 1, 2, ..., N, p = 1, 2, ..., Stest.
t1,2,...,S j1
(
In a similar way, the generalization indicator for the forest of decision trees will be determined.
Generalization indicator will take values in the range from zero to one, and will be the greater, the smaller the error of a model at instance recognition, and difference of recognized instance to nearest by features instance of a training sample is more.
The generalization indicator of the trained model is defined as (
If generalization indicator is significantly greater than one, then the model shows great ability to generalization, if the generalization indicator much smaller than one, then the model does not shows no generalizing properties.
Generalization indicator for a forest of decision trees is defined as (
NS N N w n
exp((Etr Etest )2 ).
NS I gfeonrest T N t N t
w n t 1
exp((Etr Etest )2 ).
The errors here are defined for the ensemble of trees.
Nonlinearity is a dependency that cannot be explained by a linear combination of variable inputs [39, 40].
The nonlinearity indicator for classification problems is defined similarly to [39] as
(
S1 f xp 1 xs f (xp ) 2 S S 0 S S .
S(S 1) s1 ps1 jN1 xsj xjp 2
The nonlinearity indicator for estimation problems is defined as (
j1 where Inl ( x, y ) is a nonlinearity indicator of the sample, determined according to [41, 42].
~ If the indicator I nl is equal to one, then we can conclude that the model corre~ sponds to the sample in complexity. If the indicator I nl is less than one, then the smaller its value, the greater the effect of retraining will be, and it would show possi~ ble redundancy of a model. If the indicator I nl value exceeds one, then the model is not sufficient for good approximation (require additional training or change the model structure).
Robustness is a model property to reliably solve a problem when receiving incomplete and / or damaged data. In addition, the results must be consistent, even if some part of the model is damaged [43, 44].
The robustness of the model on the basis of a decision tree in relation to the input
signals is defined as (
I Rxb
min 1, 2 j1m,2i, n..., N max {x sj } s1,2,...,S min {x sj } s1,2,...,S min {x sj } , s1,2,...,S 2
min 2 1 S f xs* xbs*xbs ,b j,b1,2,...,N yis S s1 xsj*xsj xj
x j , x j
min s 1,2,..., S { x sj } max {x sj } x s 1,2,..., S
min s 1,2,..., S
{x sj } ,..., max {x sj } max {x sj } s 1,2,..., S x s 1,2,..., S
min s 1,2,..., S {x sj } , x 0,1 is a constant that regulates the accuracy of determining a robustness indicator on model inputs.
The indicator is normalized smallest change in the input signal, resulting in a significant increase of model error.
The robustness of a model based on a decision tree with respect to weights
(parameters) is defined as (
The indicator is a normalized least change in weight values, leading to significantly increase in model error.
The integral robustness indicator for a decision tree can be defined as (
I Rb I RbxI Rwb.
The indicator IRb will take values in the range [0, 1]. The closer its value to zero, the lower the robustness of the model, the more sensitive the model to a change in input signals or parameter values. The closer the value of the indicator IRb to one, the greater the robustness of the model, the less sensitive the model to changes in input signals or parameter values.
In this way, robustness indicators for the forest of decision trees can be determined.
The homogeneity of the elements lies in the fact that models are built from many simple unified standard elements that perform elementary actions and are interconnected by various connections [45].
The homogeneity of the functions of the nodes of a decision tree is defined by
analogy with [25] as (
I hn
N n N n 2 {1 | fi f j } i 1 j i 1
N n ( N n 1)
.
T Ihtn Nnt ( Nnt 1) Ihfnorest t 1
T Nnt (Nnt 1) t 1 .
(
The homogeneity indicator will vary from zero to one: the more its value, the more uniform the corresponding elements of the model.
The homogeneity of the node functions of the of the forest trees is defined as (
The averaged normalized indicator of the sensitivity of the output of the decision
tree to a change in the input signal is defined as (
1 S N I tol SN y max y min s1 j1 max 1 , 2 ,
The value of the sensitivity indicator will be in the range [0, 1]. The higher the value of the sensitivity indicator, the stronger the model reacts to changes in the input signal, the greater are its categorization capabilities. However, too high sensitivity may indicate a weak model resistance to noise and interference in the input signal.
The average indicator of the sensitivity of the forest to changes in the input signal is defined as (36):
Itofolrest 1 T
Itol (treet ).
T t 1
Plasticity determines the complexity of the model’s behavior, which is considered as a result of the interaction of many elements, each of which limits the action of others and is limited by others on the way to the formation of global observable behavior [49, 50]. As an analogue of neural plasticity, where neuron nodes are considered as plastic elements for neural network models, with respect to decision trees, we will consider the plasticity of nodes. As an analogue of synaptic plasticity (modification of the strength of the synaptic connection between nodes, implemented by the scales in neural network models) as applied to the decision tree, we will consider the plasticity of tunable parameters of tree nodes.
The relative indicator of plasticity of the nodes of the model by analogy with [25] is defined as (37):
Nn np (i) Inp i1
Nnmaxnmpax .
The indicator Inp will take values in the range from zero to one: the greater its value, the higher the level of plasticity of the model nodes.
The relative plasticity indicator of the adjustable model parameters, by analogy with [25], is defined as (38): (36) (37)
The coefficient Isp will take values in the range from zero to one: the bigger its value, the higher the level of plasticity of the model parameters.
The relative indicator of plasticity of a model is defined as (39) [25]:
The relative indicator of plasticity will take values in the range from zero to one: the greater its value, the higher the level of plasticity of the model and, therefore, it has better adaptive abilities.
For the forest of decision trees, we define the plasticity indicators as (40)–(42): Isp
Nn sp (i) i1 Nn2round wmax wmin
w .
Variability is an ability to obtain several different models for approximating dependencies from the same data sample using the same method [51-53]. As applied to decision trees, the variability of models is determined by the choice of a feature for the root node and the order in which features are added for checks at other nodes, the method of determining threshold values in nodes, etc.
The absolute indicator of the variability of the model we define similarly to [25] as (43):
I pl Inp Isp .
Infporest I forest sp I pfolrest 1 T
I ntp , T t 1 1 T
Istp , T t 1 1 T
I tpl .
T t 1
M N Iv v (n(, i))v (w(, i)),
1 i1 where v (n(, i)) is a variability of the verification of the i-th node of -th layer of the model: v (n(, i)) = 1, if a non-random feature hit into the node; v (n(, i)) =N, if a feature for checking in the node selected as random from all original feature set v (n(, i)) = N* , if a feature for checking in the node is randomly selected from the set of N* not yet considered features (44): (38) (39) (40) (41) (42) (43) where v (w(, i)) is a variability of determining the values of the parameters of i-th node of -th model layer: v (w(, i)) is equal to the number of parameters in the node that can be configured non-deterministically, if all parameters in the node depend on previous nodes, then v (w(, i)) = 1.
The more the Iv value, the more different models can be obtained based on the corresponding paradigm.
The absolute indicator of the variability of the forest of decision trees is defined as (45):
Ivforest T Ivt .
t1
Noise resistance is the property of the model to provide the correct response to an input signal containing noise [54].
The indicator of the resistance of the trained model to additive noise in the input signal at the j-th input is defined similarly to [24] as (46):
exp Itol j 1 S 2 s1 1 2 , 1 ( f ( x s ) f (( j ) x s ))2 , 2 ( f ( x sj ) f (( j ) x s ))2 , ( j ) xgs xgs max x p
p1,2,...,S g xgs , g j, g 1,2,..., N ; ( j ) xgs xgs max x p
p1,2,...,S g xgs , g j, g 1,2,..., N ; min x p , g j, p1,2,...,S g min x p , g j, p1,2,...,S g
sp1ms,21,i.,n....,.S,S;xsj x jp y s y p min .
j1,2,...,N pm1,2a,.x..,Sx jp pm1,2i,n...,Sx jp where is a given noise level, 0 < < 1. In order to automate the process of setting it is proposed to use such formula (47):
The greater the value of the indicator of model resistance to noise in the input signal on j-th input, the less important this input to the decision making. (45) (46) (47)
This indicator is applicable both to a model based on a single decision tree and to a forest-based model.
Adaptability is a property of structures to dynamically and independently change their behavior in response to an input stimulus [55]. In relation to a model based on decision trees, adaptability is determined, first of all, by plasticity, which determines the resources for adaptation: the greater the plasticity, the more adaptive the properties of the model.
Plasticity is a necessary but insufficient prerequisite for adaptability. Along with plasticity, the adaptive properties of the model are influenced by the sensitivity of the model, which determines the strength of the reaction of the model to the minimum change in the values of its parameters.
The adaptability indicator of a model is defined as (48):
I adapt I pl Itol . (48)
The larger the adaptability indicator value, the greater the possibility has models to adapt to a given task.
The adaptability indicator in this way can be determined for the forest of decision trees.
Symmetry reflects the proportionality in the arrangement of the parts of the whole in a space, the complete correspondence (in location, size) of one half of the whole to the other half [56]. In relation to decision trees, it is possible to determine the indicators of symmetry and asymmetry.
The indicator of symmetry of the structure of the decision tree is defined as (49): n Isym
2 Nn Nn
Nn2 Nn i1 ji11 fi f j .
The greater the Insym value, the greater the symmetry of the structure of the decision tree.
The asymmetry indicator of the model structure based on the decision tree as (50): I asym 1 I snym.
n The greater the value of Inasym the greater the asymmetry of the model structure. The symmetry indicator of the nodes of the decision tree is defined as (51): The higher the Iwsym value the more the symmetry of model connections.
The asymmetry indicator of model connections based on the decision tree is defined as (52):
w Isym 1 N n Nn N 2 1wij wji .
n i1 j1 Iasym 1 Iswym.
w (49) (50) (51) (52) The larger the Iwasym value the greater the asymmetry of the model connections.
The general indicator of the symmetry of a model based on a decision tree is defined as (53):
The higher the value of Isym the bigger the symmetry of the decision tree .
The general indicator of the asymmetry of a model based on a decision tree is defined as:
I sym I snym I swym.
Iasym 1 Isym.
The larger the Iasym the greater the asymmetry of the model.
For the forest of decision trees, the symmetry and asymmetry indicators can be determined as the average values of the corresponding indicators of the individual trees included in the forest.
Emergence (integrity) is a regularity that manifests itself in the system in the appearance of new properties in it, which are absent in its elements. The integrity property is associated with the purpose for which the system is created. Let Co is an intrinsic complexity, which is the total complexity (content) of system elements without interconnecting them (in the case of pragmatic information, the total complexity of the elements that affect the achievement of the goal), Cv is a mutual complexity characterizing the degree of interconnection of elements in the system (i.e. the complexity of its schema or structure). The degree of system integrity in accordance with [24, 57, 58] is defined as (55):
The emergence of a model based on a decision tree is defined similarly to [24] as (56):
I Сv / Co .
Nn Nn v (i, j) I i1 j1
.
Nn o ( j) j1 (53) (54) (55) (56) (57) The larger the I value, the more holistic is the model.
For a forest-based model, emergence is defined as (57):
T 2 T I t t 1 .
Interpretability (logical transparency) is a model property to be understandable for human perception and analysis [59, 60]. Obviously, a model is more interpretable if it is hierarchical, and the average number of node connections does not exceed 5-7 (this number is caused by the peculiarities of the human psyche). Since each node in the decision trees has only one input, it is necessary to consider mainly the number of outcomes from the node.
Heuristically we define interpretability through hierarchy and the number of nodes [25] as (58):
Ih Iinterp. Nn Nn {wij | j i} i1 j 1 .
The level of model interpretability increases with increasing of Iinterp. value. For the forest of decision trees, we define the interpretability as (59):
Iinfoterreps.t 1 max {I ht} t 1,2,...,T 2Nn2 . where L is the Lipschitz constant for the training sample [62] as (61): L(tree) is a Lipschitz constant (complexity) of the model [63, 64], which, as applied to the binary decision tree, is estimated as (62):
Learnability is the property of a model to improve its work (to learn or adapt), using examples to turn it to solve a particular problem [61].
The decision tree model learning indicator similarly to [25] is defined as (60): (58) (59) (60) (61) (62) (63) Ilr
I pl L(tree)
NSL
, L
max s, p1,2,....,S; s p y s y p / xs x p ,
N ' L(tree) 10 K
N '/ K N ' N ' K .
0 2 T I tpl L(treet )
The greater the value of the learnability indicator, the model has bigger potential for solving the problem of approximating dependence y = f(x) given in tabular form.
For a forest of decision trees, we define the learnability indicator as (63): I aut 1 N met
N n ia*ut aut ( wi ) 1 N mauett i1
.
N n I afuotrest
min t 1, 2,..., T
{ I atut }.
Autonomy is an agent's ability to act without direct human intervention by control on its own actions and internal state. Autonomy also implies the possibility of learning based on experience [65].
Since the trained computational intelligence models, as a rule, in the process of their functioning in decision-making does not require human intervention, they are equally have a property of autonomous functioning. However, in the process of training the level of autonomy for different models and different training methods may vary considerably.
Therefore, we will consider further characteristics of model autonomy only in relation to the process of its learning. Since the ability to learn is determined by plasticity, the autonomy of learning (self-adaptivity) will be characterized by an indicator that depends on the plasticity characteristics of the model. On the other hand, the dependence of model learning from the human may be characterized by its influence (portion) on the formation of the structure and parameters of the model.
Combining these considerations, we obtain the indicator of autonomy of model training method (64): (64) (65) As Iaut increases, the level of model autonomy in the training process increases.
For a forest of decision trees, the indicator Iaut can be defined as a smallest of the indicators of forest trees (65): 5
Integral indicators of model quality Information quality criteria is a family of integral indicators, depending on the model error E , the training sample volume S and the number of adjusted model parameters Nw . They include Hannan-Quinn Criterion [66], Bayesian Information Criterion [67], Akaike's Information Criterion [68], Corrected AIC [69], and Unbiased AIC [69]. A number of criteria in addition to the error, the sample size and the number of adjustable parameters also take into account the maximum possible number of adjustable parameters N max . They include a Minimum Description Length [70] , Shortest Data w Description [71], Consistent AIC [69] and Mallow Criterion [72].
At model constructing and comparing it is usually assumed to be identical the sample size. Therefore, it is advisable to exclude the sample size from the comparison criteria. At the same time, various synthesized models may not use all of the features of the sample. Therefore the number of features used in the models N' should be seen as an important property of the models at their comparison.
On the basis of these considerations we can define integral information criterion of a model as (66):
The IIC criterion will take values in the range from zero to one. The less its value the worse the model, and the bigger its value the better the model. Here, for different models, the error values E and the maximum number of adjustable parameters Nwmax ,
Having resulted similar, we receive (69):
I ef ' 2 arcsec NN ' NNwmwax N nmax NS exp E , N n N w
N ' 1, N nmax 1, N nmax 1.
I ef ' 2 arcsec N 2 SN wmax N nmax
exp E , N ' N w 2 N n
N ' 1, N nmax 1, N nmax 1.
An alternative generalized efficiency indicator Ief' may be used as for comparing models and methods for their synthesis, as to optimize the process of model building. (68) (69)
Ief ' indicator can be used also for models based on forest of decision-trees. For the case of ensembles, N and S will remain unchanged, and the error E, the number of selected features N', and the number of adjustable parameters N max will be deterw mined for the entire ensemble of trees. 6
The set of indicators proposed above is extensive and for its application in practice it is advisable to analyze the proposed indicators.
The Fig. 1 presents the classification of a set of proposed indicators characterizing the properties of models based on decision trees and forests.
Horizontally at Fig. 1, indicators are divided into groups according to the complexity of data compilation: sample (indicators characterize the properties of the sample and are model independent), tree model (indicators are defined for a single decision tree model), forest (indicators are defined for a set of decision tree models of a forest).
The more to the right the indicator is located horizontally at Fig. 1, the higher the level of complexity of data generalization by the model is required to determine it.
Vertically at Fig. 1, indicators are divided into groups according to the level of computational complexity relative to primary characteristics: basic properties (easily identifiable characteristics of data and models), primary indicators (indicators determined on the basis of basic properties), secondary indicators (indicators determined on based on primary indicators), and integrative indicators (indicators determined on the basis of indicators of previous levels).
The higher the level of the indicator, the more difficult it is to calculate it with respect to the primary properties of the data and models. 7
The problem of creation of a quality model for models based on decision trees and forests is solved.
The set of indicators characterizing properties of decision trees and forests is proposed. It allows to quantitatively evaluate such model properties as diversity, equivalence, retraining, confidence in a decision-making, hierarchy, equifinality, generalization, nonlinearity, robustness, homogeneity, sensitivity to the input signals, plasticity, variability adaptability, symmetry, asymmetry, emergence (integrity), interpretability (logical transparency), learnability, and autonomy as for individually considered (single) tree model, as for a an ensemble of tree models (forest).
The prospects of further study are to obtain estimates of the computational (time) and spatial (memory) complexity of calculating the proposed indicators, to conduct an experimental study of the set of the proposed indicators for assessing the properties of models in solving practical problems of diagnosis and automatic classification on features, to identify the relationships between different indicators of the properties of models based on decision trees and forests. I Rb Iinterp.
Icert (x) Idiv(tree, x, y ) Idiv (tree, X ,Y ) I div (tree) Icert(xs ) I h I I x Rb w Rb I tol
Itol j max n N' M Nn N N f fi f max fmin
min o ( j) i*
aut np (i) sp(i)
I ' ef ~ I nl I gen I lr I adapt IG I nl I aut I I v I hn Ieq (t1, t2 )
w w w max max w Nw N max w w ij max wi, j wim,jin wj Nw(i) w
i, j Ieqf (tree, x, y )
L(tree) Level
Sample
Tree model
Forest model Fig. 1. Analysis of decision tree and forest indicators
Iasym I sym n Iasym w I asym I pl
E I n sym I w sym I I np sp El x ( y )
j ( x, xtest ) v (n(, i)) v (w(, i))
N
met N aut
met aut (wi )
Idiv ( forest, x, y )
I div ( forest, X ,Y )
I forest ( x) cert I forest ( xs ) cert
Idiv ( forest )
Icfeorrtest ( xs , k ) Ieqf ( forest , x, y ) I forest h
I afuotrest I forest lr I forest interp.
I forest I forest v
I forest sp
Itofolrest w w treet f ,max f ,min N
wf T max 36. Beven, K.J., Freer, J.: Equifinality, data assimilation, and uncertainty estimation in mechanistic modelling of complex environmental systems. Journal of Hydrology, 249: 11–29 (2001). 37. Pedrycz, W.: Fuzzy modelling: paradigms and practice. Springer, Berlin (1996). 38. Hoekstra, A.: Generalisation in feed forward neural classifiers. Technische Universiteit
Delft, Delft (1998).
39. Hoekstra A., Duin, R.: On the nonlinearity of pattern classifiers. In: Proceedings of 13
International conference on Pattern recognition, Vienna, 25-29 August 1996, vol. 4, pp. 271–
275. IEEE, Los Alamitos (1996)
40. Grossberg, S.: Nonlinear neural networks: principles, mechanisms, and architectures.
Neural Networks, 1(
In: Proceedings of Fourth International Conference on Intelligent Technologies, Chiang
Mai, 17-19 December 2003, pp. 478–482. Chiang Mai University, Chiang Mai (2003).
49. Gerrow, K. Synaptic stability and plasticity in a floating world. Current Opinion in
Neurobiology, 20 (
doi: 10.3389/fncom.2013.00007
54. Rusiecki, A., Kordos, M., Kamiński, T., Greń, K.: Training Neural Networks on Noisy
Data. In: International Conference on Artificial Intelligence and Soft Computing (ICAISC
2014) pp. 131-142. Springer, Cham (2014).
55. Martín J.A., de Lope, J., Maravall, D.: Adaptation, Anticipation and Rationality in Natural
and Artificial Systems: Computational Paradigms Mimicking Nature. Natural Computing,
8(
World Scientific, Singapore (2005). 57. Bunge, M. A.: Emergence and Convergence: Qualitiative Novelty and the Unity of
Knowledge, University of Toronto Press, Toronto (2003).
58. Wan, P.: Emergence à la Systems Theory: Epistemological Totalausschluss or Ontological
Novelty? Philosophy of the Social Sciences, 41 (
Theory, pp. 476-546 North-Holland, Amsterdam (1998).
60. Molnar, Ch.: Interpretable Machine Learning. A Guide for Making Black Box Models
Explainable. https://christophm.github.io/interpretable-ml-book/
61. Valiant, L.: A theory of the learnable. Communications of the ACM, 27 (
http://www.encyclopediaofmath.org/index.php?title=Lipschitz_constant&oldid=30687 65. Russell, S., Norvig, P.: Artificial intelligence: a modern approach. Prentice Hall, Upper
Saddle River (2009).
66. Hannan, E. J., Quinn, B. G.: The determination of the order of an autoregression. Journal
of the Royal Statistical Society, Ser. B, 41(
Automatic Control, 19 (
Applications. MIT Press, Cambridge (2005).
71. Rissanen, J.: Modeling by shortest data description. Automatica, 14 (