Data are a valuable resource that keeps a great potential for recovery of the useful analytical information. One of the most promising toolkits to solve the problems of data mining could be the usage of neural network technology. The problem of initial parameters values and ways of neural network construction on example of a multilayer perceptron are considered. Also information about the task and available raw data are taken into account. The modular BP-SOM network, which combines the multi-layered feed-forward network with the Back-Propagation (BP) learning algorithm and Kohonens self-organising maps (SOM), is suggested for visualization of the internal information representation and the resulting architecture assessment. The features of BP-SOM functioning, methods of rule extraction from trained neural networks and the ways of the result interpretation are presented.
results of the assessment based on knowledge of the problem and the available source data. After that, the training and testing processes are taking place. Their results are used in the decision-making process, that the network meets all the requirements.
Another complication in using neural network approach could be related with
the results interpretation and its preconditions. Especially clearly, this problem
appears for the multilayer perceptron (MLP) [
The choice in favor of neural network architecture can be based on knowledge
of the problem being solved and the available source data, their dimension and
the samples scope. There are different approaches for choosing the initial values
of the neural network characteristics. For example, the ”Network Advisor” of
the ST Neural Networks package offers by default one intermediate layer with
the number of elements equals to the half of the sum of the quantity of inputs
and outputs for the multilayer perceptron. In general, the problem of choosing
the number of hidden elements for the multilayer perceptron should account
two opposite properties, on the one hand, the number of elements should be
adequate for the task, and on the other, should not be too large to provide
the necessary generalization capability and avoid overfitting. In addition, the
number of hidden units is dependent on the complexity of the function, which
should be reproduced by the neural network, but this function is not known in
advance. It should be noted that while the number of elements increases, the
required number of observations also increases. As an estimate, it is possible to
use the principle of joint optimization of the empirical error and the complexity
of the model, which takes the following form [
The accuracy of the neural network function approximation increases with the number of neurons in the hidden layer.
When there are ℎ neurons the error could be estimated as (︀ 1 )︀ . Since the number of outputs in the network does not exceed, and typically much smaller than the number of inputs, so the main number of weights in the two-layer ℎ network would be concentrated in the first layer, i.e. the total number of weights in the network as follows: (︀ )︀ . input dimension. In this case, the average approximation error is expressed by
The network description is associated with the models complexity and is basically comes down to the consideration of the amount of information in the transmission of its weights values through some communication channel. If we accept the hypothesis about the network settings, its weights and the number of neurons, the amount of information (in the absence of noise) while trans = · ℎ , where - is the ferring the weights will be − log ( event before the message arrives at the receiver input. For a given accuracy this description requires about − log ( ) ∼ one pattern associated with the complexity of the model could be estimated as: ∼ , where is the number of patterns in the training set. The error decreases monotonically with increasing the number of patterns. So Haykin, using the results from the work of Baum and Hessler, gives recommendations about the volume of a training sample relative to the number of weighting coefficients and taking into account the proportion of errors allowed during the test, which can be expressed by the inequality: ≥ , where is the proportion of errors which allowed during testing. Thus, when 10% of errors are acceptable then the number of training patterns must be 10 times greater than the number of available bit. Therefore, a specific error for ), where is the probability of this weighting coefficients in the network.
Thus, both components of the network generalization error from expression (1) were considered. It is important that these components are differently depend on the network size (number of weights), which implies the possibility of choosing the optimal size that minimizes the total error: where , ℎ are the quantity of neurons in input and hidden layers, is the number of weights and is the amount of patterns in the training sample. (1) ∼ + ≥ 2 · ︃√ √ Minimum error (equal sign) is achieved with the optimal number of the weights ∼ · which corresponds to the number of neurons in the (2) (3) 3
After the network architecture was selected the learning process is carried out. In
particular, for a multi-layer perceptron this may be the error back-propagation
(BP) algorithm. One of the most serious problems is that the network is trained
to minimize the error on the training set, rather than an error that can be
expected from the network while it will process completely new patterns. Thus,
the training error will differ from the generalization error for the previously
unknown model of the phenomenon in the absence of the ideal and the infinitely
large training sample [
Since the generalization error is defined for data, which are not included in the
training set, the solution may consist of separation of all the available data into
two sets: a training set, which is used to match the specific values to the weights,
and validation set, which evaluates the predictive ability of the network and
selects the models optimal complexity. The training process is commonly stops
with consideration of ”learning curves” which track dependencies of learning and
generalization errors according to the neural network size [
Another class of learning curves uses the dependencies of the neural network
internal properties to its size, and then mapped to an error of generalization.
For example, in [
In simplified form, the following criteria could be formulated to assess the already constructed neural network model: – if the training error is small, and testing error is large, it means that the network includes too much synapses; – if the training error and testing error is large, then the number of synapses is too small; – if all the synapse weights are too large, it means that there are too few synapses.
After evaluation of the neural network model, the decision-making process is taking place about the necessity of changing the number of hidden elements in one or another direction, and the learning process should be repeated. It is worth to mention that modified decision trees, which are based on the firstorder predicate logic, could be applied as a means of decision making support and neural network structure construction automation. 4
Generally speaking, there are two approaches to extract rules from the multilayer neural networks. The first approach is based on extraction of global rules that characterize the output classes directly through the input parameter values. An alternative is in extraction of local rules, separating the multilayer net on a set of single-layer networks. Each extracted local rule characterizes a separate hidden or output neuron based on weighted connections with other elements. Then rules are combined into a set, which determines the behavior of the whole network.
The NeuroRule algorithm is applicable for rules extraction from the trained
multilayer neural networks, such as perceptron. This algorithm performs the
network pruning and identifies the most important features. However, it sets
quite strict limitations on the architecture, the number of elements, connections
and type of activation functions. As an alternative approach may be highlighted
TREPAN type algorithms which extract structured knowledge not only of the
extremely simplified neural networks, but also arbitrary classifiers in the form of
a decision tree [
The decision of such kind of problems could be based on the usage of the
modular neural network BP-SOM[
The main idea is to increase the reactions similarity of the hidden elements
while processing the patterns from the sample, which belong to the same class.
The traditional architecture of the direct distribution network [
The results of the Kohonen maps self-organization are used for changing of
the weight coefficients in the process of network training. This provides an effect,
in which the activation of neurons in the hidden layer will become more similar
to all other cases processing vectors of the same class [
It characterizes the reaction of the hidden layer neurons of the BP-SOM network, which was trained to solve the problem of classifying for two classes. Map on the left corresponds to the base algorithm of back-propagation (BP), the right - with the influence of the Kohonen map, i.e. BP-SOM training. Here white cells correspond to class A, and black cells to class B. In turn, the size of the shaded region of the cell determines the accuracy of the result. So, completely white or completely black cell is characterized by the accuracy of 100% .
This could ensure structuring and visualization of information extracted from data, improve the perception of the under study phenomenon and help in the network architecture selection process. For example, if it is impossible to isolate the areas at the SOM for the individual classes, then there are not enough neurons and their number should be increased. Moreover, this approach can simplify the rules extraction from already trained neural network and provide the result in a hierarchical structure of the consistent rules such as ”if-then”. 5.3
As an example, let’s consider a small test BP-SOM network that will be trained
to solve the classification problem which is defined by the next logical function
[
( 0, 1, 2) = ( 0 ∧ ¯1 ∧ ¯2) ∨ ( ¯0 ∧ 1 ∧ ¯2) ∨ ( ¯0 ∧ ¯1 ∧ 2)
This function is set to True (Class 1) only in case where one of the arguments is True, otherwise the function value is False (Class 0). Two-layer neural network could be used for implementation, which consists of three input elements, three neurons in the hidden layer, and two neurons in the resulting output layer. The dimension for the Kohonen map for the intermediate layer is 3×3 (Figure 4).
Four elements of the Kohonen map acquired class labels with certainty 1 and 5 elements were left without a label, and their reliability is equal to 0 after training (Figure 4).
One of the methods for rules extraction from such a neural network could be
the algorithm, which was designed for classification problems with digital inputs.
It consists of the following two steps [
Thus, it is possible to get the following two rules from the Table 1: – – ( 0 = 0 ∧ 1 = 0 ∧ 2 = 0) ( 0 = 1 ∧ 1 = 1 ∧ 2 = 1)
However, it is rather problematic to use this way to distinguish rules for elements 7 and 9. Of course, it is possible to compose a disjunction of all the possible options for each SOM-element, but these rules would be complicated in perception.
Additional available information consists of the attributes sum for each patterns from the training sample. It is easy to notice that each SOM-element is responsible for a certain obtained sum value (Table 2).
1 3 7 9 Sum
0
Class 0
3
0
1
1
2
0
attribute values of the input vectors and their weight coefficients for extraction
rules, which corresponds to Kohonen map elements. This may be done by
backpropagation of minimum and maximum values of the neuron activation back
to the previous layer, i.e. we have to apply the function −1(
activation function) to the output neuron value [
+ )) = + , (6) where is the -th neuron output from the current layer, is the output of the -th neuron of the previous layer. Assuming that sigmoid was used as the neurons activation function, then in this case we will get: ) = − ln( 1
Additionally, it is known that self-organizing map elements, which are connected to the elements of the first hidden layer of the perceptron, will respond to proximity of the weight vectors and outputs of the hidden layer neurons. Therefore, it is proposed to replace the back-propagation neural activation of the hidden layer to the weight vector values of the self-organization map element during the rules construction for each SOM-element. For example, if the hidden layer contains neuron , and the weight between this neuron and SOM-element was denoted by , then the restriction takes the form: −1( ) − 1 ≈ ( )/( −1( ) − ).
(8) ︁∑
Such restrictions, which were obtained for all the neurons of the hidden layer, could be used for construction of following rules: (∧ ( −1( ) − ≈ ︁∑ )) ( = ( )).
(9) Similar rules are required for all SOM-components, the reliability of which exceeds a certain threshold.
If we apply the considered method for the initial example, then four sets of restrictions would be obtained. Each set includes three restrictions, which corresponds to the number of neurons in the hidden layer of the perceptron.
For SOM-element 1 : – 1 ≈ 687 * 0 + 687 * 1 + 687 * 2; – 1 ≈ 738 * 0 + 738 * 1 + 738 * 2; – 1 ≈ 1062 * 0 + 1062 * 1 + 1062 * 2. – 1 ≈ 0 + 1 + 2;
Restrictions for element 9: – 1 ≈ 0.5 * 0 + 0.5 * 1 + 0.5 * 2;
Taking into account that 0, 1, 2 ∈ {0, 1} then the best results may be achieved
For element 3 all restrictions coincide and have the form: – 1 ≈ 0.33 * 0 + 0.33 * 1 + 0.33 * 2; Thus, the values are as follows: 0 = 1, 1 = 1, 2 = 1.
For element 7 restrictions coincide and take the form: This corresponds to the case when only one of the attributes is equal to 1. This condition characterizes the case when two of three attributes are 1.
Thus, it is obtained the following set of rules, if all restrictions would be generalized: – – – – ( 0 + 1 + 2 ≈ 0) ( 0 + 1 + 2 ≈ 3) ( 0 + 1 + 2 ≈ 1) ( 0 + 1 + 2 ≈ 2) ( ( ( ( It is easy to notice that this set correctly describes all the elements of a selforganizing network. 6
Approaches for the selection of the initial values of the neural network parameters were considered. The analysis of the training process, its stopping criteria and evaluation of the received architecture were carried out. Combining different neural network architectures, such as multi-layer perceptron with the backpropagation learning algorithm, and Kohonen’s self-organizing maps, could bring additional possibilities in the learning process and in the rules extraction from trained neural network. Self-organizing maps are used both for information visualization, and for influencing the weights changes during the network training process. That provides an effect, in which the neurons activation in the hidden layer will become increasingly similar to all the cases processing vectors of the same class. This ensures the extracted information structuring and has the main purpose to improve the perception of the studied phenomena, assist in the process of selecting the network architecture and simplify the extraction rules. The results could be used for data processing and hidden patterns identification in the information storage, which could become the basis for prognostic and design solutions.
Acknowledgeents. Work is carried out with the financial support of the RFBR (grant 15-07-01117-a).