This paper is devoted to the complex problem of synthesis of artificial neural networks. Firstly, the existing methods, recommendations and solutions of the problem are consider. As a new solution, the mechanism of using the modification of the genetic algorithm to determine the weights of the hidden and output layers (network training) is proposed. Testing and comparison of the results with the results of the existing methods were carried out for the correct evaluation of the method.
In modern medicine, the main task for the use of information technology is to significantly improve the quality indicators in the diagnosis and therapy of various diseases. The existing methods and algorithms of modeling nonlinear systems are faced with problems of high dimensionality of tasks, the requirements of high accuracy and generalizing ability of the obtained models. These problems can be solved with the help of per-boron and iterative methods, which are based on the principles of selection, evolution and adaptation, which are methods of heuristic self-organization. At the same time, in real life it is quite difficult to create an adequate model of a complex object using only one method of inductive modeling. Usually it is required to combine modern methods and technologies of heuristic self-organization, to apply multilevel modeling, to develop hybrid algorithms.
Insufficiency or overabundance of data are frequent problems in solving tasks: there is not enough experiments (in the modeling of physical objects), not enough or difficult to allocate informative data on patients (to build a prediction of health). It is necessary to determine the parameters of the new element, to predict the outcome of the disease, to recommend treatment. Often artificial neural networks (ANNs) are used to solve such problems.
In a number of works [
However, despite all the advantages of ANN, there is a large number of difficulties
in their implementation in medicine. Creation of ANNs is reduced to performance of
the main steps:
the choice of the structure of ANN;
set weights of all neurons of the ANN (training of ANN) [
At the present to implement these tasks are not the rigorous methods of solution, there are only common recommendations. The proposed methods are aimed at solving local problems, which often leads to an unsatisfactory structure of ANNs and a significant training time.
In this paper, the authors propose a method of using a genetic algorithm for the synthesis of ANNs – creation of the structure and configuration of the network weights (learning with the teacher). 2
The basis of ANNs are neurons with a structure similar to biological analogues. Each neuron can be represented as a microprocessor with several inputs and one output. When neurons are joined together, a structure is formed, which calls a neural network. Vertically aligned neurons form layers: input, hidden and output. The number of layers determines the complexity and, at the same time, the functionality of the network, which is not fully investigated.
For researchers, the first stage of creating a network is the most difficult task. The
following recommendations are given in the literature [
Based on these recommendations, the number of layers and the number of neurons in the hidden layers is chosen by the researcher, based on his personal experience.
In a number of works [
It should be noted that the BP method was one of the first methods for ANNs
training. Most of all brings trouble indefinitely long learning process. In complex tasks, it
can take days or even weeks to train a network, and it may not train at all. The cause
may be one of the following [
It should also be noted the possibility of retraining the network, which is rather the result of erroneous design of its topology. With too many neurons, the property of the network to generalize information is lost. The training set will be examined by the network, but any other sets, even very similar ones, may be misclassified.
The Backpropagation through time (BPTT) method has become a continuation,
which is why it is faster. Moreover, it solves some of the problems of its predecessor.
However, the BPTT experiences difficulties with local optima. In recurrent neural
networks (RNN), the local optimum is a much more significant problem than in
feedforward neural networks. Recurrent connections in such ANNs tends to create chaotic
reactions in the error surface, resulting in local optima appearing frequently. Also in
the blocks of RNN, when the error value propagates back from the output, the error is
trapped in the part of the block. This is referred to as the “error carousel”, which
constantly feeds the error back to each of the valves until they become trained to cut off
this value. Thus, regular back propagation is effective when training an RNN unit to
memorize values for very long durations [
The Hebb method does not guarantee the convergence of the learning process, i.e.
the error of approximation of the function by the neural network may exceed the
permissible value. The main disadvantage of the using Hebb method is that the
convergence of the algorithm decreases with increasing dimension n of the input vector. For
n > 5, it is difficult to guarantee convergence. This method is usually used as some
element in other learning algorithms. The most preferable is the use of the Hebb rule
in unsupervised learning algorithms [
For the training by connectionist temporal classification (CTC) [
The above problems explain the urgency of developing a new approach to the synthesis of ANNs. The main purpose of the authors is to research and test a new method to solve the second problem, namely, training ANNs, which will be based on the use of genetic algorithms, as a means of determining the weights of the hidden and output layers. 3
In the method, which is proposed to find a solution using a population of neural
networks, that is, each individual is a separate ANN [
After initial initialization, all individuals have coded networks in their genes
without hidden neurons, and all input neurons are connected to each output neuron. That
is, at first, all the presented ANNs differ only in the weights of the interneuron bonds.
In the process of evaluation, based on the genetic information of the individual under
consideration, a neural network is first built, and then its performance is checked,
which determines the fitness of the individual. After evaluation, all individuals are
sorted in order of reduced fitness, and a more successful half of the sorted population
is allowed to cross, with the best individual immediately moving to the next
generation. In the process of reproduction, each individual is crossed with a randomly
selected individual from among those selected for crossing. The resulting two
descendants are added to the new generation. Once a new generation is formed the mutation
operator starts working. However, it is important to note that the selection of the
truncation significantly reduces the diversity within the population, leading to an early
convergence of the algorithm, so the probability of mutation is chosen to be rather
large, about 15-25% [
If the best individual in the population does not change for more than 7 generations, the algorithm is restarted. During the restart, the entire population is reinitialized and the solution search process starts from scratch. This makes it possible to realize the exit from the areas of local minima due to the relief of the objective function, as well as a large degree of convergence of individuals in one generation. 3.1
It is obvious that the chosen method requires special genetic operators that implement crossover and mutation.
At crossover two parental individuals which produce two descendants are used. Common neurons and connections are inherited by both offspring, and the value of connections in the networks of descendants are formed by a two-point crossover. Elements of ANN, of distinct “played out” between generations.
An important feature is that neurons with the same indices are considered identical, despite the different number of connections and position in the network, as well as the fact that one of these neurons could have a different index, which changed as a result of correction of indices after mutation. For this purpose, two coefficients were introduced that regulate the size and direction of the network.
The first of them characterizes the degree of “connectedness” of neurons in the network and is calculated by the formula: fc
Nc 2FB1N s N s 1 Ni Ni 1 1 FBNo No 1 where Nc is the number of connections in the network, Ni , No , Ns are respectively, the number of input, output neurons and the total number of neurons in the network, FB is a variable indicating the permitted occurrence of feedbacks ( FB =1) or not ( FB =0). It is worth noting that connections from hidden neurons to the output can appear in any case. Thus, the smaller f c the more likely a new relationship will be added as a result of the mutation. There are 4 possible uses fc : (1) (2) calculated by the formula; squared; multiplied by some factor; squared and multiplied by some factor.
The use of the second coefficient is based on the assumption that the more elements in the sum of the input and output vectors of the training choice (the greater the total number of input and output neurons), which is probably a more complex network is necessary to solve the problem. The second coefficient is calculated by the formula: f n
Ni No
N s .
That is, the more neurons in the network, the less will be f n and the less likely will be selected mutation that adds a new hidden neuron. As well as fc there are possible
calculated by the formula; squared; multiplied by some factor; squared and multiplied by some factor.
For any of the described cases, the algorithm uses a ligament fn fc , because for use it is necessary to take into account the degree of connectivity of already existing neurons.
Thus, using mutations can be “pointwise” to change the parameters of the structure of the ins.
Chaotic the addition (removal) of neurons and connections can lead to situations where, for example, in a network of many neurons and few connections. It would be more logical to apply different types of mutations depending on the features of the network architecture represented by the mutating individual.
Removing links ins gives a side effect: there may be “hanging” neurons that have no incoming connections, as well as dead-end neurons, that is, without output connections. In cases where the function of neuronal activation is such that at zero weighted sum of inputs its value is not equal to zero, the presence of “hanging” neurons makes it possible to adjust the neural displacement. It is worth noting that, on the other hand, the removal of links may contribute to the removal of some uninformative and uninformative input features. 3.2
Consider the dependence of the type of mutation on the values f c and f n . Adaptive mutation mechanism is one of the key features of the proposed method.
The choice of mutation type is determined based on the values of f c and fn fc . This approach, on the one hand, does not limit the number of hidden neurons “from above”, on the other hand, it prevents the immeasurable increase of the network, because the addition of each new neuron to the network will be less likely. The mutation of the weight of a random existing bond occurs for all mutating individuals with a probability of 0.5.
Let us consider in more detail how to choose the type of mutation. Fig. 1 shows the block diagram of the selection of the type of mutation. Here RV is a random variable, N h is the number of hidden neurons in the mutating network. For short the selection and calculation f c and f n , and mutation of the weight by accident you select the relationship in the schema is not specified.
Conventionally, the entire algorithm can be divided into two “branches” on the first conditional transition: 1. branch increase f c is carried out for the fulfilment of the conditions of transition; 2. branch reduction f c , performed if the transition condition is not met. Since the removal of a neuron can lead to both reduction and increase depending on the number of connections of the neuron, this option is present in both branches. Thus, the main factor for the regulation of the structure of the obtained ANN is the degree of their “connectivity”. In other words, how fully the possible network connections are implemented.
Multiplication f n by fc is necessary in order to change the number of neurons adequately network topology, because the addition (removal) of neurons need information about the feasibility of changes. This information can be obtained indirectly from the value of the characteristic. 3.3
The value of the mean square error is replaced by the criterion of maximum separation of support vectors. In this case, the optimal linear weights can be estimated using quadratic programming, as in the traditional support vector machine.
One of the problems of neuroevolutionary method realization is the algorithm of ANN output calculation with arbitrary topology.
ANN can be represented as a directed planar graph. Based on the fact that the network structure can be any, loops and cycles containing any nodes are allowed in the graph, except for the nodes of the corresponding input neurons. Let denote the set of nodes of the graph by V vi | i 0; Nv 1, and a set of arcs through E e j | j 0; Ne 1, where Nv and Ne are accordingly, the number of nodes and arcs in the graph, and N v N s , and N e N c . The arc, which goes from node k to node 1 denote by an ordered pair ek,l vk , vl , the weight of the corresponding link will be denoted by wk,l .
Give the index to the nodes of the graph as neurons, that is, the nodes that are the input neurons, called input. have an index out of range 0; Nl 1 . By analogy, the indexes of outgoing nodes belong to the interval Nl ; Nl No 1, and indexes for hidden nodes will be set in the interval Nl No ; Nv 1.
Let introduce an additional characteristic for all nodes of the graph equal to the minimum length of the chain to any of the input nodes and denote it li . Let's call li the layer to which the ith node belongs. Thus, all input nodes belong to the 0th layer, not all input nodes that have input arcs from the input belong to the 1st layer, all other nodes with input arcs from nodes of the 1st layer will belong to the layer with index 2, etc .in this case, there may be situations when the node does not have input arcs, we will call it a hanging node with the layer number li 1 .
For arcs, we also introduce an additional characteristic bk,l for the arc ek ,l , which is necessary to determine whether the arc corresponds to forward or reverse. It will be calculated as follows:
1, ll lk 0 bk,l 1, ll lk 0 .
(3) That is, if the index of the layer of the end node of the arc is greater than the index of the layer of the beginning node, then we will consider such an arc as a straight line, otherwise we will consider the arc as an inverse.
Since each node of the graph represents a neuron, we denote by sumi the value of the weighted sum of inputs, and through oi is the value of the output (the value of the activation function of the ith neuron-node). Then, oi f sumi where f is the function of neuron activation.
Let's divide the whole process of signal propagation from the input nodes into stages, and during one such stage the signals “manage” to pass only one arc. The number of the stage is denoted by s. For the very first stage s=1. For short assumed that all arcs have the same length, and the signals are sewn on them instantly. We denote the feature that the output of node i was updated at this stage through ai , that is, if ai 1, then the output of the node at stage s is calculated, otherwise, if ai 1 – not.
Let's introduce one more designation X xi | i 0; Nl 1 it is vector of input signals. Then the algorithm for calculating the ANN output is as follows: 1. oi xi , ai 1, for all i 0; Nl 1; 2. oi 0 , for all i Nl ; Ns 1 ; 3. s=1; 4. sumi 0 , ai 1, for all i Nl ; Ns 1 ; 5. if s 1 , than go to the step number 7; 6. calcultion of the feedback network. For all input reverse arcs e j,k node vk , where k Nl ; Ns 1: sumk sumk o j , if l j s ; 7. if ai 0 , than fn(i) for all i Nl ; Ns 1 ; 8. if the stop criterion is not met, than s=s+1 and go to the step number 4. Here fn(i) is a recursive function that calculates the output of the 1st node taking into account all straight arcs. Works on the following algorithm:
2. for all input arcs ek,l node vi : if ak 1 , than sumi sumi ok , else fn(k) ;
The stopping criterion of the ANN output calculation algorithm can be one of the following: stabilization of values at the output of ANN; s exceeds the set value.
It is more reliable to calculate the output until the values at the output of ANN do not change, but for the case when the network contains cycles and/or loops, its output may never become stable. Therefore, the required additional stopping criteria limiting the maximum number of stages of calculation of network output. For networks with no feedback ( FB =0) in many cases, allow the maxli 1 phases. 4
For a full assessment, we will conduct a series of experiments. A sample of data on
patients diagnosed with Mesothelioma will be used as data. The sample is publicly
available [
Mesothelioma is a rare, aggressive form of cancer that develops in the lining of the
lungs, abdomen, or heart. Caused by asbestos, mesothelioma has no known cure and
has a very poor prognosis [
During the experiments, the special attention was paid to the main characteristics of
the methods, namely: the time spent, the boundary values of the error, the mean value
of the error and the average error of final network. The results are shown in table 2.
As experiments have shown, in all cases, the modified genetic method showed better
results than existing methods. On the other hand, it should be noted that the difference
in the mean error of the finite network in the second group of experiments (when
comparing the back propagation in time and the modified genetic algorithm) is not so
great. To improve the operation of the proposed method in the case of recurrent ins, it
is possible to resort to the strategy of using support vector machine (SVM) to clarify
the weights of the output layer, as proposed in [
Fig. 2 shows the distribution of iterations during the experiments.
As can be seen from the diagram, the modified genetic method was more highly
iterative than the existing methods, but the time spent on the iteration was less. That is, we
can conclude that the iterations are not complex and to reduce them we can resort to
parallelization [
The problem of finding the optimal method of synthesis of ANN requires a comprehensive approach. Existing methods of ANNs training are well tested, but they have a number of nuances and disadvantages. The paper proposes a mechanism for the use a modified genetic algorithm for its subsequent application in the synthesis of ANNs. Based on the analysis of the experimental results, it can be argued about the good work of the proposed method. However, to reduce iterativity and improve accuracy, it should be continued to work towards parallelization of calculations and the using of SVM.
The work was performed as part of the project “Methods and means of decisionmaking for data processing in intellectual recognition systems” (number of state registration 0117U003920) of Zaporizhzhia National Technical University.