The main aim of the paper is researching the possibility of application artificial neural networks as the neural network models that can be used in medical diagnostics. One of the most problematic and complex issues of neural network models implementation is the initial stage of synthesis. The article presents a comparison of existing methods of synthesis, as well as a new method. The experiments confirm the effectiveness and expediency of the proposed method.
The diagnosis stage plays a crucial role in medicine. Timely accurate diagnosis
facilitates the choice of therapy and significantly increases the probability of treatment of
the patient. The using of neural networks is one of the ways to improve the efficiency
of medical diagnosis [
The accuracy of the diagnosis and the speed with which it can be delivered depend
on many factors: the patient's condition, the available data on the symptoms and signs
of the disease and the results of laboratory tests, the total amount of medical
information on the observation of such symptoms in a variety of diseases and, finally, the
qualification of the doctor. A major role in this process is played by the human factor,
which often leads to errors [
Some of the specific medical diagnosis difficulties that need to be considered are
listed below [
The basis for a reliable diagnosis is a wealth of practical experience, which can be reached only the middle of a doctor's career and is absent at the end of training, of course. This is especially true for rare or new diseases, where experienced doctors are in the same situation as beginners.
The quality of diagnosis depends on the skill, knowledge and intuition of the doctor.
Emotional problems and fatigue adversely affect the work of the doctor.
Training of specialists is a long and expensive procedure, and therefore in many, even in developed countries, there is a lack of skills.
Medicine is one of the fastest growing and developing fields of science. New results disqualify the old ones, new drugs appear every day. The same applies to the diseases themselves, which take new forms.
These factors necessitate the search for new solutions and tools, for example, the
use of artificial neural networks (ANNs) [
The ANN technologies are designed to solve difficult-to-formalize problems, which,
in particular, are reduced to many problems of medicine [
A number of difficulties are associated with the use of neural networks in practical
problems. One of the main problems of application ANN technologies is a previously
unknown degree of complexity of the projected ANN, which will be enough for a
reliable diagnosis. This complexity can be unacceptably high and will require more
complex network architecture. It is known, for example, that the simplest single-layer
neural networks are able to solve only linearly separated problems [
The basis of ANNs are neurons with a structure similar to biological analogues. Each neuron can be represented as a microprocessor with several inputs and one output. When neurons are joined together, a structure is formed, which calls a neural network. Vertically aligned neurons form layers: input, hidden and output. The number of layers determines the complexity and, at the same time, the functionality of the network, which is not fully investigated.
For researchers, the first stage of creating a network is the most difficult task. The
following recommendations are given in the literature [
Based on these recommendations, the number of layers and the number of neurons in the hidden layers is chosen by the researcher, based on his personal experience. 3
The ANN are attractive from an intuitive point of view, because they are based on a primitive biological model of nervous systems. In this connection, there is an assumption that to improve it may be appropriate to apply another borrowing from nature, for example, evolutionary calculations and, in particular, neuroevolution. Neuroevolution in this paper refers to the automatic modification of neural networks using genetic algorithms. With this methodology, possible variations of neural networks with different topologies are grown, which with each iteration, called generation solve the problem better. Despite genetic programming, as well as evolutionary calculations in general, do not guarantee finding the optimal result, this approach eventually allows us to come to the results applicable to solving practical problems. However, it will take a reasonable amount of time to achieve such results. Thus, the level of complexity of the neural network that arises when it is necessary to create a neural network is significantly reduced, because when it is created, it is only necessary to select the parameter that evaluates the work of the neural network and provide a suitable set of data.
Despite the fact that most of the works devoted to the neuroevolutionary approach
offer only a theoretical approach to solving problems of neural network optimization,
it is possible to find several promising and noteworthy methods [
From the early works of noteworthy cellular Frederick Gruau method [15], [16] uses a special grammar for the representation of neural network structures. One individual represented an entire neural network, with each neuron considered as a biological cell, and the growth of the network was determined through the mechanisms of sequential and parallel "division" of neurons - cells. However, this method involves the implementation of a large number of specific operators that provide simulation of cell activity.
The Hierarchical SANE (Symbiotic, Adaptive NeuroEvolution) [17] method uses a different approach. It is consider the development of two independent populations, one of which individuals are separate neurons, and the other contains information about the structures of an artificial neural network. The disadvantages of this method include the fact that the number of hidden neurons and connections is limited.
The ESP method [18] is a development of the sane method. Its main difference is that the network structure is fixed and is given a priori. The population of neurons is divided into subpopulations, in each of which the evolution is independent. Due to parallelization of the solution search, as well as simplification of the problem due to the rejection of the evolution of the artificial neural network structure, ESP works much faster than SANE, sometimes by an order of magnitude, but for the successful operation of the method it is required to choose the appropriate structure of the neural network [19].
One of the most potentially successful attempts to get rid of the disadvantages of direct coding while preserving all its advantages is the method proposed in 2002, called NEAT — Neural Evolution through Augmenting Topologies [15], [20]. Designed by Kenneth Stanley, the NEAT method allows to customize the structure of the network, and without restrictions on its complexity. The solution proposed by the authors is based on the biological concept of homologous genes (alleles), as well as on the existence in nature of the synapsis process — the alignment of homologous genes before the crossover. The technique assumes that two genes (in two different individuals) are homologous if they are the result of the same mutation in the past. In other words, with each structural mutation (gene addition), a new gene is assigned a unique number, which then does not change during evolution. The method uses a number of techniques, such as historical labels and specialization of individuals, to make the process of evolution significantly more efficient [21].
Summing up, it can be noted that the joint use of evolutionary methods and artificial neural networks allows us to solve the problems of configuration and training of artificial neural networks both individually and simultaneously. One of the advantages of this synthesized approach is largely a unified approach to solving a variety of problems of classification, approximation, control and modeling. The use of qualitative evaluation of the functioning of artificial neural networks allows the use of neuroevolutionary methods to solve the problems of the study of adaptive behavior of intelligent agents, the search for game strategies, signal processing. Despite the fact that the number of problems and open questions concerning the development and application of neuroevolutionary methods (coding methods, genetic operators, methods of analysis, etc.) is large, often for the successful solution of the problem with the use of neuroevolutionary method adequate understanding of the problem and neuroevolutionary approach, as evidenced by a large number of interesting and successful works in this direction [15]. 4
In the method, which is proposed to find a solution using a population of neural networks: P NN 1 , NN 2 ,..., NN n , that is, each individual is a separate ANN Ind i NN i [19–21]. During initialization population divided into two halves, the genes g Indi g1, g 2 ,..., g n of the first half of the individuals is randomly assigned g Indi g1 Rand, g 2 Rand,..., g n Rand. Genes of the second half of the population are defined as the inversion of genes of the first half g Indi g1 Rand , g 2 Rand ,..., g n Rand . This allows for a uniform distribution of single and zero bits in the population to minimize the probability of early convergence of the method ( p min ).
After initialization, all individuals have coded networks in their genes with-out hidden neurons (Nh), and all input neurons (Ni) are connected to each output neuron (No). That is, at first, all the presented ANNs differ only in the weights of the interneuron connection wi. In the process of evaluation, based on the genetic information of the individual under consideration, a neural network is first built, and then its performance is checked, which determines the fitness function ( f fitness ) of the individual. After evaluation, all individuals are sorted in order of reduced fitness, and a more successful half of the sorted population is allowed to cross, with the best individual immediately moving to the next generation. In the process of reproduction, each individual is crossed with a randomly selected individual from among those selected for crossing. The resulting two descend-ants are added to the new generation G P` Ind1 , Ind 2 ,..., Ind n . Once a new generation is formed the mutation operator starts working. However, it is important to note that the selection of the truncation significantly reduces the diversity within the population, leading to an early convergence of the algorithm, so the probability of mutation is chosen to be rather large ( p mut 15 - 25% ) [22].
If the best individual in the population does not change for a certain number of generations (by default, it is proposed to set this number at eight), this individual is forcibly removed, and a new best individual is randomly selected from the queue. This makes it possible to realize the exit from the areas of local minima due to the relief of the objective function, as well as a large degree of convergence of individuals in one generation. The general scheme of the method demonstrated at Fig.1. 4.1
It is obvious that the chosen method requires special genetic operators that implement crossover and mutation.
At crossover two parental individuals which produce two descendants are used. Common neurons and connections are inherited by both offspring, and the value of connections in the networks of descendants are formed by a two-point crossover. Elements of ANN, of distinct played out between generations.
An important feature is that neurons with the same indices are considered identical, despite the different number of connections and position in the network, as well as the fact that one of these neurons could have a different index, which changed as a result of correction of indices after mutation. For this purpose, three coefficients were introduced that regulate the size and direction of the network.
The first of them characterizes the degree of connectedness of neurons in the network and is calculated by the formula:
Nc fcon 2FB1Ns Ns 1 Ni Ni 1 1 FBNoNo 1 (1) where N c is the number of connections in the network, N i , No , N s are respectively, the number of input, output neurons and the total number of neurons in the network, FB is a variable indicating the permitted occurrence of feedbacks ( FB =1) or not ( FB =0). It is worth noting that connections from hidden neurons to the output can appear in any case. Thus, the smaller f con the more likely a new relationship will be added as a result of the mutation [23].
P NN 1 , NN 2 ,..., NN n g Indi g1 , g 2 ,..., g n w i
Ind i
G P` Ind1, Ind 2 ,..., Ind n The use of the second coefficient is based on the assumption that the more elements in the sum of the input and output vectors of the training choice (the greater the total number of input and output neurons), which is probably a more complex network is necessary to solve the problem. The second coefficient is calculated by the formula: (2) (3)
Ni No
Ns fcomp;.diff
Ni No
Ns
That is, the more neurons in the network, the less will be f top.diff and the less likely will be selected mutation that adds a new hidden neuron [23].
The third criterion is also based on the assumption that a more complex network should be used to solve more complex problems. However, this criterion characterizes the conditional complexity of the network. This criterion is based on the concept of cyclomatic complexity [24], [25].
For any of the described cases, the algorithm uses a ligament f con f top.diff f comp.diff , because for use it is necessary to take into account the degree of connectivity of already existing neurons.
Thus, using mutations can be pointwise to change the parameters of the structure of the ins.
Chaotic the addition (removal) of neurons and connections can lead to situations where, for example, in a network of many neurons and few connections. It would be more logical to apply different types of mutations depending on the features of the network architecture represented by the mutating individual [26–28].
Removing links gives a side effect: there may be hanging neurons that have no
incoming connections, as well as dead-end neurons, that is, without output connections
[26], [27], [
Consider the dependence of the type of mutation on the values f con , f top.diff and f comp.diff . Adaptive mutation mechanism is one of the key features of the proposed method.
The choice of mutation type is determined based on the values of f con , f top.diff and f comp.diff . This approach, on the one hand, does not limit the number of hidden neurons from above, on the other hand, it prevents the immeasurable increase of the network, because the addition of each new neuron to the network will be less likely. The mutation of the weight of a random existing bond occurs for all mutating individuals with a probability of 0.5.
Let us consider in more detail how to choose the type of mutation. Fig. 2 shows the block diagram of the selection of the type of mutation. Here RV is a random variable, Nh is the number of hidden neurons in the mutating network.
Conventionally, the entire algorithm can be divided into two branches on the first conditional transition: ─ branch increase f c is carried out for the fulfilment of the conditions of transition; ─ branch reduction f c , performed if the transition condition is not met. Multiplication f con f comp .diff is necessary in order to change the number of neurons adequately network topology, because the addition (removal) of neurons need information about the feasibility of changes. This information can be obtained indirectly from the value of the characteristic. 4.3
On condition using the support vector machine, the optimality criterion for calculating
the output weights may not be specified. If the value of the mean square error is
replaced by the criterion of the maximum separation of the support vectors, then the
optimal linear weights of the output can be estimated using, for example, quadratic
programming, as in the traditional method of support vectors, for this it is advisable to
use the Evoke operator [
k li yt w0 wi, j K t , i j, i1 j0 (4) where t Rn is the output of a recurrent neural network f at a time t ; K , is a predefined kernel function; wi, j is weights corresponding to k training sequences i , each length li , and are calculated using the support vector machine.
The value of the mean square error is replaced by the criterion of maximum separation of support vectors. In this case, the optimal linear weights can be estimated using quadratic programming, as in the traditional support vector machine.
One of the problems of neuroevolutionary method realization is the algorithm of ANN output calculation with arbitrary topology.
ANN can be represented as a directed planar graph. Based on the fact that the network structure can be any, loops and cycles containing any nodes are allowed in the graph, except for the nodes of the corresponding input neurons. Let denote the set of nodes of the graph by V vi | i 0; N v 1, and a set of arcs through E e j | j 0; Ne 1, where N v and N e are accordingly, the number of nodes and arcs in the graph, and Nv Ns , and Ne Nc . The arc, which goes from node k to node 1 denote by an ordered pair ck ,l vk , vl , the weight of the corresponding link will be denoted by wk,l .
Give the index to the nodes of the graph as neurons, that is, the nodes that are the input neurons, called input, have an index out of range 0; N l 1 . By analogy, the indexes of outgoing nodes belong to the interval N l ; N l N o 1, and indexes for hidden nodes will be set in the interval N l N o ; N v 1.
Let introduce an additional characteristic for all nodes of the graph equal to the minimum length of the chain to any of the input nodes and denote it chi . Let's call chi the layer to which the ith node belongs. Thus, all input nodes belong to the 0th layer, not all input nodes that have input arcs from the input belong to the 1st layer, all other nodes with input arcs from nodes of the 1st layer will belong to the layer with index 2, etc .in this case, there may be situations when the node does not have input arcs, we will call it a hanging node with the layer number chi 1 .
For arcs, we also introduce an additional characteristic bk ,l for the arc c k ,l , which is necessary to determine whether the arc corresponds to forward or reverse. It will be calculated as follows:
1, chl chk 0 bk,l 1, chl chk 0 (5)
That is, if the index of the layer of the end node of the arc is greater than the index of the layer of the beginning node, then we will consider such an arc as a straight line, otherwise we will consider the arc as an inverse.
Since each node of the graph represents a neuron, we denote by sumi the value of the weighted sum of inputs, and through oi is the value of the output (the value of the activation function of the ith neuron-node). Then, oi f fitnesssumi where f fitness is the function of neuron activation.
Let's divide the whole process of signal propagation from the input nodes into stages, and during one such stage the signals manage to pass only one arc. The number of the stage is denoted by s. For the very first stage s=1. For short assumed that all arcs have the same length, and the signals are sewn on them instantly. We denote the feature that the output of node i was updated at this stage through ai , that is, if ai 1, then the output of the node at stage s is calculated, otherwise, if ai 1 – not.
Let's introduce one more designation X xi | i 0; N l 1 it is vector of input signals. Then the algorithm for calculating the ANN output is as follows: 1. oi xi , ai 1 , for all i 0; Nl 1; k Nl ; N s 1: sumk sumk o j , if ch j s ; 7. if ai 0 , than fn(i) for all i N l ; N s 1; 8. if the stop criterion is not met, than s=s+1 and go to the step number 4.
Here fn(i) is a recursive function that calculates the output of the 1st node taking into account all straight arcs. Works on the following algorithm: 1. if chi 0 , than go to the step number 3; 2. for all input arcs c k ,l node vi : if ak 1 , than sum i sum i o k , else fn(k) ; 3. oi f sumi ; 4. exit.
The stopping criterion of the ANN output calculation algorithm can be one of the following: ─ stabilization of values at the output of ANN; ─ s exceeds the set value.
It is more reliable to calculate the output until the values at the output of ANN do not change, but for the case when the network contains cycles and/or loops, its output may never become stable. Therefore, the required additional stopping criteria limiting the maximum number of stages of calculation of network output. For networks with no feedback ( FB =0) in many cases, allow the maxchi 1 phases. 5
During testing, the main task is to track the speed of the proposed method, quality and stability. Since synthesized ANN can be further used as diagnostic models for medical diagnosis, testing should be carried out on the relevant test data. Also, testing will be carried out in 2 stages: the first stage will consist in the synthesis of ANN only with the help of modified genetic algorithm, and the second – in additional processing of the initial layer by the support vector machine. This strategy will allow to know more clearly how useful the support vector machine is.
Data for testing were taken from the open repository – UC Irvine Machine
Learning Repository. Data sample was used: Breast Cancer Coimbra Data Set [
The relative error value in this case will be calculated as the ratio of the classification error to the total sample size (number of instances).
E
errorclass
100% , where E is relative error; errorclass is classification error; Numbersampl the number of instances in the sample.
Standard deviation gives an idea about how one or the other, the ANN accurately predicts the user's rating, since the estimation is calculated the difference between the result of the work of ANN’s and known result. It is also important to know that this indicator can be calculated only with a sufficient amount of observations. Otherwise, the calculation of the SD will be uninformative and its use will not lead to improvement of the results of the ANN.
SD
1
Numbersampl 2
xi x ,
i1
(6)
(7)
where SD is standard deviation, xi is ith element of the set, Numbersampl the number
of instances in the sample, x is the mean value of these observations [
As the table shows, the modified GA according to the time of fulfillment is ahead of ESP in time, however, inferior to the NEAT. However, it should be noted that in the analysis of error values, the proposed method is significantly ahead of existing methods.
Let's repeat testing, but now with additional use of the support vector machine. The results are shown in Table 3.
As can be seen from the table, the modified GA using the support vector machine is inferior to the opponents in terms of execution time. However, on indicators of minimum, maximum errors and average errors of the output ANN significantly better than their competitors. Therefore, we can conclude that the use of the support vector machine really significantly improves the results of the synthesis.
As you can see from the diagram, the modified genetic method was more iterative
than the existing methods, but the time spent on iteration was less. That is, it can be
concluded that iterations are not complex and for their reduction it is possible to resort
to parallelization, which will significantly speed up the work even when using the
support vector machine [
The problem of finding the optimal method of synthesis of ANN requires a comprehensive approach. Existing methods of ANNs training are well tested, but they have a number of nuances and disadvantages. The paper proposes a mechanism for the use a modified genetic algorithm for its subsequent application in the synthesis of ANNs. Based on the analysis of the experimental results, it can be argued about the good work of the proposed method. However, to reduce iterativity and improve accuracy, it should be continued to work towards parallelization of calculations.
The work was performed as part of the project “Methods and means of decisionmaking for data processing in intellectual recognition systems” (number of state registration 0117U003920) of Zaporizhzhia National Technical University. communications and Computer Engineering (TCSET 2018), pp. 903–908, (2018). doi: 10.1109/TCSET.2018.8336342. 13. Oliinyk, A., Subbotin, S., Lovkin, V., Leoshchenko, S., Zaiko, T.: Feature Selection Based on Parallel Stochastic Computing," 2018 IEEE 13th International Scientific and Technical Conference on Computer Sciences and Information Technologies (CSIT), pp. 347-351.
Lviv (2018). doi: 10.1109/STC-CSIT.2018.8526729 14. Oliinyk A., Fedorchenko I., Stepanenko A., Rud M., Goncharenko D.: Evolutionary method for solving the traveling salesman problem. Problems of Infocommunications. Science and Technology : 5th International Scientific-Practical Conference PICST2018, Kharkiv, 9–12 October 2018, Kharkiv, Kharkiv National University of Radioelectronics, pp. 331339 (2018). doi: 10.1109/INFOCOMMST.2018.8632033. 15. Tsoy, Y.R.: Evolutionary Algorithms Design: State of the Art and Future Perspectives.
Proceedings of IEEE East-West Design and Test Workshop (EWDTW'06), Sochi, Russia, pp. 375-379 (2006). 16. Gruau, F.: Genetic synthesis of Boolean neural networks with a cell rewriting developmental process. In Proceedings of the International Workshop on Combination of Genetic Algorithms and Neural Networks (COGANN-92). Los Alamos, CA: IEEE Computer Society Press, pp. 55-74 (1992). 17. Moriarty, D., David, R., Miikkulainen, R.: Hierarchical evolution of neural networks. Evolutionary Computation Proceedings, pp. 428–433 (1998). doi: 10.1109/ICEC.1998.699793. 18. Greer, B., Hakonen, H., Lahdelma, R., Miikkulainen, R.: Numerical optimization with neuroevolution. Evolutionary Computation CEC '02(1), pp. 396–401 (2002). doi: 10.1109/CEC.2002.1006267. 19. Enforced Subpopulations (ESP) neuroevolution algorithm for balancing inverted double pendulum, http://blog.otoro.net/2015/03/10/esp-algorithm-for-double-pendulum. 20. Stanley, K.O., Miikkulainen, R.: Evolving Neural Networks through Augmenting Topologies. The MIT Press Journals, vol. 10, num. 2, pp. 99-127 (2002). 21. Whiteson, S., Whiteson, D.: Stochastic optimization for collision selection in high energy physics. Proceedings of the 19th national conference on Innovative applications of artificial intelligence, IAAI'07, vol. 2, pp. 1819-1825 (2007). 22. Hochreiter, S., Schmidhuber, J.: Long Short–Term Memory. Neural Computation, vol. 9.
issue 8, pp. 1735–1780 (1997). 23. Tsoy Yu.R.: Development of a genetic algorithm for setting up an artificial neural network [Razrabotka geneticheskogo algoritma nastroyki iskusstvennoy neyronnoy seti]. Tomskiy politehnicheskiy universitet (2004). 24. Papadimitriou, F.: Mathematical modelling of land use and landscape complexity with ultrametric topology. Journal of Land Use Science vol. 8(2), pp. 1-21 (2011). 25. Papadimitriou, F.: Artificial Intelligence in modelling the complexity of Mediterranean landscape transformations. Computers and Electronics in Agriculture, pp. 87-96 (2012). 26. Leoshchenko, S., Oliinyk, A., Subbotin, S., Gorobii, N., Zaiko, T.: Synthesis of artificial neural networks using a modified genetic algorithm. Proceedings of the 1st International Workshop on Informatics & Data-Driven Medicine (IDDM 2018), pp. 1-13 (2018). dblp key: conf/iddm/PerovaBSKR18 27. Yarymbash, D., Yarymbash, S., Kotsur, M., Divchuk, T.: Analysis of inrush currents of the unloaded transformer using the circuitfield modelling methods. Eastern-European Journal of Enterprise Technologies, vol. 3, № 5 (93), pp. 6-11 (2018). doi: 10.15587/17294061.2018.134248 28. Yarymbash, D., Yarymbash, S., Kotsur, M., Divchuk, T.: Enhancing the effectiveness of calculation of parameters for short circuit of threephase transformers using field simulation