-This paper illustrates a process of finding optimum structure of neural network which will be used to solve problem of Iris data set. This work presents dependencies between number of layers, neurons and efficiency of network providing the best configuration for given data set.
Computers can help on automated models implemented to
co-work with other devices in service. We can find various
application of intelligent systems in medicine, technology,
transport, etc. We expect from computer to control possible dangers
and advise the best options, while assisting humans [
Mainly computational intelligence is assisting in data
processing to discover knowledge from input data, there are
mathematical models implemented to find incomplete
information and work with this issues [
Neural network is a biologically inspired model of
mathematical computations which structure is based on
architecture of human brain. In the simplest approach
brain consists of approximately 8:6 1010[
Why one should even bother applying neural network into any field of computations? Mostly due to the fact that artificial neural networks are able to generalize obtained knowledge which means that after proper training such network should be able to predict correctly value of given example not included in training data. Second advantage of these models is robustness for random fluctuations or lack of part of values in data set. Generally neural networks are used to solve problems that seems to be incomputable or too complicated to solve by classical algorithms.
One can observe that artificial neuron is really simplified versions of human neuron, however still keeps three most important features of neuron: -taking input from other neurons (dendrites) -computing and processing impulses taken as input -sending on computed impulse to further neurons(axon)
One cycle of computations in neurons may be described as follow: values are being multiplied by appropriate weights and summed together in a cell of neuron after which sum is taken as argument to activation function which value is set to be impulse send as the output of neuron. Equation describing this process is presented below:
n y = f ( P w(xi)j xi), i=1 (1) where w(xi)j xi f
weight between i and j neuron in consecutive layers, Figure 2. Graph of different activation functions tested in this paper value of neuron i, activation function In (1) it is assumed that previous layers consists of n neurons.
Activation functions is an abstract indicator that says about action taken in neuron. In simplest case once information is important it outputs 1 otherwise neuron outputs 0. This model is called a binary step function, despite its simplicity it is able to solve a few problems, however it is insufficient with more complex problems due to the lack of desired features detailed in following list: - Finite range results in stability while gradient - based method are used for learning - Continuous differentiable is necessary for every gradient based method, because of that feature binary step function cannot be used in models with gradient based learning algorithms. - Identity near origin fasten the learning process once initial weights are small numbers.
In this paper results will be obtained using only four different activation functions:
where
f (x) = 1+e1 x determines steepness of function near origin, 2 f (x) = tanh (x) = 1+e 2x 1
Modification proposed by Yann LeCun [
Feed forward neural network is constructed by neurons stacked in a rows called layers that are fully connected with previous and subsequent layer. First and last layer are respectively called input and output layer, each layer between these two is called a hidden layer. Role of neurons in input layer is to store the initial data and send it further. Flow of information takes place from left to right, so once initial data is provided the result will appear at the output layer which ends one full cycle of computations for network. (2) (3) (4) (5) w(i)j i According to (1) output of neural network is just a sum of random weights multiplied by initial values, the only parameter that might be modified in neural network are weights so learning algorithm is just a process of changing weights in such way that output is exactly the same as presented in data set. In this paper backpropagation algorithm will be discussed.
To train the network it is necessary to provide a data set that consists of vector of input signals (x1; x2; :::; xn) and corresponding desired output z. Thanks to this fact it is possible to compute the difference between output signal y and desired value z. Let = z y. Next step is to propagate error through every neuron according to equation: n j = P w(j)i i, i=1 (6) weight between i and j neuron in consecutive layer, value of error on neuron i in consecutive layer, weight between i and j neuron in consecutive layers, value of neuron j, activation function on neuron i value of error on neuron i in consecutive layer, coefficient that affects speed of learning.
Once error is computed for each neuron following step is to actualize weights using equation below: number of iterations, number of neurons in network, Learning is an iterative process that takes place until output layer is smaller than desired precision.
III. CONVERGENCE OF GRADIENT DESCENT ALGORITHM
WITH RESPECT TO
While analyzing equation (7) it is obvious that has a direct impact on pace of learning of neural network since derivative determines direction where error function descends. It is crucial to select balanced value of . Too small value may result in slow learning process while value bigger than necessary will end with divergence of algorithm. Four different scenarios of values are presented in graphical interpretation. Every test in this paper was carried out with = 0:02 as it appears to be most suitable value of this parameter.
IV. IRIS DATA SET
Iris data set is a classic and one of the best known sets
for pattern recognition described in [
In order to create training data set, 35 samples were chosen from each class, so that 30% of whole set is left for testing accuracy of trained model. The aim for neural network is to classify Iris, described by these four dimensions, into one of three given classes.
V. TESTS
For further tests is equal to 0.02, it will be pointed out once will change. In every case desired error is set to be smaller than 10 5. Maximum epochs are set to 20000. In order to compare obtained results it is necessary to come up with general formula determining number of weights in single network: n q = 4 3 Q i, i=1 where i neurons on each hidden layer, n number of hidden layers, (8)
As batch learning approach is used in this case every iteration is equal to updating every weight in whole neural network structure. Since then it seems natural to come up with equation describing numbers of updates done throughout whole process of learning each network.
Q = q r (9) for
Q from equation (9) will be used in next section to compare efficiency of different network structures.
It is clearly visible in table below that in case network consists of only input and output layers (0 hidden layers) it fails to classify properly each of three types, usually being able to classify correctly two of them while having issues with third one. This is caused by the fact that one class is linearly separable from the other two, however the latter are not linearly separable from each other.
One can observe that Hyperbolic Tangent function is most
uniform function while being most effective taking into
consideration number of iterations and number of neurons needed
to obtain desired precision. Similar behaviour can be viewed
with Arctangent function, however it fails to work for networks
that were properly computed with Hyperbolic Tan.
On the other side Modified Tangent performed below
expectations, however it is important to remember that all tests were
carried out for fixed . Since steepness of this function is
modified and value of should be set with respect to derivatives
[
Three graphs presented in this paper represents Error function over consecutive iterations. Figure 6 presents nearly perfect plot that is smooth and monotonic at most of a time. On contrary to the Figure 5 which presents heavy oscillations of Error function caused by wrong coefficient. Exactly the same situations appears to be on Figure 7 at around 400 iteration, luckily it is a situation when > opt and after a few iterations of oscillations it finally converges. Values presented in table show that network failed to obtain desired precision in 20000 iterations.
to achieve properly working model. As most important and discussed in this paper are:
Activation function - as turned out, it has a huge impact on capabilities of network, not only these provided by differentiability but also can significantly accelerate process of learning, Learning coefficient - highly depends on activation function, is a crucial factor in converging process of gradient - based methods of learning, Number of hidden layers and neurons - in simple problems like that one discussed in this paper 2 hidden layers are enough, Number of iterations and desired precision - again, strongly depends on problem one tries to solve, there is no general rule that describes how to set these values, however it is important to set upper limit of iterations in order to stop the computations at some point once learning algorithm is unable to achieve desired precision,
Among various architectures neural networks are one of most efficient processors of information about controlled objects, moreover these architectures can cooperate with information presented in a form of image, voice sample, network statistics, etc.
Each implemented solution needs adjusted architecture to exactly fit the model of decision support. Therefore research on performance of various types can give valuable information about performance, what can improve implementations of neural networks.