Adaptive learning of evolving hyper basis function neural network

Abstract

In the article architecture and learning method of the artificial evolving hyper basis neural network are proposed. The neural network under consideration tunes not only its synaptic weights, but automatically determines neurons number, coordinates of the kernel activation function centers and parameters of the receptive fields in online mode providing high speed data processing.

Keywords

• 1 Artificial neural networks
• adaptive learning
• hyper basis function neural network
• selforganizing T
• Kohonen's map
• V
• Epanechnikov's activation kernel function

1. Introduction

To date, artificial neural networks (ANNs) are widely used to solve various problems of Data Mining and first of all for intelligent control, identification, pattern recognition, classification, clustering, forecasting, emulation in conditions of uncertainty and significant nonlinearity. If data should be processed in a sequential online mode, a convergence rate of a learning process comes to the forefront, that significantly limits the ANNs class suitable for work under these conditions. ANNs, which use kernel activation functions (radial-basis, bell-shaped, potential), are very effective from the speed optimization point of view in the learning process.

Radial-basis function neural networks (RBFN) are widely used because their output signal depends linearly on synaptic weights. The main idea of this ANNs is connected with potential function method [1], Parsen’s estimations [2], kernel [3] and nonparametric [4] regressions. Universal approximation properties and ability to process data sequentially in online mode are its main benefits. However, the RBFN is exposed to the so-called “curse of dimensionality” which means that when the input space dimensionality increases, there’s an exponential growth of the adjustable parameters’ (weights’) amount. To overcome this problem for solving practical tasks possible by using hyper basis function neural network (HBFN) [5], with have more advantages comparatively to traditional RBFN.

2. Radial basis and hyper basis function neural networks

In Figure 1 standard architecture of the radial-basis function network is shown whose hidden layer implements some nonlinear transformation of the input space R! into higher dimension (h > n) hidden space R" and its output layer is formed by adaptive linear associators that form the network response by performing a nonlinear transformation of the form

" " y(#(k) = w$ + - w%φ%/x(k)1 = - w%φ%/x(k)1 = w( φ2/x(k)1 %&' %&$

( where x(k) = /x'(k), x) (k), … , x!(k)1 , φ%/x(k)1 = φ%(‖x(k) − c%‖, σ%) − radial-basis function, depended of distance ‖x(k) − c%‖ between input vector x(k) and activation function centers c% and ( width parameter σ%, k − is current discrete time, φ2/x(k)1 = 91, φ( /x(k)1; , φ/x(k)1 = 9φ'/x(k)1; , ( φ) /x(k)1,…, φ"/x(k)1; .

The most commonly standard Gaussian function as activation function in the radial-basis ANNs is used in the form

%/x(k)1 = exp ?− ‖+(-1)/!" 0!‖"@ , l = 1,2, . . . , h ( 1 ) where centers c% and wight σ% parameters are determined beforehand and do not tuned during learning process. The learning process itself is connected with adjusting of the synaptic weights vector w = (w$, w', … , w")( , for that different modifications of least-squares method or traditional gradient procedures are usually used.

Using the multidimensional construction instead of the Gaussian ( 1 ) it is possible to improve the approximating properties of the network %/x(k)1 = exp 9−(x(k) − c%)( Σ%/'(x(k) − c%); = exp 9−‖x(k) − c%‖)2#$; ( 2 ) ! where Σ3/' covariance matrix that determines shape, size and receptive field orientation of − ℎ kernel activation function. This is the main difference between hyper basis function network and traditional RBFN.

If Σ3 = 3) (here = ( × ) − identity matrix) the receptive field is a hypersphere with a center 3 and radius 3; if Σ3 = (') , )) , … , 4) ) − that is hyperelipsoid with axes coincide with input space axes and have length 5, at 67 axe, and finally, if Σ3 is arbitrary positive definite matrix Σ3 = 38Λ33 diagonal matrix of the eigenvectors Λ3 determines receptive field size and rotation orthogonal matrix 3 − its orientation.

Speaking about hyper basis function ANNs learning it should be noted that here not only synaptic weights vector can be adjusted but and centers 3, and matrixes Σ3. So, introducing into consideration the transformation implemented by the neural network in the form

7 (9() = - 33 9‖() − 3‖)2#$; ,

% the learning algorithm could be written [6]: ⎧ ⎪ ⎪

3( + 1) = 3() + >( + 1)( + 1)3 9‖( + 1) − 3()‖)2%#$(=); , 3( + 1) = 3 () − ?( + 1)( + 1)3( + 1)3@ 9‖( + 1) − 3‖)2%#$(=); ×

× Σ3/'()(( + 1) − 3()), ⎨ ⎪Σ3/'( + 1) = Σ3/'() + 2( + 1)( + 1)3( + 1)3@ 9‖( + 1) − 3( + 1)‖)2%#$(=); × ⎪ 8 ⎩ × /( + 1) − 3 ( + 1)1/( + 1) − 3( + 1)1 ( 4 ) where >( + 1), ?( + 1), 2( + 1) − learning rate parameters for the corresponding variables.

Using as activation functions Gaussians ( 1 ) and ( 2 ) leads to learning procedure ( 4 ) becomes too cumbersome from computational point of view that naturally slows down the learning speed. In this regards we propose to introduce into consideration multidimensional modification of the V. Epanechnikov’s function [7] in the form 3 9‖( + 1) − 3()‖)2#$ ; = 1 − ‖( + 1) − 3()‖)2#$,

% % whose derivations have the form ⎧ ⎪ 3 ∇?%() = 2()3Σ3/'(() − 3), = −()3 91 − ‖() − 3‖)2#$; ,

% ⎪⎨f() ⎩ Σ3/' g = −()(() − 3)(() − 3)8, ( 3 ) ( 5 ) ( 6 ) ⎨c ⎪ ;2%#$ ⎩ Relations ( 5 ) avoid to rewrite the system ( 3 ) in the form () ;A%B‖C(=)/?%‖"&#$D

% and learning algorithm – ⎧∇?%3 9‖() − 3‖)2#$ ; = −2Σ3/'(() − 3), ⎪ %

d = (() − 3)(() − 3)8. learning criterion 3&$ (here () − external reference signal) and derivations by all tuned parameters ;<(=) ;>% = −()3 9‖() − 3‖)2%#$; , ⎧ ⎪ ∇?%() = 2()33@ 9‖() − 3 ‖)2#$ ; Σ3/'(() − 3),

% ⎨ ⎪⎩`;;<2(%#=$)a = −()33@ 9‖() − 3‖)2%#$; (() − 3)(() − 3)8, ⎧ 3( + 1) = 3() + >( + 1)( + 1) 91 − ‖( + 1) − 3()‖)2%#$(=); , ⎪3( + 1) = 3() − ?( + 1)( + 1)3( + 1)Σ3/'()(( + 1) − 3()), ⎨ Σ3/'( + 1) = Σ3/'() + 2( + 1)( + 1)3( + 1) × ⎪ 8 ⎩ × /( + 1) − 3( + 1)1/( + 1) − 3( + 1)1 .

It is easy to see that procedure ( 6 ) from the computational point of view is simpler that algorithm ( 4 ). 3. Hybrid neural network and evolution of its architecture

The question how to choose number of neurons in the network ℎ and initial center locations 3 it is very important. The easiest way to overcome this problem is to use Subtractive Clustering algorithm [8], that is effective enough for processing data in the batch mode. But this algorithm requires to select huge amount of the free parameters. If solving task is connected with processing of non-stationary data, then it is necessary to re-initialize the network from time to time.

Dynamic Decay Adjustment (DDA) also is one of the possible methods to tune neural network with kernel activation functions [9]. This method is belonging to constructive learning algorithms and works fast enough. But, in the mode of online processing of the non-stationary signals this method becomes noneffective.

Resource Allocation Network [10] uses hybrid learning based on both optimization and memory learning (the principle "Neurons in data points"), using of competition elements. At the same time, in the learning process using gradient procedures, both synaptic weights and the parameters of the neuron’s centers closest to the received observation are adjusted. It can be noted that the standard Epanechnikov’s functions are used instead of traditional Gaussians as activation functions in the network. The disadvantage of Resource Allocation Network is its high computational complexity.

In this regard, it is necessary to develope artificial evolving hyper basis function neural network, that tunes not only its all parameters but and automatically determines number of neurons in online mode with high speed of data processing.

In the figure 2 architecture of a hybrid evolving artificial neural network that is based on a hyperbasis neural network with a variable number of neurons and T. Kohonen's self-organizing map (SOM) [11], that controls their number and adjusts centers location in the self-learning mode is demonstrated.

The functioning process of this system is as follows. When the first observation () is fed to the neural network input, where the first neuron is formed according to the principle of "Neurons at data points", i. e. almost instantly. In the next incoming of the data they firstly fed to SOM, where they are compared with the already existing centroids and then if no coincidences are found, a new center of the kernel function is formed and, accordingly, a new neuron in HBFN.

According to the approach under consideration next method of the controlled numbers of neurons activation function is put into consideration:

Stage 11: to encode all the values of the input variables into the interval −1 ≤ 5 ≤ 1 and set the receptive field radius of the neighborhood function in the interval ≤ 0,33;

Stage 21: observation ( 1 ) after this is fed set ' = ( 1 ); Stage 31: observation ( 2 ) is set: - if ‖( 2 ) − '‖ ≤ , that '( 1 ) is corrected by the rule ' + ( 2 )

2 - if < ‖( 2 ) − '( 1 )‖ ≤ 2, '( 1 ) is corrected according to the self-learning of the selforganizing Kohonen’s map by principle of "Winner takes more" (WTM) [11] 3( 2 ) = ; with neighborhood function '( 2 ) = '( 1 ) + ( 2 )'( 2 )(( 2 ) − '( 1 )) 3( 2 ) = c0,1 − p ‖( 2 ) − '( 1 )‖ ) q d if ‖( + 1) − 3()‖ ≤ then 2 (Epanechnikov’s function with receptive field with radius equal 2)

- if ‖( 2 ) − '( 1 )‖ > 2, new kernel activation function is formed with center ) ( 2 ) = ( 2 ).

This completes the first iteration of the activation functions forming of the hyper basis neural network. Let us for the -th moment of time ≤ ℎ activation functions 3/()1 are formed with centers 3() and ( + 1)-th observation is fed to processing. Next forming of the activation functions is performed as follows:

Stage 1k+1: to determine neuron-winner for each distance ‖( + 1) − 3()‖ minimal among all = 1,2, … , ;

Stage 2k+1: if < ‖( + 1) − 3()‖ ≤ 2, then 3( + 1) = 3() + ( + 1)3( + 1)(( + 1) − 3()); 3( + 1) = c0,1 − p ‖( + 1) − 3()‖ )

q d ; 2 - if ‖( + 1) − 3()‖ > 2 then kernel activation function is formed with center EF'( + 1) = ( + 1);

- if in the process of activation functions formation arises a situation ‖( + 1) − 3()‖ > 2 and = ℎ, then it is necessary to increase the receptive field radius and return to stage2k+1 with an increased radius of the function 3( + 1).

As can be seen, this procedure is a hybrid of N. Kasabov's evolving algorithm [12] and T. Kohonen's self-organizing map. However, the proposed neural network is designed not only to solve clustering problems, but to control the number of neurons in the hyper basis neural network.

3( + 1) =

; 3() + ( + 1) 2

4. Evolving cascaded hyper basic function neural network

If we have a short learning sample then it is possible to overcome the problem of overfitting by dividing an initial task in some way to subtasks of lower dimensionality and by grouping the obtained solutions to get a required result. From a computational point of view, the most convenient method in this case is the Group Method of Data Handling (GMDH) [14-17] that have demonstrated its efficiency while solving a number of practical tasks. In [17], a multilayer GMDH-neural network is considered. It contains two-input N-adalines as nodes and each node’s output is a quadratic function of an input signal. Each neuron’s synaptic weights are defined in a batch mode using the conventional least squares method. One way use some more hidden layers to provide a necessary approximation quality. That is why an online learning procedure becomes impossible. In this connection it’s expedient to introduce a simplified architecture which is based on simple nodes and can be tuned under conditions of a short learning sample.

Let’s introduce an architecture of the compartmental R-neuron that is shown in the Figure 4. Generally speaking, it’s a simplified architecture of the conventional RBFN with two inputs 5 and G, , = 1,2, … , , where is a dimensionality of initial input space.

The compartmental R-neuron contains activation functions 75G/5G, 75G, Σ75G1, + 1 synaptic weights that are joined in vector 35G = /3$ , 35'G, … , 35EG1, two-dimensional centers’ vectors 75G = 5G /75, 7G18, (2 × 2) −matrices of receptive fields’ activation functions Σ5G, a two-dimensional input 7

Multidimensional Epanechnikov kernels are used as activation functions 7 /5G, 75G, Σ75G1 5G 75G /5G, 75G, Σ75G1 = 1 − u5G − 75GuH)2(')I#$ Introducing a ( + 1) × 1-vector

8 …,E5G/5G(), E5G, ΣE 1; 5G

and a learning criterion which have a bell-shape form determined by a positive definite matrix of the receptive field Σ5G. An 7 advantage of the activation function ( 7 ) comparing to conventional ones is linearity of its derivatives with respect to all the parameters that makes it possible to adjust synaptic weights as well as both centers and receptive fields (which is very important when we have a very short learning sample). At the same time, the compartmental R-neuron implements a transformation in the form

E E (3 = 35$G + - 357G75G /5G, 75G, Σ75G1 = 35$G + - 357G p1 − u5G − 75GuH)2(')I#$q.

7&' 7&'

It should be noticed that if we have a two-dimensional case it is easy to locate centers at the regular lattice’s nodes and to define receptive fields as circles.

of activation functions 5G = 91, 35G95G(), 35G, ΣG5G; , vector 5G = /5, G18, output (3; = 1,2, … , ; = 1,2, … , is a number of observations in a processing sample or an index of the current discrete time. ( 10 )

The algorithms ( 9 ) and ( 10 ) are effective only in those cases when a required solution is stationary which means that optimal values of the synaptic weights aren’t variable in time. But most of real-word practical tasks are characterized by an opposite situation. A high-performance adaptive learning 3J = ∑=J&'/() − (3()1 = ∑=J&' 3) () = ∑=J&' p() − /35G185G()q ) it's easy to obtain a required solution with the help of the traditional least squares method in the form J F J

8 35G = w- 5G() 95G(); x

- 5G()() =&' =&' where (∙)F is a symbol of the Moore-Penrose inversion. If data are fed sequentially in an online mode, recurrent from of the expression ( 8 ) can be used in the form ⎧ ⎪ ⎪ ⎨ ⎪ ⎪5G() = 5G( − 1) − ⎩ 35G() = 35G( − 1) + 8 5G( − 1) p() − 935G( − 1); 5G()q

8 1 + /5G()1 5G( − 1)5G()

8 5G( − 1)5G() 95G(); 5G ( − 1)

8 1 + /5G()1 5G( − 1)5G()

5G(), , 5G(0) = , ≫ 0.

( 7 ) ( 8 ) ( 9 ) algorithm with both tracking and filtering properties [18] can be used for adaptive identification of nonstationary objects and non-stationary time series prediction in the form ⎧⎪35G() = 35G ( − 1) + >/'() p() − 935G( − 1); 8 5G()q 5G() = ⎨ = 35G( − 1) + >/'()3()5G(), ⎪ ) ⎩ >() = >( − 1) + u/()1u , 0 ≤ ≤ 1.

To tune centers and covariance matrices, one could use a procedure ⎧ ⎪ ⎪ ⎪ ⎪ ⎪ ⎨ 35G() = 35G ( − 1) + >()3()5G(),

) >/'() = >() = >( − 1) + u/()1u , /' 75G() = 75G( − 1) + ?()3()35G () 9Σ75G( − 1); 95G() − 75G( − 1); = = 75G( − 1) + ?()3()7(), ?/'() = ?() = ?( − 1) + ‖7()‖) ,

/' /' ⎪9Σ75G(); = 9Σ75G( − 1); ⎪ ⎪ /' ⎪ = 9Σ75G( − 1); − 2()3()7(), ⎪ ⎩ 2/'() = 2() = 2( − 1) + 97()78(); .

An architecture in Figure 4 combines the GMDH ideas and cascade neural networks. − 2()3()35G() 95G() − 75G(); 95G() − 75G(); ( 11 ) 8 =

The first hidden layer of the network is formed similarly to the first hidden layer of the GMDH neural network [17] and contains a number of neurons that equals to a number of combinations of in 2, that is 4) . A selection block SB executes a sorting procedure by accuracy, for example, the most accurate signal among all output signal (3['] is ('[']∗ in the sense of variations, then comes ()[']∗ and the worst one is (M['+"]∗. The SB outputs ('[']∗ and ()[']∗ are fed to the only neuron in the second cascade layer − [) ] that computes a signal [) ] which is later joined in the third cascade with the selection block’s output (N[']∗. A process of cascades’ increasing goes on unit the required accuracy is obtained. An overall neurons’ number of this network is defined by a value 24) − 1. The neural network can process information that is fed in real time by adjusting both its parameters and its architecture in time [19].

5. Experimental results

In first series of experiments, we used synthetics generated data set that describes Normal Gauss bivariate distribution. In this data set number of clusters was chosen arbitrary and all data were generated in the interval [−1; 1]. Result of processing synthetic data set by evolving hyper basis function neural network (EHBFNN) are shown at Figure 5. − 1 − where OP = J' ∑GP&' GP/GP − /P1/GP − /P18, j = 1,2, … , m − the inter-cluster distance matrix for clusters; /P = J' ∑GP&' GPGP − gravity data set center ; GP − number of observation belonged to the j-th clusters; >P = J ∑GP&' ∑=J&' G()/() − GP1/() − GP18 − intra-cluster ' distance matrix for clusters, = ! 1, () − ℎ ,

0, where where G − crisp membership function.

In situation when observations came to process sequentially in online mode is it necessary to organize calculation of the Calinski-Harabash index on the sliding window of dimensional ( = 1,2, … , ) and we could rewrite this index in the form

1 ) (, ) = − 1 ∑GP&' GP()uGP() − /P()u

1 ) − ∑GP&' G()u() − GP()u /P() = 1 = R&=/SF' ().

Visualizations of clustering results by proposed evolving hyper basis function neural network are presented at the Figure 7 (Figure 7a – data set “Wine”; Figure 7b – dada set “Ionosphere”) min 40.93 47.62 41.90 55.35 64.82 62.78 a) b) Figure 7: Visualizations of clustering results by proposed evolving hyper basis function neural network: a) data set “Wine”, b) dada set “Ionosphere”

6. Conclusions

The approach, which combines training of both synaptic weights and activation functions’ centers, and which is based on both supervised learning and self-learning, is proposed in this paper. The main advantage of the proposed approach is that it can be used in an online mode, when a training set is fed to a system’s input sequentially, and its volume is not fixed beforehand. The results can be used for solving a wide class of Dynamic Data Mining and Data Stream Mining problems.

7. References

References

[1] M.A. Aizerman , E.M. Braverman , L.I. Rozonoer , Method of potential functions in the theory of machine learning . Moscow: Nauka , 1970 .

[2] E. Parzen , “ On the estimation of a probability density function and the mode” , Ann. Math. Statist. 3 , ( 1962 ), 1065 - 1076 .

[3] S. Haykin , Neural Networks . A Comprehensive Foundation. Upper Saddle River , N.J.: Prentice Hall , Inc., ( 1999 ).

[4] T.V. Varyadchenko , V. Ya . Katkovnik, “ Nonparametric method of inversion of regression functions”, Stochastic control systems , Novosibirsk: Nauka, ( 1979 ), pp. 4 - 14 .

[5] Y. Zhon , T. Mu, Zh-H. Pang , Ch. Zheng, “ A survey on hyper basis function neural network”, System Science & Comtrol Engineering, 7 ( 1 ), ( 2019 ), 495 - 507 .

[6] Ye. Bodyanskiy , O. Tychenko , A . Deineko, “ An evolving radial basis neural network with adaptive learning of its parameters and architecture” , Automatic Control and Computer Science , 49 ( 5 ), 2015 , 255 - 260 .

[7] V. A. Epanechnikov , “ Nonparametric estimation of multivariate probability density” , Probability Theory and its Application , 14 ( 2 ), ( 1968 ), 156 - 161 .

[8] S. Chiu , “ Fuzzy model identification based on cluster estimation” , Journal of Intelligent & Fuzzy Systems , 2 ( 3 ), ( 1994 ).

[9] N. Karimi , S. Kazem , D. Ahmadian , H. Adibi , L. V. Ballestra , “ On a generalized Gaussian radial basis function: Analysis and applications”, Eng . Anal. with Boundary Elements . ( 2020 ), vol. 112 , 46 - 57 ,

[10] M. Dehghan , V. Mohammadi , “ The numerical solution of Fokker-Planck equation with radial basis functions (RBFs) based on the meshless technique of Kansa's approach and Galerkin method” , Eng. Anal. Boundary Elements , ( 2014 ), vol. 47 , 38 - 63 .

[11] T. Kohonen , Self-Organizing Maps . Berlin: Springer-Verlag. 1995 .

[12] N. Kasabov . Evolving Connectionist Systems . London: Springer-Verlag. 2003 .

[13] A. Frank , and A. Asuncion , UCI Machine Learning Repository, [http://archive.ics.uci.edu/ml]. Irvine, CA: University of California, School of Information and Computer Science, 2013 .

[14] A.G. Ivakhnenko , Self-learning systems for recognition and automatic control . Technika, Kyev , 1969 .

[15] A.G. Ivakhnenko, Long-term prediction and complex systems control . Technika, Kyev, 1975 .

[16] A.G. Ivakhnenko , H.R. Madala , Inductive Learning Algorithms for Complex Systems Modeling . CRC Press, London-Tokio, 1994 .

[17] Z. Zhao , Y. Lou , Y. Chen , H. Lin , R. Li , G . Yu, “ Prediction of interfacial interactions related with membrane fouling in a membrane bioreactor based on radial basis function artificial neural network (ANN)” , Bioresource Techno , ( 2019 ), vol. 282 , 262 - 268 .

[18] Ye . V. Bodyanskiy , O.K. Tyshchenko , A.O. Deineko , Evolving neuro-fuzzy systems with kernel activation function . Saarbruecken, Germany: LAP Lambert Academic Publishing. 2015 .

[19] R. Xu , D.C. Wunsch . Clustering, IEEE Press Series on Computational Intelligence. Hoboken, NJ: John Wiley & Sons, Inc., 2009 .