In the article hyper basic function neural network (HBFN) in convolutional neural networks instead of fully connected perceptron layers, which solve the problem of classification is proposed. HBFNs are generalization of traditional radial-basic function networks, which like multilayer perceptrons are also universal approximators. The peculiarity of hyper basic function neural networks is that their receptive fields are hyperellipsoids with arbitrary orientation of the axes in the feature space. It is assumed that, along with the synaptic weights, receptive field parameters are adjusted: their centers and matrices of the axis orientation. This approach allows to reduce the total number of synaptic weights, and accordingly the number of R-neurons (activation functions) of the network. Using multidimensional V. Epanechnikov's functions with a hyperellipsoidal receptive field that can be adjusted during the learning process as activation functions permit to accelerate the learning process of investigated network. It should be noted that applying kernel functions in convolutional neural networks permits to avoid the undesirable effect of "exploiding gradient", which often occurs in fully connected layers. The computational experiment results on standard data sets confirm the effectiveness of the proposed approach and, first of all, the high learning speed.
For today the deep neural networks (DNN) are widely used for solving large class of tasks that are
connected with Data Mining and especially Big Data Mining, first of all, pattern
recognitionclassification, forecasting, identification-emulation, natural language processing, etc. [
The first part is – autoencoder that is used for input signal-image compression, that is commonly
given in the matrix form. As a result of the input 2D-signal processing by the convolutional and
pooling layers in the output of the last pooling layer at this part of the system n-dimensional vector
signal is formed. In fact, if the input signal-image has form of the ( 1 × 2)-matrix, then usually ≪
1 2, i. e. this part of the CNN realizes compression task of the input matrix signal. The second part
of the CNN in general case solves classification task usually, it is standard multilayer perceptron
(MLP) whose nodes are F. Rosenblatt’s elementary perceptrons, with piecewise activation functions.
Here it is necessary to note, that in practical implementation of the shallow neural networks (SNN)
so-called
sigmoidal squashing functions [
At the same time, learning of the standard MLP with squashing activation function is met with
extremely unpleasant effect of the vanishing gradient, that significantly complicates the process of
synaptic weights adjustment based on the error back propagation method. That is why, in the DNN
instead of classic sigmoids or hyperbolic tangents linear piecewise functions are used, that provide
desired approximation quality [
The simplest way to overcome this problem is to implement instead of the standard MLP in the
CNN, so-called, radial-basis function neural networks (RBFN), that include only one layer of the
tuned synaptic weights and whose output signals are linearly depended on these weights, that allows
to use for learning fairly simple optimization algorithms [
However, the RBFN implementation instead of the MLP also does not solves all arising problems
because, firstly, conventional radial-basis function neural networks suffer from the so-called “curse of
dimensionality”, when amount of the activation function in the R-neurons are exponentially increased
dependently from the input space dimension, and, secondly, arises the problem of hyperspherical
receptive fields centers distribution of the kernel activation functions usually as multidimensional
Gaussians or Cauchyans type. The problem of the centers distribution usually is solved by using of
one or another clustering algorithms (in [
An alternative to traditional RBFN, in our opinion, can be so-called hyper basis neural networks
(HBFN) [
The prototype of hyper basis function network (HBFN) is traditional radial-basis function network
[
To the inputs of the hyper basis function neural network from the last pooling layer of the convolutional network the vector signals sequence ( ) = ( 1( ), 2( ), … , ( ), … , ( )) ∈ is fed to processing, where = 1,2, … , − is an observations number or discrete time of the training set = { (1), (2), … , ( ), … , ( )} ⊂ . This signal is passed on the R-neurons inputs, formed by kernel activation functions ( ( )) = where − the receptive field center of the function (∙), Σ−1 − positive ( receptive field orientation, 0( ( )) = 1. Usually, it can be multidimensional Gaussian × ) −matrix of the Cauchyan and other bell-shaped constructions.
( ( )) = −‖ ( ) − ‖Σ−1 , 2 −1 , (1) (2)
Visualization of the Gaussian with different width parameters is shown in the Figure 2. that in fact are essentially ordinary adders with adjusteable synaptic weights at the inputs. Thus, at the outputs of the adders the signals are appeared in the form ( ) = 0 + ∑ℎ=1 ( ( )) = ∑ℎ=0 ( ( )) = ( ( )) (3) where ( ( )) = (1, ( ( ))) ,
( ) = 1( ( )), … , ( ) , … , ℎ( ( )) , = network. 1,2, … , , ℎ > − determines the amount of the hyper basis functions, that are included in the 1,2, … ,
are additionally appear.
In the case when the quadratic error is used as a learning criterion, the signals (3) are the outputs of the all network, if the cross-entropy criterion and reference signal one hot coding is used, at the outputs of the system the every observation membership level for the all possible classes = Next it is easy to write the gradient procedure of the RBFN’s synaptic weights tuning: ( + 1) = ( ) + ( + 1) ( + 1) − ( ) ( + 1), ( ), Σ −1( ) × × ‖ ( + 1) − ( )‖2−1( ) Σ (here
( ) = 0( ), 1( ), … , ℎ( )
synaptic weights), kernel functions centers [
( + 1) = ( ) − ( + 1) ( + 1) − ( + 1) ( + 1), ( ), Σ −1( ) × × ′ ‖ ( + 1) − ( )‖2−1( ) Σ−1( ) ( + 1) − ( ) ,
Σ
receptive hyperelipsoids orientation [
The learning algorithms (4)-(6) are enough simple form the computational point of view, but they are not protected from the possible effect of the “exploding gradient” in the case when the derivatives × ‖ ( + 1) − ( + 1)‖2Σ−1 ( ( + 1) − ( + 1))( ( + 1) − ( + 1)) (6) of the function type (1), (2) acquire small values.
In the [
In this paper the modified V. Epanechnikov’s function with hyperellipsoidal receptive field with arbitrary orientation is proposed to use instead of (7): whose derivatives acquire zero value only at one point, where ( + 1) = and automatically does not suffer from the “exploding gradient” effect. In this situation the gradient learning procedure (4)(6) takes very simple form: ( ) = 1 − ‖ ( + 1) − ‖Σ−1
2 ( ) = 1 − ‖ ( )− ‖2
2 ( + 1) = ( ) + ( + 1) × × ( + 1) − ( ) ( + 1), ( ), Σ −1( )
× × 1 − ‖ ( + 1) − ( )‖2−1( ) , Σ ( + 1) = ( ) − с( + 1) × × ( + 1) − ( + 1) ( + 1), ( ), Σ −1( )
× × ( + 1)Σ−1( ) ( + 1) − ( ) ,
Σ−1( + 1) = Σ−1( ) − с( + 1) × × ( + 1) − ( + 1) ( + 1), ( + 1), Σ−1( ) × (11) × ( + 1) ( + 1) − ( + 1) ( ( + 1) − ( + 1)) that essentially simplifies its numerical implementation.
For numerical analysis of the approach under consideration, the convolutional neural network VGG-16 in which the fully connected layers of the multilayer perceptron were replaced by a hyper basic function neural network with a different number of R-neurons and, accordingly, tuned synaptic weights was used as a prototype. Architecture of the proposed convolutional hyper basis function network is shown in the Figure 3.
For the program realization of the proposed convolutional hyper basis function network model
programming environment "Python 3.8" and “Keras” library were used [
For solving binary classification task data set “Dogs&Cats” from “Kaggle” was used. This data set consists of the 25000 colorful dogs and cats images. The output set was divided into tree subsets: the training data set – 17500 images, test data set – 3750 images and validation one – 3750 images.
For the multiclass classification data set “CIFAR-10”, that consists of the 60000 colorful images 32 × 32 from different classes: planes, cars, birds, cats, dogs and others with 6000 images in every class. At the same time training subset consists of the 45000 samples, test subset – 5000 samples and validation one 10000 samples.
In the learning process for solving binary classification task designed hyper basis function neural networks were tuned by five epochs. The results of this process are shown in the Figure 4. The results of the multiclass classification are shown in the Figure 5 and for network learning in this situation 50 epoch were used.
To improve classification accuracy the learning rate parameters were significantly reduced and assumed as a constant value at the level ( ) = 10−6 for all tuned variables. The results of this learning process are demonstrated in the Figure 6 and Figure 7. Summarized numerical results of HBFN training are shown in the Table 1.
The quality of the classification results significantly depends on the R-neurons number in the network. In the Table 2 the binary classification results are demonstrated, at the same time in the second column the synaptic weights tuning results are shown and in the third column – the tuning results of the synaptic weights and receptive fields are presented.
The training results of convolutional neural network with hyper basis layer is shown in the Figure 8. Average time for learning model of the proposed convolutional hyper basis neural network was one hour and 9 minutes.
Final comparison results of binary and multiclass classification are shown in the Table 4.
Here is interesting to note that if the less is number of R-neurons amount in the HRFN then more important is tuning of the receptive fields parameters, then more is speed – then more is velocity of process tuning.
The convolutional hyper basis function neural network was proposed. This CNN differs from another because instead of the fully connected perceptron layers that solve classification task, it contains in its architecture HBFN that is generalization of conventional RBFN but besides synaptic weights it can tune too parameters of the receptive fields that are in common case hyperellipsoids with arbitrary axes orientation in feature space. This approach permits automatically to solve the problem of activations kernel functions centers distribution, to reduce their amount i. e. to increase the learning speed that is very important essentially for deep neural network. The computational experiments confirm effectiveness of the approach under consideration. 6. References