The work considers the vanishing gradient problem inherent in deep neural networks and limiting their potential due to the unstable learning process. An adaptive piecewise linear squashing activation function (APWLSAF) is proposed. This function, on the one hand, will ensure high accuracy of the deep network, and on the other hand, will make the network learning process stable due to the fact that the gradient of proposed function does not vanish and, therefore, will not be able to stop the adjusting network parameters.
The sheer number of existing adaptive activation functions suggests that in each task, researchers choose the type of function based on subjective ideas about how well it will provide the desired level of result. If the function was not chosen very well, then adjusting its parameters during network training will only slightly improve the situation, but not radically change it. For many tasks, especially those related to big data, processing online data streams, etc. it is not always possible to select the most appropriate activation functions type for individual layers of a deep network during a sequence of experiments, which ultimately makes the result quality dependent on the developer experience and intuition. Thus, it is necessary to construct an activation function that will allow a complete change in its form during network training, and will ensure high accuracy and speed of learning.
This work proposes a parametric adaptive piecewise linear activation function, which can change shape from sigmoidal to linear, and an algorithm for its training.
Modern commercial applications designed, for example, for classification, object recognition in video, and performance prediction, are based on use of shallow and deep feedforward networks. Their architectures can be considered as multilayer perceptrons, which have been known for a long time [15]. The nature of transformation performed by a neural network as a model depends on activation function of neurons in its composition.
Among the squashing activation functions used in traditional shallow and deep neural networks, and satisfy the conditions of Cybenko's approximation theorem [16], the hyperbolic tangent function has become the most widespread networks. type:
the derivatives of which take zero value when output of internal activation signal goes out of bounds of interval [-1, 1]. Therefore, Satlin is not used at all in gradient learning of neural
It is the vanishing gradient effect that has led to the fact that in DNN squashing activation functions are not used at all, and the most widespread are piece-wise linear functions of ReLU where ( ) - j-th SNN neuron output signal at the moment of discrete time k = 1, 2,…, n, (∙)nonlinear activation function of this neuron, 0 - bias term, n is the number of inputs to j-th neuron, - tuned synaptic weight on i-th input of j-th neuron, ( ) - input signal on i-th input ( + 1) × 1 - input vector, = 0, 1, . . . , , . . . , - ( + 1) × 1 - vector of adjustable synaptic weights, ( ) - a signal of internal activation of j-th neuron, - gain parameter, which determines the form of this activation function.
Derivative of this feature used in gradient tuning process of neuron looks like ʹ = from where it follows that when output signal approach the values ±1 the derivative goes to zero, that is, learning process stops due to the so-called "vanishing gradient” effect.
It is interesting to see that to Cybenko theorem conditions also corresponds the adaptivelinear function Satlin ( ) = ( ) = ( ), − 1 ≤ ( ) ≤ 1 −1, ( ) < −1
1, ( ) > 1 = ( ) = ( ) = tanh ( ) = ( ) = 0 + ( ) =
( ) =
=0
1 − −2 ( )
1 + −2 ( )
(
(
ℎ u j (k ) , au j (k ) , 0 < a < 1, otherwise u j (k ) ≥ 0 or PreLU: process.
( ) = PRe L U ( ) =
0, ( ) = = The advantage of these functions is simplicity of their derivative, which facilitates the learning Since these functions do not satisfy approximation theorem conditions, number of neurons and layers is significantly increased to ensure the required quality in DNN, which leads, first, to a significant reduction of learning rate, and secondly, it requires an increase training data number which are not always available when solving real practical problems.
Therefore, it is advisable to introduce an activation piece-wise linear function, which will have simple derivative, is close enough to squashing functions with their approximation properties, and does not suffer from the effect of "vanishing gradient".
As such a function is offered a formulation of Adaptive Piece-Wise Linear Squashing Activation Function (APWLSAF)
Its graph is shown in Figure 1. 2 ( ) = ⎧ ( ), 1 −
⎪ ⎩⎪
1 − ( ) , ( ) > 1 . ⎨ 1 + ( ) − 1, ( ) < −1
-1 ≤ ( ) ≤ 1 1 ) u (F 0 A S L W P A -1 -2 -3 0 u -2 -1 1 2 3 It is easy to see that when 0 = 0 = 0 APWLSAF is converted to Satlin, and when = = 0 we get an elementary linear function ALA (Adaptive Linear Associator). This activation function is characterized by the simplicity of its derivatives: and, if you set additional limitations ≥ , ≥ , it is protected from the effect of the "vanishing gradient" inherent in Satlin.
APWLSAF-based neural network approximation properties can be improved if we not only set up synaptic weights in the learning process, but also tune the parameters of activation functions and .
The process of adjusting each neuron in network is implemented by minimizing the accepted
criterion of learning ( ), most often quadratic, with the help of so-called δ-rule, which is
essentially a procedure of gradient optimization ( ) on tuned synaptic weights .
(
( ) Е ( ) е ( ) е ( ) е ( ) ( ) ( ) (here ( )-externallearningsignal),thentheprocedurefor optimizinglearningwilllooklike: 2 = 21е22( ) = = 2 ( ) − ( ) = ( − 1) − ( )е ( ) 2 2 е ( )
= where ( ) - the so-called δ-error; ( ) is a search step parameter that is selected from one For our case, the learning algorithm can be recorded in a sufficiently simpler form: ( − 1) + ( )е ( ) ( ), − 1 ≤ u ( ) ≤ 1
( ) = ( − 1) + ( )е ( ) ( ), u ( ) > 1 or ( − 1) + ( )е ( ) ( ), u ( ) < −1 ( − 1) + ( )е ( ) ( ), − 1 ≤ u ( ) ≤ 1 ( ) = ( − 1) + ( )е ( ) ( ),
u ( ) > 1 ( − 1) + ( )е ( ) ( ), u ( ) < −1
where ( ) = ( ), ( ) = ( ).
Widrow-Hoff algorithm [17, 18] in the form of
The learning process can be optimized by performance by using a modified
Kaczmarz(
+ ‖ ( )‖2 , − 1 ≤ u ( ) ≤ 1 ( − 1) + е ( ) ( )
+ ‖ ( )‖2 , u ( ) > 1 ⎪ ( − 1) + е ( ) ( ) ⎪
+ ‖ ( )‖2 , u ( ) < −1 where ≥ 0 is regulatory parameter that protects learning process from zeroing gradient effect.
It is possible to improve approximate properties of neuron with APWLSAF by adjusting not only synaptic weights , but also parameters of activation function and , while receiving the next vector ( ) on neuron inputs, setup ( ) and ( ) is first implemented and then the vector of synaptic weights ( ) is refined. That is, the procedure of error backpropagation at individual neurons level is actually implemented. Note, that
( ) = ( − 1) + ( )е ( ) ( ) − 1 = if и ( ) > 1, and if и ( ) < −1, then = ( − 1) + ( ) ( ) − ( − 1) ( ) ( − 1) ( ) − 1 , ( ) = ( − 1) + ( )е ( ) ( ) + 1 = = ( − 1) + ( ) ( ) − ( − 1) ( ) ( − 1) ( ) + 1 , ⎧ ( − 1) + е ( ) ( ) +‖ ( )‖2 ,
− 1 ≤ u ( ) ≤ 1 ⎪ ( ) = ( − 1) + е ( ) ( )2 , ( ) = ( ) ( ), u ( ) > 1.
⎨ + ( ) ⎩⎪ ( − 1) + е ( ) ( )2 , ( ) = ( ) ( ), ( ) < −1 + ( ) (16)
increasing classification accuracy is very important in problems of image recognition and processing [19 – 22]. An experimental study of the effectiveness of the APWLSAF was carried out to solve the problem of recognizing people's emotions from photographs. The task remains relevant for many applications, where result depends on quality of user interaction and taking into account his emotional status can influence the process [23].
In the experiment some photos from the Extended Cohn-Kanade (CK+) dataset [24] were used. Picked dataset consisting of 821 images of seven emotion classes: anger (126 photos), disgust (67 photos), fear (111 photos), happy (154 photos), sadness (107 photos), surprise (98 photos) and neutral (158 photos). Images were scaled and transformed to grayscale. All experiments were carried out using the TensorFlow framework. Examples of images are shown in Figure 2.
It can be seen that widespread and frequently used activation functions (ReLU, PreLU, leakyReLU) quickly achieve the highest possible accuracy for the task (in the range of 60-70%). Increasing the training duration leads to only a slight increase in accuracy. At the same time, the APWLSAF shows a trend towards increasing accuracy and higher performance (80-85%). Here it is also important to pay attention to the fact that the training dataset was small, and against this background, the gain in accuracy of approximately 15%, which is given by the APWLSAF with all other network parameters unchanged, is an important result. It confirms that the proposed function allows deep networks to learn complex tasks on small data samples faster and more accurately.
The new adaptive piece-wise linear squashing activation function (APWLSAF) is proposed,
which combines the properties of squashing functions of shallow neural networks (especially
three-layer perceptrons) and piece-wise linear functions in deep neural networks without
suffering from effect a "vanishing" gradient. The neuron tuning algorithm with APWLSAF,
characterized by high speed and ease of numerical implementation, has been introduced. The
results of the computer experiment confirm the effectiveness of proposed approach.
[14] Ye. Bodyanskiy, A. Deineko, V. Skorik, F. Brodetskyi. “Deep Neural Network with Adaptive
Parametric Rectified Linear Units and its Fast Learning.” International Journal of
Computing 21(
Systems 2 (1989): 303 – 314. [17] S. Kaczmarz, ”Approximate solution of systems of linear equations”, International Journal of
Control, 57:6, 1993, pp. 1269-1271, doi:10.1080/00207179308934446. [18] B. Widrow, M. Hoff, ”Adaptive Switching Circuits”, IRE WESCON Convention Record, Part 4, pp. 96-104. New York IRE. [19] S. Khlamov, V. Savanevych, Big astronomical datasets and discovery of new celestial bodies in the Solar System in automated mode by the CoLiTec software, in: Knowledge Discovery in Big Data from Astronomy and Earth Observation (1st ed., part IV), chapt. 18 Astrogeoinformatics, Elsevier, 2020, pp. 331-345. doi: 10.1016/B978-0-12-8191545.00030-8. [20] V. Savanevych, S. Khlamov, O. Briukhovetskyi, T. Trunova, I. Tabakova, Mathematical methods for an accurate navigation of the robotic telescopes, in: Mathematics, special issue Mathematics in Robot Control for Theoretical and Applied Problems, vol. 11 issue 10 (2246), 2023. 19 p. doi: 10.3390/math11102246. [21] V. E. Savanevych, S. V. Khlamov, V. S. Akhmetov, A. B. Briukhovetskyi, V. P. Vlasenko, E. N.
Dikov, I. Kudzej, P. A. Dubovsky, D. E. Mkrtichian, I. S. Tabakova, T. O. Trunova. “CoLiTecVS software for the automated reduction of photometric observations in CCD-frames.” Astronomy and Computing 40 (100605) (2022): 15. doi: 10.1016/j.ascom.2022.100605. [22] V. Akhmetov, S. Khlamov, V. Khramtsov, A. Dmytrenko. “Astrometric reduction of the widefield images.: Advances in Intelligent Systems and Computing IV: Springer Nature Switzerland 1080 (2020): 896–909. doi: 10.1007/978-3-030-33695-0_58. [23] N. Kulishova, Emotion Recognition Using Sigma-Pi Neural Network, in: Proceedings of 2016 IEEE First International Conference on Data Stream Mining & Processing (DSMP), Lviv Ukraine, 2016, pp. 327-331. [24] P. Lucey, J.F. Cohn, T. Kanade, J. Saragih, Z. Ambadar, I. Matthews, "The Extended CohnKanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression", in: Proceedings of IEEE workshop on CVPR for Human Communicative Behavior Analysis, San Francisco, USA, 2010.