141
Matrix Deep Neural Network and Its Rapid Learning in
Data Science Tasks
Iryna Pliss1, Olena Boiko2, Valentyna Volkova3, Yevgeniy Bodyanskiy4
1. Control Systems Research Laboratory, Kharkiv National University of Radio Electronics, UKRAINE, Kharkiv, Nauky ave., 14,
email: iryna.pliss@nure.ua
2. Control Systems Research Laboratory, Kharkiv National University of Radio Electronics, UKRAINE, Kharkiv, Nauky ave., 14,
email: olena.boiko@ukr.net
3. Samsung Electronics Ukraine Company, LLC R&D (SRK), UKRAINE, Kyiv, Lva Tolstogo St., 57,
email: v.volkova@samsung.com
4. Control Systems Research Laboratory, Kharkiv National University of Radio Electronics, UKRAINE, Kharkiv, Nauky ave., 14,
email: yevgeniy.bodyanskiy@nure.ua
Abstract: The matrix deep neural network and its to process images represented in the form of
learning algorithm are proposed. This system allows
reducing the number of tunable weights due to the
{ }
( n1 × n2 ) -matrices X ( k ) = xi1i2 ( k ) (where i1 = 1, 2,..., n1
rejection of the operations of vectorization- and i2 = 1, 2,..., n2 ), which must be vectorized before
devectorization. It also saves the information between submission to the network, i. e. they must be presented in the
rows and columns of 2D inputs. form of vectors [10], the dimension of which can be quite
Keywords: deep learning, multilayer network, data large, that leads to the effect of “curse of dimensionality”.
mining, 2D network. This effect can be avoided by processing the original
matrix using convolution, pooling and encoding operations.
I. INTRODUCTION
As a result a vector of dimension smaller than ( n1n2 × 1) is
Nowadays, artificial neural networks (ANNs) are widely fed to the perceptron’s input.
used to solve many problems arising in Data Science. Here, Although DNNs provide high quality of the information
multilayer perceptron (MLP) [1,13-18] is the most widely processing, their training time is too long, and the training
used. On the basis of MLP deep neural networks (DNNs) process itself may require considerable computing resources.
[2-4,19,21] were developed, that have improved However, it is possible to speed up the information
characteristics in comparison with their prototypes, namely processing by bypassing the operations of vectorization-
traditional shallow neural networks. devectorization, i.e. by storing information that will be
In the general case, a multilayer perceptron that contains processed not in the form of a vector, but in the form of a
L information processing layers ( L − 1 hidden layer and one matrix.
output layer) realizes a nonlinear transformation that can be The abovementioned problem is solved by the matrix
written in the form neural networks [5,6,11,12], that are quite complex from the
Yˆ ( k ) =Ψ ( X ( k )) =
L L
(
Ψ [ ] W [ ] ( k − 1) Ψ [ ] ×
L −1
computational point of view.
)
In this connection, it seems expedient to develop
(
× W[
L −1]
( k − 1) Ψ[ L − 2] (...Ψ[1] (W [1] ( k − 1) X ( k ) ) ) architecture and algorithms for tuning a deep matrix neural
network that is characterized by the simplicity of the
where: numerical realization and high speed of its synaptic weights
- Yˆ ( k ) denotes vector output signal of corresponding learning.
dimensions; II. ADAPTIVE BILINEAR MODEL
- X ( k ) denotes vector input signal of corresponding The proposed matrix DNN is based on the adaptive matrix
dimensions; bilinear model introduced earlier by the authors [7, 8]
- Ψ [l ] are diagonal matrices of activation functions on each { }
Yˆ ( k ) ==
yˆ j1 j2 A ( k − 1) X ( k ) B ( k − 1) ,
layer; j1 = 1, 2,..., n1 ; (1)
- W [ ] ( k − 1) are matrices of synaptic weights that are
l
j2 = 1, 2,..., n2
where A ( k − 1) , B ( k − 1) are ( n1 × n1 ) , ( n2 × n2 ) -matrices
adjusted during the learning process based on error
backpropagation;
- l = 1, 2,..., L ; of tunable parameters that are adjusted during online
learning-identification process.
- k = 1, 2,... is discrete time index.
For this, either the gradient adaptation procedure
In the DNN family, the most popular are the convolutional
neural networks (CNNs) [20,22-25] that are mainly designed
ACIT 2018, June 1-3, 2018, Ceske Budejovice, Czech Republic
142
A ( k )= A ( k − 1) + η A ( k ) ×
{ } Ψ ( A ( k − 1) X ( k ) B ( k − 1) ) =
Yˆ ( k ) =yˆ j1 j2 =
(4)
× E ( k ) B T ( k − 1) X T ( k ) , = Ψ U (k ),
(2)
B ( k )= B ( k − 1) + η B ( k ) × which is in fact the matrix generalization of the
× X T ( k − 1) AT ( k ) E A ( k )
transformation that is realized by any of the layers of a
multilayer perceptron.
is used or its version optimized by speed [7] that can be In Eq. (4) Ψ denotes a ( n1 × n2 ) -matrix of activation
written as
) A ( k − 1) + (Tr E ( k ) B T ( k − 1) ×
functions, that acts elementwise on the matrix of internal
A(k = activation signals of the system that are denoted by
× X T ( k ) X ( k ) B ( k − 1) E T ( k ) ) × {
U ( k ) = u j1 j2 ( k ) . }
× (Tr E ( k ) B T ( k − 1) X T ( k ) X ( k ) × In this case, the adjustment of the parameters of the
nonlinear matrix model in Eq. (4) can be realized on the basis
×B ( k − 1) B T ( k − 1) X T ( k ) X ( k ) × of the modified δ -rule
a j1 j2 (=k ) a j1 j2 ( k − 1) + η A ( k ) e j1 j2 ( k ) ×
×B ( k − 1) E T ( k ) ) E ( k ) ×
−1
( )
n2
×B T ( k − 1) X T ( k ) , (3)
×ψ ′ u j1 j2 ( k ) ∑ b j1 j2 ( k − 1) xi1i2 ( k ) =
B ( k )= B ( k − 1) + (Tr E A ( k ) A ( k ) X ( k ) ×
T i2 =1
= a j1 j2 ( k − 1) + η A ( k ) e j1 j2 ( k ) ×
× X T ( k ) AT ( k ) E A ( k ) ) (Tr A ( k ) ×
(
×ψ ′ u j1 j2 ( k ) xˆi1 (= )
k ) a j1 j2 ( k − 1) +
× X ( k ) X T ( k ) AT ( k ) E A ( k ) E AT ( k ) ×
+ η A ( k ) δ j1 j2 ( k ) xˆi1 ( k ) ,
× A ( k ) X ( k ) X T ( k ) AT ( k ) ) ×
−1 (5)
b j1 j2 (=
k ) b j1 j2 ( k − 1) + η B ( k ) eA j1 j2 ( k ) ×
× X T ( k − 1) AT ( k ) E A ( k ) ,
( )
n1
×ψ ′ u A j1 j2 ( k ) ∑ a j1 j2 ( k − 1) xi1i2 ( k ) =
that is the matrix generalization of the Kaczmarz–Widrow– i1 =1
Hoff learning algorithm (here η A ( k ) , η B ( k ) are learning
= b j1 j2 ( k − 1) + η B ( k ) eA j1 j2 ( k ) ×
rate parameters,
E ( k ) = Y ( k ) − A ( k − 1) X ( k ) B ( k − 1) , (
×ψ ′ u A j1 j2 ( k ) xˆi2 (= )
k ) b j1 j2 ( k − 1) +
E A ( k ) =Y ( k ) − A ( k ) X ( k ) B ( k − 1) , + η B ( k ) δ A j1 j2 ( k ) xˆi2 ( k ) .
Y ( k ) is reference matrix signal). On the basis of Eq. (4) it is easy to introduce into
consideration a multilayer matrix neural network that realizes
The learning algorithm in Eq. (3) can be given additional the transformation
(
filtering properties if the learning rate parameters in Eq. (2)
are calculated using the recurrence relations that can be
written in the form
Yˆ ( k ) =Ψ A[ ] ( k − 1) Ψ A[ ] ( k − 1) ×
L L −1
( (
η A−1 ( k=) rA ( k=) β rA ( k − 1) + ( (
× ...Ψ A[ ] ( k − 1) X ( k ) B[ ] ( k − 1) ... ×
1 1
) ) (6)
+ Tr ( E ( k ) B T ( k − 1) × × B[
L −1]
( k − 1) ) ) B[ L] ( k − 1) )
× X T ( k ) X ( k ) B ( k − 1) × Using the learning algorithm from Eq. (5) and error
× B T ( k − 1) X T ( k ) X ( k ) × backpropagation, it is possible to obtain the adaptive
procedure for tuning all parameters of the matrix DNN in
× B ( k − 1) E T ( k ) ) Eq. (6):
and - for the output layer:
η B−1 ( k=) rB ( k=) β rB ( k − 1) + a[jLj] (= k ) a[j1 j]2 ( k − 1) + η A ( k ) δ [j1 j]2 ( k ) oˆi[1 ] ( k ) ,
L L L −1
[ L]
1 2
× Tr ( A ( k ) X ( k ) X T ( k ) × k ) b[j1 j]2 ( k − 1) + η B ( k ) δ A[ j]1 j2 ( k ) oˆ[A i2 ] ( k )
b j1 j2 (=
L L L −1
× AT ( k ) E A ( k ) E AT ( k ) A ( k ) × where
× X ( k ) X T ( k ) AT ( k ) ) 1 2
(
δ [j Lj] ( k ) = ψ ′ u [jLj] ( k ) e j j ( k ) ,
1 2
) 1 2
where 0 ≤ β ≤ 1 is smoothing parameter [9].
n2
On the basis of the model from Eq. (1), it is easy to
oˆi[1 ] ( k )
=
L −1
∑ b[ ] ( k − 1) o[ ] ( k ) ,
i2 =1
L
j1 j2
L −1
i1i2
introduce its nonlinear modification that can be written in the
following form: 1 2
(
δ A[ Lj] j ( k ) = ψ ′ u [AL]j j ( k ) eA j j ( k ) ,
1 2
) 1 2
ACIT 2018, June 1-3, 2018, Ceske Budejovice, Czech Republic
143
n1
III. COMPUTATIONAL EXPERIMENTS
oˆ[A i2 ] ( k ) = ∑ a[j1 j]2 ( k ) oi[1i2 ] ( k ) ;
L −1 L L −1
i1 =1 The efficiency of the proposed system and learning
- for the l th hidden layer, 1 < l < L : methods was demonstrated on the classification task.
A number of experiments was carried out on the MNIST
a[jl ]j (=k ) a[j1]j2 ( k − 1) + η A ( k ) δ [j1 ]j2 ( k ) oˆi[1 ] ( k ) ,
l l l −1
dataset that was introduced by Yann LeCun and Corinna
[l ]
1 2
k ) b[j1 ]j2 ( k − 1) + η B ( k ) δ A[ ]j1 j2 ( k ) oˆ[A i2 ] ( k )
b j1 j2 (=
l l l −1 Cortes [26].
This dataset is widely used for training and testing in
where machine learning, namely in the classification task. This
( )
n1 dataset contains 60000 training observations and 10000 test
δ [jl ]j ( k ) = ψ ′ u [jl ]j ( k ) ∑ δ [jl +j 1] a[jl +j 1] ( k ) ,
1 2 1 2 1 2 1 2 observations.
i1 =1
Each observation is an image of size 28x28 pixels that
n2
represents a handwritten digit. In general the dataset has
oˆi[1 ] ( k )
=
l −1
∑ b[ ] ( k − 1) o[ ] ( k ) ,
i2 =1
l
j1 j2
l −1
i1i2 10 classes (digits from 0 to 9).
Some examples of the images from this dataset are
( )
n2
δ A[l ]j j ( k ) = ψ ′ u [Al ]j j ( k ) ∑ δ A[l +j1j] ( k ) b[jl +j 1] ( k ) ,
presented in Fig. 1.
1 2 1 2
i2 =1
1 2 1 2 The elements of an image are represented by pixel values
n1
from 0 to 255, where 0 means white pixel (background) and
oˆ[A i2 ] ( k ) = ∑ a[j1]j2i1 ( k ) oi[1i2 ] ( k ) ;
l −1 l l −1 255 means black pixel (foreground). These values were
i1 =1 preprocessed before training using normalization. The inputs
- for the first hidden layer: for the network were ( n1 × n2 ) -matrices, where n=
1 n=
2 28 .
a[j1]j (= k ) a[j1]j2 ( k − 1) + η A ( k ) δ [j1 ]j2 ( k ) oˆi[1 ] ( k ) ,
1 1 0
Every hidden layer also had size of n1 × n2 = 28 × 28 .
[1]
1 2
The results of the computational experiments are presented
k ) b[j1 ]j2 ( k − 1) + η B ( k ) δ A[ ]j1 j2 ( k ) oˆ[A i]2 ( k )
b j1 j2 (=
1 1 0
in Table 1.
where
TABLE 1. EXPERIMENTAL RESULTS
( )
n1
δ [j1]j ( k ) = ψ ′ u [j1]j ( k ) ∑ δ [j 2j] a[j2]j ( k ) ,
1 2 1 2 1 2 1 2
i1 =1 Number of layers Error on test set,
n2 in the network %
oˆi[1 ] ( k )
=
0
∑ b[ ] ( k − 1) x ( k ) ,
i2 =1
1
j1 j2 i1i2 3 25
5 20
( )
n2
δ A[1]j j ( k ) = ψ ′ u [j1]j ( k ) ∑ δ A[ 2]j j ( k ) b[j2j] ( k ) ,
1 2 1 2
i2 =1
1 2 1 2
10 18
n1
oˆ[A i]2 ( k ) = ∑ a[j1]j2 ( k ) xi1i2 ( k ) .
0 1
i1 =1
Fig.1. Examples of the images from the MNIST dataset.
ACIT 2018, June 1-3, 2018, Ceske Budejovice, Czech Republic
144
IV. CONCLUSION [12] M. Mohamadian, H. Afarideh, and F. Babapour,
“New 2D Matrix-Based Neural Network for
In this paper the matrix deep neural network and its Image Processing Applications,” IAENG (International
learning algorithm are proposed. They allow significantly to Association of Engineers) International Journal of
reduce the number of adjustable weights due to the rejection Computer Science, 42(3), pp. 265-274, 2015.
of the vectorization-devectorization operations of 2D input [13] K. Suzuki, Artificial Neural Networks: Architectures
signals. and Applications. NY: InTech, 2013.
One of the main advantages of the proposed system is that [14] K. L. Du and M. Swamy, Neural Networks and
it also preserves the information between rows and columns Statistical Learning. Springer-Verlag London, 2014.
of 2D inputs of the system. [15] D. Graupe, Principles of Artificial Neural
The considered DNN in comparison with traditional Networks (Advanced Series in Circuits and Systems).
multilayer perceptrons has increased speed, determined by Singapore: World Scientific Publishing Co. Pte. Ltd.,
reduced number of adjustable parameters and optimization of 2007.
the learning algorithm, and the simplicity of numerical [16] L. Rutkowski, Computational intelligence. Methods and
implementation. techniques, Berlin-Heidelberg: Springer-Verlag, 2008.
The proposed system can be used to solve a wide range of [17] R. Kruse, C. Borgelt, F. Klawonn, C. Moewes,
machine learning tasks, particularly connected with the M. Steinbrecher, and P. Held, Computational
problems of image processing, where input signals are intelligence, Berlin: Springer, 2013.
presented to the system for data processing in the form of a [18] D. T. Pham and X. Liu, Neural Networks for
matrix. Identification, Prediction and Control, London:
REFERENCES Springer-Verlag, 1995.
[19] I. Arel, D. Rose, and T. Karnowski, “Deep machine
[1] C. M. Bishop, Neural Networks for Pattern Recognition. learning – a new frontier in artificial intelligence
Oxford : Clarendon Press, 1995. research,” IEEE Computational Intelligence Magazine,
[2] Y. LeCun, Y. Bengio, G. Hinton, “Deep Learning,” vol. 5, no. 4, pp. 13-18, 2010.
Nature, vol. 521, pp. 436-444, 2015. [20] K. Kavukcuoglu, P. Sermanet, Y-L. Boureau, K. Gregor,
[3] J. Schmidhuber, “Deep Learning in neural networks: An M. Mathieu, Y.. LeCun, “Learning Convolutional
overview,” Neural Networks, vol. 61, pp. 85-117, 2015. Feature Hierachies for Visual Recognition,” in
[4] I. Goodfellow, Y. Bengio, and A. Courville, Deep Proceedings of the 23rd International Conference on
Learning. MIT Press, 2016. Neural Information Processing Systems, vol. 1,
[5] P. Daniušis and P. Vaitkus, “Neural networks pp. 1090-1098, 2010.
with matrix inputs,” Informatica, 19, №4, pp. 477-486, [21] C. Dan, U. Meier, and J. Schmidhuber, “Multi-column
2008. deep neural networks for image classification,” in
[6] J. Gao, Y. Guo, and Z. Wang, “Matrix neural networks,” Proceedings of the 2012 IEEE Conference on
in Proceedings of the14th International Symposium on Computer Vision and Pattern Recognition (CVPR),
Neural Networks (ISNN), Part II, Sapporo, Japan, 2017, pp. 3642-3649, 2012.
pp. 1–10. [22] A. Krizhevsky, I. Sutskever, and G. E. Hinton,
[7] Ye. V. Bodyanskiy, I. P. Pliss, and V. A. Timofeev, “ImageNet classification with deep convolutional
“Discrete adaptive identification and neural networks,” in Proceedings of the 25th
extrapolation of two-dimensional fields,” Pattern International Conference on Neural Information
Recognition and Image Analysis, 5, №3, pp. 410-416, Processing Systems (NIPS’12), vol. 1, pp. 1097-1105,
1995. 2012.
[8] S. Haykin, Neural Networks: A Comprehensive [23] K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual
Foundation. Upper Saddle River, N. J. : Prentice Hall, Learning for Image Recognition,” in 2016 IEEE
Inc., 1999. Conference on Computer Vision and Pattern
[9] S. Vorobyov, Ye. Bodyanskiy, “On a non-parametric Recognition (CVPR), pp. 770-778 2016.
algorithm for smoothing parameter control in adaptive
[24] Y. LeCun, K. Kavukcuoglu, and C. Farabet,
filtering,” Engineering Simulation, vol. 16, p. 314-320,
“Convolutional networks and applications in vision,” in
1999.
Proceedings of 2010 IEEE International Symposium
[10] Y. Guo, Y. Liu, A. Oerlemans, S. Lao, S. Wu, and
on Circuits and Systems (ISCAS), pp. 253-256, 2010.
M. S. Lew, “Deep learning for visual understanding:
[25] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed,
A review,” Neurocomputing, vol. 187, pp. 27-48, 2016.
D. Anguelov, D. Erhan, V. Vanhoucke,
[11] P. Stubberud, “A vector matrix real time
and A. Rabinovich, “Going deeper with convolutions,”
backpropagation algorithm for recurrent neural
in Proceedings of the IEEE Conference
networks that approximate multi-valued periodic
on Computer Vision and Pattern Recognition, pp. 1-9,
functions,” International Journal on
2015.
Computational Intelligence and Application, 8(4),
[26] http://yann.lecun.com/exdb/mnist/
pp. 395-411, 2009.
ACIT 2018, June 1-3, 2018, Ceske Budejovice, Czech Republic