I. INTRODUCTION

Nowadays, artificial neural networks (ANNs) are widely used to solve many problems arising in Data Science. Here, multilayer perceptron (MLP) [1,[13][14][15][16][17][18] is the most widely used. On the basis of MLP deep neural networks (DNNs) [2][3][4]19,21] were developed, that have improved characteristics in comparison with their prototypes, namely traditional shallow neural networks.

In the general case, a multilayer perceptron that contains L information processing layers ( 1 L − hidden layer and one output layer) realizes a nonlinear transformation that can be written in the form

( ) ( ) ( ) [ ] [ ] ( ) [ ] ( [ ] ( ) [ ] [ ] [ ] ( ) ( ) ( ) ( ) ( ) 1 1 2 1 1 ˆ1 1 ... 1 L L L L L Y k X k W k W k W k X k − − − = Ψ = Ψ − Ψ ×  × − Ψ Ψ −  

where: - is discrete time index. In the DNN family, the most popular are the convolutional neural networks (CNNs) [20,[22][23][24][25] that are mainly designed to process images represented in the form of ( )

1 2 n n × -matrices ( ) ( ) { } 1 2 i i X k x k = (where 1 1 1, 2,..., i n = and 2 2 1, 2,..., i n =

), which must be vectorized before submission to the network, i. e. they must be presented in the form of vectors [10], the dimension of which can be quite large, that leads to the effect of "curse of dimensionality".

This effect can be avoided by processing the original matrix using convolution, pooling and encoding operations.

As a result a vector of dimension smaller than ( )

1 2 1 n n × is fed to the perceptron's input.

Although DNNs provide high quality of the information processing, their training time is too long, and the training process itself may require considerable computing resources. However, it is possible to speed up the information processing by bypassing the operations of vectorizationdevectorization, i.e. by storing information that will be processed not in the form of a vector, but in the form of a matrix.

The abovementioned problem is solved by the matrix neural networks [5,6,11,12], that are quite complex from the computational point of view.

In this connection, it seems expedient to develop architecture and algorithms for tuning a deep matrix neural network that is characterized by the simplicity of the numerical realization and high speed of its synaptic weights learning.

II. ADAPTIVE BILINEAR MODEL

The proposed matrix DNN is based on the adaptive matrix bilinear model introduced earlier by the authors [7,8] ( ) { } ( ) ( ) ( )

1 2 1 1 2 2 ˆˆ1 1 ,

1, 2,..., ;

1, 2,...,

j j Y k y A k X k B k j n j n = = − − = =(1)

where ( )

1 A k − , ( ) 1 B k − are ( ) 1 1 n n × , ()2 2 n n × -matrices ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) T T 1 1 , 1 1 A B T T A A k A k k E k B k X k B k B k k X k A k E k η η = − + ×   × −   = − + ×   × −  (2)

is used or its version optimized by speed [7] that can be written as

( ) ( ) ( ) ( ) ( ( ) ( ) ( ) ( )) ( ) ( ) ( ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( )) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ( ) ( ) ( )) ( ) ( ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( )) ( ) ( ) ( ) T T T T T T T 1 T T T T T T T T T 1 T T T T 1 1 1 1 1 1 1 1 , 1 1 , A A A A A A k A k Tr E k B k X k X k B k E k Tr E k B k X k X k B k B k X k X k B k E k E k B k X k B k B k Tr E k A k X k X k A k E k Tr A k X k X k A k E k E k A k X k X k A k X k A k E k − −  = − + − ×   × − ×   × − ×   × − − ×   × − ×   × −   = − + ×   × ×   × ×   × ×   × −   (3)

that is the matrix generalization of the Kaczmarz-Widrow-Hoff learning algorithm (here

( ) A k η , ( ) B k η are learning rate parameters, ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( )1 1 , 1 ,A E k Y k A k X k B k E k Y k A k X k B k  = − − −   = − −   ( ) Y k is reference matrix signal).

The learning algorithm in Eq. ( 3) can be given additional filtering properties if the learning rate parameters in Eq. ( 2) are calculated using the recurrence relations that can be written in the form

( ) ( ) ( ) ( ) ( ) ( ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( )) 1 T T T T T 1 1 1 1 1 A A A k r k r k Tr E k B k X k X k B k B k X k X k B k E k η β − = = − + + − × × − × × − × × − and ( ) ( ) ( ) ( ) ( ) ( ) ( ( ) ( ) ( ) ( ) ( ) ( ) ( )) 1 T T T T T 1 B B B A A k r k r k Tr A k X k X k A k E k E k A k X k X k A k η β − = = − + × × × × ×

where 0 1 β ≤ ≤ is smoothing parameter [9].

On the basis of the model from Eq. ( 1), it is easy to introduce its nonlinear modification that can be written in the following form:

( ) { } ( ) ( ) ( ) ( ) ( ) 1 2 ˆˆ1 1 , j j Y k y A k X k B k U k = = Ψ − − = = Ψ  (4)

which is in fact the matrix generalization of the transformation that is realized by any of the layers of a multilayer perceptron.

In Eq. ( 4) Ψ denotes a ( )

1 2

n n × -matrix of activation functions, that acts elementwise on the matrix of internal activation signals of the system that are denoted by

( ) ( ) { } 1 2 j j U k u k = .

In this case, the adjustment of the parameters of the nonlinear matrix model in Eq. ( 4) can be realized on the basis of the modified δ -rule

( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) 1 2 1 2 1 2 2 1 2 1 2 1 2 2 1 2 1 2 1 2 1 1 2 1 2 1 1 2 1 2 1 2 1 1 2 1 2 1 2 1 1 2 1 2 1 1 1 1 1 ˆ1 ˆ, 1 1 1 j j j j A j j n j j j j i i i j j A j j j j i j j A j j i j j j j B A j j n A j j j j i i i j j B A j j a k a k k e k u k b k x k a k k e k u k x k a k k k x k b k b k k e k u k a k x k b k k e k η ψ η ψ η δ η ψ η ψ = = = − + × ′ × − = = − + × ′ × = − + + = − + × ′ × − = = − + × ′ × ∑ ∑ ( ) ( ) ( ) ( ) ( ) ( ) ( ) 1 2 2 1 2 1 2 2 ˆ1 ˆ. A j j i j j B A j j i u k x k b k k k x k η δ                    = − +   +   (5)

On the basis of Eq. ( 4) it is easy to introduce into consideration a multilayer matrix neural network that realizes the transformation

( ) [ ] ( ) [ ] ( ) ( ( ( [ ] ( ) ( ) [ ] ( ) ( ) ( ) [ ] ( ) )) [ ] ( ) ) 1 1 1 1 ˆ1 1 ... 1 1 ... 1 1 L L L L Y k A k A k A k X k B k B k B k − − = Ψ − Ψ − × × Ψ − − × × − −   (6)

Using the learning algorithm from Eq. ( 5) and error backpropagation, it is possible to obtain the adaptive procedure for tuning all parameters of the matrix DNN in Eq. ( 6): -for the output layer:

[ ] ( ) [ ] ( ) ( ) [ ] ( ) [ ] ( ) [ ] ( ) [ ] ( ) ( ) [ ] ( ) [ ] ( ) 1 2 1 2 1 2 1 1 2 1 2 1 2 2 1 1 1 , 1 L L L L j j j j A j j i L L L L j j j j B A j j A i a k a k k k o k b k b k k k o k η δ η δ − −  = − +   = − +   where [ ] ( ) [ ] ( ) ( ) ( ) 1 2 1 2 1 2 L L j j j j j j k u k e k δ ψ′ = , [ ] ( ) [ ] ( ) [ ] ( ) 2 1 1 2 1 2 2 1 1 1 ˆ1 n L L L i j j i i i o k b k o k − − = = − ∑ , [ ] ( ) [ ] ( ) ( ) ( ) 1 2 1 2 1 2 L L A j j A j j A j j k u k e k δ ψ′ = , [ ] ( ) [ ] ( ) [ ] ( ) 1 2 1 2 1 2 1 1 1 1 ˆn L L L A i j j i i i o k a k o k − − = = ∑ ; -for the l th hidden layer, 1 l L < < : [ ] ( ) [ ] ( ) ( ) [ ] ( ) [ ] ( ) [ ] ( ) [ ] ( ) ( ) [ ] ( ) [ ] ( ) 1 2 1 2 1 2 1 1 2 1 2 1 2 2 1 1 1 , 1 l l l l j j j j A j j i l l l l j j j j B A j j A i a k a k k k o k b k b k k k o k η δ η δ − −  = − +   = − +   where [ ] ( ) [ ] ( ) ( ) [ ] [ ] ( ) 1 1 2 1 2 1 2 1 2 1 1 1 1 n l l l l j j j j j j j j i k u k a k δ ψ δ + + = ′ = ∑ , [ ] ( ) [ ] ( ) [ ] ( ) 2 1 1 2 1 2 2 1 1 1 ˆ1 n l l l i j j i i i o k b k o k − − = = − ∑ , [] ( ) [ ] ( ) ( )[ ] ( ) [ ] ( ) 2 1 2 1 2 1 2 1 2 2 1 1 1 n l l l l A j j A j j A j j j j i k u k k b k δ ψ δ + + = ′ = ∑ , [ ] ( ) [ ] ( ) [ ] ( ) 1 2 1 2 1 1 2 1 1 1 1 ˆn l l l A i j j i i i i o k a k o k − − = = ∑

; -for the first hidden layer: Each observation is an image of size 28x28 pixels that represents a handwritten digit. In general the dataset has 10 classes (digits from 0 to 9). Some examples of the images from this dataset are presented in Fig. 1.

[ ] ( ) [ ] ( ) ( ) [ ] ( ) [ ] ( ) [ ] ( ) [ ] ( ) ( ) [ ] ( ) [ ] ( ) 1 2 1 2 1 2 1 1 2 1 2 1 2 2 1 1 1 0 1 1 1 0 1 , 1 j j j j A j j i j j j j B A j j A i a k a k k k o k b k b k k k o k η δ η δ  = − +   = − +   where [ ] ( ) [ ] ( ) ( ) [ ] [ ] ( ) 1 1 2 1 2 1 2 1 2 1 1 1 2 2 1 n j j j j j j j j i k u k a k δ ψ δ = ′ = ∑ , [ ] ( ) [ ] ( ) ( ) 2 1 1 2 1 2 2 0 1 1 ˆ1 n i j j i i i o k b k x k = = − ∑ , [ ] ( ) [ ] ( ) ( ) [ ] ( ) [ ] ( ) 2 1 2 1 2 1 2 1 2 2 1 1 2 2 1 n A j j j j A j j j j i k u k k b k δ ψ δ = ′ = ∑ , [ ] ( ) [ ] ( ) ( )

The elements of an image are represented by pixel values from 0 to 255, where 0 means white pixel (background) and 255 means black pixel (foreground). These values were preprocessed before training using normalization. The inputs for the network were ( )

1 2

n n × -matrices, where 1 2 28 n n = = .

Every hidden layer also had size of 1 2 28 28 n n × = × .

The results of the computational experiments are presented in Table 1.

TABLE 1. EXPERIMENTAL RESULTS

Number of layers in the network

Error on test set, %

IV. CONCLUSION

In this paper the matrix deep neural network and its learning algorithm are proposed. They allow significantly to reduce the number of adjustable weights due to the rejection of the vectorization-devectorization operations of 2D input signals.

One of the main advantages of the proposed system is that it also preserves the information between rows and columns of 2D inputs of the system.

The considered DNN in comparison with traditional multilayer perceptrons has increased speed, determined by reduced number of adjustable parameters and optimization of the learning algorithm, and the simplicity of numerical implementation.

The proposed system can be used to solve a wide range of machine learning tasks, particularly connected with the problems of image processing, where input signals are presented to the system for data processing in the form of a matrix.