Deep Neural Networks in Digital Economy1

Abstract

M odern generations of artificial neural networks are more complex and flexible architectures than neural networks of the first generations. Increasing application possibilities, improved existing artificial neural networks. Solving the problem of developing artificial neural networks that must solve a number of complex mathematical problems, problems aimed at developing the most rapid and flexible learning algorithms. Overview of modern research in the field of applied artificial intelligence and the most interesting generations of artificial neural networks. Of the most relevant to date neural network architectures, we should highlight deep neural networks, impulse neural networks and capsular neural networks. Deep Learning Neural Networks are based on the teaching of representations and not on specialized algorithms designed for specific tasks. M any deep learning methods were known as early as the 1980s, but the results were unimpressive, while advances were made in the theory of artificial neural networks (pre-training of neural networks using a special case of an undirected graphical model, the so-called limited Boltzmann machine). One of the most biologically inspired models of artificial neural networks is the so-called impulse or spike neural networks, which are the third generation of neural networks. Pulsed neural networks or spike basically contain the principles of operation of biological neurons. Unlike other neural network architectures, in impulse networks, neurons exchange short impulses of the same amplitude between themselves, which allows achieving tremendous energy efficiency. Neuromorphic chips are built on the basis of pulse architectures to simulate the operation of a biological neural network. Capsule networks use small groups of neurons, called capsules, which are concentrated in layers, use spatial relationships, and can recognize objects in video and images. When multiple capsules in the same layer match as a result of recognition, they activate the capsule with a layer above and so on until the network brings together the whole image. This approach is the newest in the field of artificial neural networks.

Keywords

• Deep Learning
• Digital Economy
• Neural Networks
• Capsule Neural Networks
• Spiking neural network

Introduction

Modern generations of artificial neural networks are more complex and flexible architectures than neural networks of the first generations. Increasing application possibilities, improved existing artificial neural networks. Solving the problem of developing artificial neural networks that must solve a number of complex mathematical problems, problems aimed at developing the most rapid and flexible learning algorithms. Overview of modern research in the field of applied artificial intelligence and the most interesting generations of artificial neural networks. It is also necessary to pay special attention to the state of artificial intelligence in Russia and abroad. 2

Deep Learning

Among the most actively developing models of artificial neural networks is to highlight the deep models of neural networks (Deep Learning Neural Networks). Similar models of neural networks have already proven their effectiveness in image recognition and classification tasks. The most popular architecture is the so -called convolutional neural network. Technological giants allocate billions of dollars for research and development in the field of artificial intelligence. An example is a convolutional neural network. I have a network developed by Google - GoogLeNet. Also, among the well-known neural networks - champions, AlexNet should be highlighted.

Of the most relevant to date neural network architectures, we should highlight deep neural networks, impulse neural networks and capsular neural networks that appeared literally at the end of 2017, which literally exploded in the field of artificial neural networks. Therefore, we dwell on these three architectures.

Deep Neural Networks or Deep Learning NN are a class based on machine learning methods.

In-depth training is based on the teaching of represent ations (eng. Feature / representation learning), and not on specialized algorithms designed for specific tasks. Many deep learning methods were known as early as the 1980s, but the results were unimpressive [ 1 ], while advances were made in the theory of artificial neural networks (pretraining of neural networks using a special case of an undirected graphical model, the so-called limited Boltzmann machine) x (above all, Nvidia GPUs, and now Google’s tensor processors) did not allow creating complex technolo gical architectures of neural networks with sufficient performance and allow to solve a wide range of tasks, do not be an effective solution before, for example, in computer vision, machine translation, speech recognition, with quality solutions, in many cases, are now comparable, and in some cases superior to "the protein" experts. Unlike machine learning, depth learning requires a much larger amount of training sample than in the case of machine learning. Also, unlike machine learning, a deep neural netwo rk can have thousands of layers. All this helps deep neural networks to achieve a sufficiently high accuracy in the tasks of analysis, classification and image recognition. But, the main drawback of in -depth training is the enormous resource-intensiveness; in order to train a deep neural network, it is sometimes necessary to make a training sample of a million images, or even more, and the learning process can take several days. For such tasks, even developed separate GPU processors, to speed up the learning process.

The most well known example of a deep learning network is the convolutional neural network. This architecture of artificial neural networks, proposed by Yan Lekun in 1988 [ 2 ] and aimed at effective image recognition. It is no secret to anyone that scientists engaged in developments in the field of artificial intelligence face the task of copying the work of biological intelligence in the best way possible, in other words, creating the most biologically inspired model o f the human brain. One of the most biologically inspired models of artificial neural networks is the so-called impulse or spike neural networks, which are the third generation of neural networks. The word spike comes from the English word spike, that is, impulse. Pulsed neural networks or spike [ 4 ] basically contain the principles of operation of biological neurons. Unlike other neural network architectures, in impulse networks, neurons exchange short impulses of the same amplitude between themselves, which allows achieving tremendous energy efficiency.

Neuromorphic chips are built on the basis of pulse architectures to simulate the operation of a biological neural network.

The active development of a new field of neuromorphic technology is associated with the development of principles, architectures and implementations of neurobiological systems. Such a neuromorphic approach implies a departure from known models of formal neural networks and attempts at software and hardware implementation of models of the functional parts of the brain and nervous system.

The practical development of this trend is currently supported by IBM [28] in terms of neuromorphic computing (neuromorphic computing), and in terms of hardware implementations of neuromorphic technology and its use - by the US defense agency DARPA, which in 2008 implemented the SyNAPSE project [ 5 ]. Examples of early development of neuromorphic ASICs are: Silicon Retina (eye model) [ 6 ], Silicon Cochlea (ear model) [ 7 ], and others. Neuromorphic technology shou ld provide for the construction of machines that have similar human perception, ability for self-organization [ 8 ], robustness in relation to changes in the environment and the control object.

Neurogrid (Stanford University). Analog approach to neuron modeling (106 neurons, 6 · 109 synapses). The variability of a large number of parameters allows the study of ensembles of neurons of different types. The problem is an outdated technological base.

SpiNNaker (University of Manchester). It is aimed at creating a neuromorphic hardware platform for the implementation of the European project Human Brain Project. The project is based on the use of special digital chips with the ability to build highly scalable modular systems with a different topology of their connec tion. Each chip contains 16 ARM9 processors and can emulate in real time the work of tens of thousands of neurons. The router provides delivery within a chip of 5 · 109 spikes / second. BrainScaleS (University of Heidelberg). The project aims to study and simulate the human brain. Created a hybrid digital-analog neurochip. A system was built to simulate ANNs from 2 · 105 neurons, 5 · 107 synapses. 4

Capsule Neural Networks

Capsule neural networks were proposed by Jeffrey Hinton, one of the researchers who suggested teaching the neural network the method of back propagation of an error. In 2012, he proposed to teach deep neural networks using the method of back-propagation of error, which later allowed for a breakthrough in image recognition tasks. In October 2017, he published work [ 9 ], in which he presented a new architecture of neural networks, called capsular neural networks. This kind of architecture can be a revolution in the field of recognition, as they can solve the problems of all favorite convolutiona l neural networks. The problem of convolutional neural network (CNN) is that in the learning process for image recognition, information about the spatial relationships of all functions is lost. CNN does not remember the position of the image elements in space and the same object, but from a different angle it can already be perceived as another image. Therefore, it is necessary to make a huge sample for the same object from all angles. Capsule neural networks can use spatial relationships to solve a similar problem.

This approach exceeded CNN quite significantly, reducing the number of errors by 45%. Capsule networks use small groups of neurons, called capsules, which are concentrated in layers, use spatial relationships, and can recognize objects in video and images. When multiple capsules in the same layer match as a result of recognition, they activate the capsule with a layer above and so on until the network brings together the whole image. Each of the capsules is designed to detect a specific function of an image in such a way that it can recognize them from different angles. This approach is the newest in the field of artificial neural networks. 5

The State of AI Researches in Russia and USA

In order to assess the state of affairs in the field of artificial intelligence in Russia and compare them with foreign competitors, it is advisable to compare the number of scientific publications over the past year and the amount of funds allocated to the development of the field.

Let us single out the most ambitious Western projects aimed at the development of artificial intelligence and teams working in this field: DARPA is a US Department of Defense directorate that is responsible for developing new technologies in the military. The task of DARPA is to preserve the technological superiority of the US armed forces, to prevent the emergence for the United States of new technical means of armed struggle, to support breakthrough research, to bridge the gap between basic research and their use in the military sphere. In the first 5 years, the management of the SyNAPSE project has allocated more than 106 million dollars. Also, annually large funds are allocated for holding competitions among developers, thereby motivating the development of new technologies.

ARPA-E - Agency for Advanced Energy Research, USA.

The Advanced Research Foundation is the analogue of DARPA, created to facilitate the implementation of research and development in the interests of Russian defense and state security, associated with a high degree of risk of achieving qualitatively new results in the military-technical, technological and socio-economic spheres.

The Russian Science Foundation is a non-profit organization established for the purpose of financial and organizational support for basic and exploratory research, the training of scientific personnel, and the development of research teams that hold leading positions in a specific field of science.

The Russian Foundation for Basic Research is a self-governing state non-profit organization in the form of a federal institution administered by the Government of the Russian Federation. The RFBR was established by Decree of the President of the Russian Federation of April 27, 1992 No. 426 “On Urgent Measures for the Preservation of the Scientific and Technical Potential of the Russian Federation”. By order of the Government of Russia of February 29, 2016, another state grant fund, the Russian Humanitarian Fund, was attached to the RFBR.

In general, in Russia for 10 years to support the development of about 23 billio n rubles. For comparison, about 200 million dollars are allocated annually in the United States (about 12 billion rubles). Obviously, the difference is enormous. Accordingly, in order to be able to compete in this area with advanced countries, it is necessary to increase the amount of funds invested in the development of artificial intelligence. It is advisable to introduce special courses of artificial intelligence in universities and to develop the potential of potential developers at the university stage. Conduct seminars and lectures with the participation of leading scientists and businessmen working with real tasks in the field of artificial intelligence and, in particular, artificial neural networks.

References

1. Bottou L . et al. Comparison of classifier methods: a case study in handwritten digit recognition //Pattern Recognition , 1994 . Vol. 2 - Conference B : Computer Vision & Image Processing ., Proceedings of the 12th IAPR International. Conference on , vol. 2 , 77 - 82 , IEEE, ( 1994 ).

2. . LeCun, B. Boser , J. S. Denker , D. Henderson , R. E. Howard , W. Hubbard and L. D. Jackel : Backpropagation Applied to Handwritten Zip Code Recognition, Neural Computation , 1 ( 4 ): 541 - 551 , Winter ( 1989 )

3. Russakovsky O. et al. Imagenet large scale visual recognition challenge // International Journal of Computer Vision , vol. 115 , № . 3 , 211 - 252 , ( 2015 ).

4. Pilato , Giovanni, Sergey A. Yarushev , and Alexey N. Averkin . "Prediction and Detection of User Emotions Based on Neuro-Fuzzy Neural Networks in Social Networks." International Conference on Intelligent Information Technologies for Industry . Springer, Cham, ( 2018 ).

5. M erolla, P.A. A million spiking-neuron integrated circuit with a scalable communication network and interface / P.A. M erolla et al. // Science 08 Aug 2014 , Vol. 345 , Issue 6197, pp . 668 - 673 , ( 2014 ).

6. Zaghloul , K.A. A silicon retina that reproduces signals in the optic nerve / K.A . Zaghloul , K. Boahen // Journal of neural engineering, Vol 3, P. 257267, ( 2006 ).

7. Wen , B. A Silicon Cochlea with Active Coupling / B. Wen, K. Boahen // IEEE Transactions on biomedical circuits and systems , December 2009 , Vol. 3 , No. 6 , pp. 444 - 455 , ( 2009 ).

8. Ranganathan , A. et al. Self-Organization in Artificial Intelligence and the Brain // Georgia Tech online . URL: http://www.cc.gatech.edu/people/home/zkira/research.html/, last accessed 2019/04/1

9. Sabour S. , Frosst N. , Hinton G. E. Dynamic routing between capsules // Advances in Neural Information Processing Systems , pp . 3859 - 3869 , ( 2017 ).