Deep Neural Networks in Digital Economy1 Alexey Averkin 1,2[0000-0003-1571-3583] and Sergey Yarushev 2[0000-0003-1352-9301] 1 Dorodnicyn Computing Centre, FRC CSC RAS, Vavilov st. 40, 119333 M oscow, Russia 2 Plekhanov University of Economics, Stremyanny lane 36, M oscow, 117997, Russia yarushev.sa@rea.com Abstract. M odern generations of artificial neural networks are more complex and flexible architectures than neural networks of the first generations. Increasing application possibilities, improved existing artificial neural networks. Solving the problem of developing artificial neural networks that must solve a number of complex mathematical problems, problems aimed at developing the most rapid and flexible learning algorithms. Overview of modern research in the field of applied artificial intelligence and the most interesting generations of artificial neural networks. Of the most relevant to date neural network architectures, we should highlight deep neural networks, impulse neural networks and capsular neural networks. Deep Learning Neural Networks are based on the teaching of representations and not on specialized algorithms designed for specific tasks. M any deep learning methods were known as early as the 1980s, but the results were unimpressive, while advances were made in the theory of artificial neural networks (pre-training of neural networks using a special case of an undirected graphical model, the so-called limited Boltzmann machine). One of the most bi- ologically inspired models of artificial neural networks is the so-called impulse or spike neural networks, which are the third generation of neural networks. Pulsed neural networks or spike basically contain the principles of operation of biological neurons. Unlike other neural network architectures, in impulse net- works, neurons exchange short impulses of the same amplitude between them- selves, which allows achieving tremendous energy efficiency. Neuromorphic chips are built on the basis of pulse architectures to simulate the operation of a biological neural network. Capsule networks use small groups of neurons, called capsules, which are concentrated in layers, use spatial relationships, and can rec- ognize objects in video and images. When multiple capsules in the same layer match as a result of recognition, they activate the capsule with a layer above and so on until the network brings together the whole image. This approach is the newest in the field of artificial neural networks. Keywords: Deep Learning, Digital Economy, Neural Networks, Capsule Neu- ral Networks, Spiking neural network. 1 The reported study was funded by RFBR according to the research project № 17-07-01558 Proceedings of the XXII International Conference “Enterprise Engineering and Knowledge M anagement” April 25-26, 2019, M oscow, Russia 2 1 Introduction Modern generations of artificial neural networks are more complex and flexible archi- tectures than neural networks of the first generations. Increasing application possibili- ties, improved existing artificial neural networks. Solving the problem of developing artificial neural networks that must solve a number of complex mathematical problems, problems aimed at developing the most rapid and flexible learning algorithms. Over- view of modern research in the field of applied artificial intelligence and the most in- teresting generations of artificial neural networks. It is also necessary to pay special attention to the state of artificial intelligence in Russia and abroad. 2 Deep Learning Among the most actively developing models of artificial neural networks is to highlight the deep models of neural networks (Deep Learning Neural Networks). Similar models of neural networks have already proven their effectiveness in image recognition and classification tasks. The most popular architecture is the so-called convolutional neural network. Technological giants allocate billions of dollars for research and development in the field of artificial intelligence. An example is a convolutional neural network. I have a network developed by Google - GoogLeNet. Also, among the well-known neu- ral networks - champions, AlexNet should be highlighted. Of the most relevant to date neural network architectures, we should highlight deep neural networks, impulse neural networks and capsular neural networks that appeared literally at the end of 2017, which literally exploded in the field of artificial neural net- works. Therefore, we dwell on these three architectures. Deep Neural Networks or Deep Learning NN are a class based on machine learning methods. In-depth training is based on the teaching of representations (eng. Feature / repre- sentation learning), and not on specialized algorithms designed for specific tasks. Many deep learning methods were known as early as the 1980s, but the results were unim- pressive [1], while advances were made in the theory of artificial neural networks (pre- training of neural networks using a special case of an undirected graphical model, the so-called limited Boltzmann machine) x (above all, Nvidia GPUs, and now Google’s tensor processors) did not allow creating complex technolo gical architectures of neural networks with sufficient performance and allow to solve a wide range of tasks, do not be an effective solution before, for example, in computer vision, machine translation, speech recognition, with quality solutions, in many cases, are now comparable, and in some cases superior to "the protein" experts. Unlike machine learning, depth learning requires a much larger amount of training sample than in the case of machine learning. Also, unlike machine learning, a deep neural network can have thousands of layers. All this helps deep neural networks to achieve a sufficiently high accuracy in the tasks of analysis, classification and image recognition. But, the main drawback of in -depth training is the enormous resource-intensiveness; in order to train a deep neural network, it is sometimes necessary to make a training sample of a million images, or even more, 3 and the learning process can take several days. For such tasks, even developed separate GPU processors, to speed up the learning process. The most well known example of a deep learning network is the convolutional neural network. This architecture of artificial neural networks, proposed by Yan Lekun in 1988 [2] and aimed at effective image recognition. Fig. 1. Convolutional neural network Perhaps the most famous example of a convolutional neural network is the AlexNet and GoogLeNet networks. The latter won the ImageNet recognition challenge [3] in 2014, with the result of 6.67% top 5 error. Let me remind you, top 5 error is a metric in which the algorithm can produce 5 variants of a picture class and an error is counted if among all these variants there is no correct one. In total, the test sample contains 150000 images and 1000 categories, that is, the task is extremely nontrivial. Today, the largest technology companies are investing billions of dollars in the de- velopment of applied artificial intelligence technologies and the development of artifi- cial neural network 3 Spiking Neural Networks It is no secret to anyone that scientists engaged in developments in the field of artificial intelligence face the task of copying the work of biological intelligence in the best way possible, in other words, creating the most biologically inspired model o f the human brain. One of the most biologically inspired models of artificial neural networks is the so-called impulse or spike neural networks, which are the third generation of neural networks. The word spike comes from the English word spike, that is, impulse. Pulsed neural networks or spike [4] basically contain the principles of operation of biological 4 neurons. Unlike other neural network architectures, in impulse networks, neurons ex- change short impulses of the same amplitude between themselves, which allows achiev- ing tremendous energy efficiency. Neuromorphic chips are built on the basis of pulse architectures to simulate the op- eration of a biological neural network. The active development of a new field of neuromorphic technology is associated with the development of principles, architectures and implementations of neurobiological systems. Such a neuromorphic approach implies a departure from known models of formal neural networks and attempts at software and hardware implementation of mod- els of the functional parts of the brain and nervous system. The practical development of this trend is currently supported by IBM [28] in terms of neuromorphic computing (neuromorphic computing), and in terms of hardware im- plementations of neuromorphic technology and its use - by the US defense agency DARPA, which in 2008 implemented the SyNAPSE project [5]. Examples of early de- velopment of neuromorphic ASICs are: Silicon Retina (eye model) [6], Silicon Cochlea (ear model) [7], and others. Neuromorphic technology shou ld provide for the construc- tion of machines that have similar human perception, ability for self-organization [8], robustness in relation to changes in the environment and the control object. Neurogrid (Stanford University). Analog approach to neuron modeling (106 neu- rons, 6 · 109 synapses). The variability of a large number of parameters allows the study of ensembles of neurons of different types. The problem is an outdated technological base. SpiNNaker (University of Manchester). It is aimed at creating a neuromorphic hard- ware platform for the implementation of the European project Human Brain Project. The project is based on the use of special digital chips with the ability to build highly scalable modular systems with a different topology of their connection. Each chip con- tains 16 ARM9 processors and can emulate in real time the work of tens of thousands of neurons. The router provides delivery within a chip of 5 · 109 spikes / second. Brain- ScaleS (University of Heidelberg). The project aims to study and simulate the human brain. Created a hybrid digital-analog neurochip. A system was built to simulate ANNs from 2 · 105 neurons, 5 · 107 synapses. 4 Capsule Neural Networks Capsule neural networks were proposed by Jeffrey Hinton, one of the researchers who suggested teaching the neural network the method of back propagation of an error. In 2012, he proposed to teach deep neural networks using the method of back-propagation of error, which later allowed for a breakthrough in image recognition tasks. In October 2017, he published work [9], in which he presented a new architecture of neural net- works, called capsular neural networks. This kind of architecture can be a revolution in the field of recognition, as they can solve the problems of all favorite convolutiona l neural networks. The problem of convolutional neural network (CNN) is that in the learning process for image recognition, information about the spatial relationships of all functions is lost. CNN does not remember the position of the image elements in 5 space and the same object, but from a different angle it can already be perceived as another image. Therefore, it is necessary to make a huge sample for the same object from all angles. Capsule neural networks can use spatial relationships to solve a similar problem. Fig. 2. Capsule neural network This approach exceeded CNN quite significantly, reducing the number of errors by 45%. Capsule networks use small groups of neurons, called capsules, which are con- centrated in layers, use spatial relationships, and can recognize objects in video and images. When multiple capsules in the same layer match as a result of recognition, they activate the capsule with a layer above and so on until the network brings together the whole image. Each of the capsules is designed to detect a specific function of an image in such a way that it can recognize them from different angles. This approach is the newest in the field of artificial neural networks. 5 The State of AI Researches in Russia and USA In order to assess the state of affairs in the field of artificial intelligence in Russia and compare them with foreign competitors, it is advisable to compare the number of sci- entific publications over the past year and the amount of funds allocated to the devel- opment of the field. Let us single out the most ambitious Western projects aimed at the development of artificial intelligence and teams working in this field: 1. SyNAPSE program: DARPA program for financing the development of neuromor- phic technologies, processors and systems that potentially scale up to a level com- parable to the brain size of animals, such as a mouse or a cat. 2. Stanford University: Brian A. Wandell, H.-S. Philip wong 3. Cornell University: Rajit Manohar 4. Columbia University Medical Center: Stefano Fusi 5. University of Wisconsin Madison: Giulio Tononi 6. University of California, Merced: Christopher Kello 7. IBM Research: Rajagopal Ananthanarayanan, Leland Chang, Daniel Friedman , Christoph Hagleitner, Bulent Kurdi, Chung Lam, Paul Maglio, Dharmendra Modha (Eng.) Russian, Stuart Parkin, Bipin Rajendran, Raghavendra Singh 6 8. Boston University: Stephen Grossberg, Gail Carpenter, Yongqiang Cao, Praveen Pilly 9. Neurosciences Institute: Gerald Edelman, Einar Gall, Jason Fleischer 10. University of Michigan: Wei Lu 11. University of California, Irvine: Jeff Krichmar 12. George Mason University: Giorgio Ascoli, Alexei Samsonovich 13. Portland State University: Christof Teuscher 14. Stanford University: Mark Schnitzer 15. Set Corporation: Chris Long DARPA is a US Department of Defense directorate that is responsible for developing new technologies in the military. The task of DARPA is to preserve the technological superiority of the US armed forces, to prevent the emergence for the United States of new technical means of armed struggle, to support breakthrough research, to bridge the gap between basic research and their use in the military sphere. In the first 5 years, the management of the SyNAPSE project has allocated more than 106 million dollars. Also, annually large funds are allocated for holding competitions among developers, thereby motivating the development of new technologies. ARPA-E - Agency for Advanced Energy Research, USA. The Advanced Research Foundation is the analogue of DARPA, created to facilitate the implementation of research and development in the interests of Russian defense and state security, associated with a high degree of risk of achieving qualitatively new re- sults in the military-technical, technological and socio-economic spheres. The Russian Science Foundation is a non-profit organization established for the pur- pose of financial and organizational support for basic and exploratory research, the training of scientific personnel, and the development of research teams that hold leading positions in a specific field of science. The Russian Foundation for Basic Research is a self-governing state non-profit or- ganization in the form of a federal institution administered by the Government of the Russian Federation. The RFBR was established by Decree of the President of the Rus- sian Federation of April 27, 1992 No. 426 “On Urgent Measures for the Preservation of the Scientific and Technical Potential of the Russian Federation”. By order of the Government of Russia of February 29, 2016, another state grant fund, the Russian Hu- manitarian Fund, was attached to the RFBR. In general, in Russia for 10 years to support the development of about 23 billio n rubles. For comparison, about 200 million dollars are allocated annually in the United States (about 12 billion rubles). Obviously, the difference is enormous. Accordingly, in order to be able to compete in this area with advanced countries, it is necessary to in- crease the amount of funds invested in the development of artificial intelligence. It is advisable to introduce special courses of artificial intelligence in universities and to develop the potential of potential developers at the university stage. Conduct seminars and lectures with the participation of leading scientists and businessmen working with real tasks in the field of artificial intelligence and, in particular, artificial neural net- works. 7 References 1. Bottou L. et al. Comparison of classifier methods: a case study in handwritten digit recog- nition //Pattern Recognition, 1994. Vol. 2-Conference B: Computer Vision & Image Pro- cessing., Proceedings of the 12th IAPR International. Conference on, vol. 2, 77-82, IEEE, (1994). 2. . LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard and L. D. Jackel: Backpropagation Applied to Handwritten Zip Code Recognition, Neural Computation, 1(4):541-551, Winter (1989) 3. Russakovsky O. et al. Imagenet large scale visual recognition challenge //International Jour- nal of Computer Vision, vol.115, №. 3, 211-252, (2015). 4. Pilato, Giovanni, Sergey A. Yarushev, and Alexey N. Averkin. "Prediction and Detection of User Emotions Based on Neuro-Fuzzy Neural Networks in Social Networks." International Conference on Intelligent Information Technologies for Industry. Springer, Cham, (2018). 5. M erolla, P.A. A million spiking-neuron integrated circuit with a scalable communication network and interface / P.A. M erolla et al. // Science 08 Aug 2014, Vol. 345, Issue 6197, pp. 668-673, (2014). 6. Zaghloul, K.A. A silicon retina that reproduces signals in the optic nerve / K.A. Zaghloul, K. Boahen // Journal of neural engineering, Vol 3, P. 257267, (2006). 7. Wen, B. A Silicon Cochlea with Active Coupling / B. Wen, K. Boahen // IEEE Transactions on biomedical circuits and systems, December 2009, Vol. 3, No. 6, pp. 444-455, (2009). 8. Ranganathan, A. et al. Self-Organization in Artificial Intelligence and the Brain // Georgia Tech online. URL: http://www.cc.gatech.edu/people/home/zkira/research.html/, last ac- cessed 2019/04/1 9. Sabour S., Frosst N., Hinton G. E. Dynamic routing between capsules //Advances in Neural Information Processing Systems, pp. 3859-3869, (2017).