Automated Complex for Aerial Reconnaissance Tasks in Modern Armed Conflicts Pylyp Prystavka [0000-0002-0360-2459], Vladyslav Sorokopud [0000-0002-3256-7031], Artem Chyrkov [0000-0001-6582-8018] and Vladyslav Kovtun 6 [0000-0002-1408-5805] National Aviation University, Kyiv, Ukraine chindakor37@gmail.com, vlad.sorokopud.i@gmail.com, a.chyrkov@nau.edu.ua Abstract. In military combat missions and/or rescue operations intelligence da- ta have a significant role. Last time using unmanned aircraft vehicles (UAV) is an effective way for its obtaining. As practice shows, the main aerial intelli- gence tasks are object detection and object tracking, and these tasks are desira- ble for automation. Existing UAVs either have no admitted task automation functionality at all or have particular functionality for civil purposes. This pub- lication describes automated complex with implemented potentially interesting objects search and the object-of-interest tracking functionality. Keywords: aerial reconnaissance, object detection, object tracking, unmanned aerial vehicle. 1 Introduction In modern armed conflicts, unmanned aerial vehicles (UAVs) have received active use at a tactical level, while their share is increasing annually. This trend is main- tained due to the high efficiency of aerial reconnaissance missions [37]. As noted in [41], one of the basic requirements for cyber intelligence systems is the automation of typical tasks. Moreover, in practice, typical tasks are to search for sus- picious (potentially dangerous) objects of a significant number of classes, and to mon- itor the target(s) object(s). So, automation of such tasks is an important issue. 1.1. Overview of existing UAVs Since the first half of the 2010s, there has been an active introduction of the UAV to solve a wide range of practical problems, primarily the agro-industrial sector, the security sector, as well as specialized dual-use and military systems. Moreover, the analysis of technologies, methods and tools (TMT) used in the UAV is not presented in open scientific publications. This is due to the fact that: 1) some TMT is military equipment or dual-use products, or 2) the vast majority of UHF are produced by pri- vate companies, therefore TMT used in them is a trade secret. So, to understand the current state of the industry, instead of analyzing scientific publications, it is neces- Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attrib- ution 4.0 International (CC BY 4.0) CMiGIN-2019: International Workshop on Conflict Management in Global Information Networks. sary to analyze the open documentation of the UAV, the websites of manufacturers or sellers, relevant specialized exhibitions, and the like. The following is an overview of the UAV present in the Ukrainian market. The most famous battle control system (of which the reconnaissance UAV is a part) developed in Ukraine is Combat Vision [1]. In this system, the aircraft is used for aerial reconnaissance, in particular the mapping of objects of interest on a map. Moreover, video analysis is not automated and is performed by the operator in a com- pletely manual mode. One of the examples of UAV with the implemented functionality of automatic (au- tomated) monitoring, search, identification of objects of interest is UAV Micro C- UAS WARMATE (Poland) [2]. A licensed copy of this complex is available on the Ukrainian market (as noted in [3] and on the websites of news agencies). It is de- signed to defeat the enemy by the method of self-destruction ("aircraft-shell"), that is, not reconnaissance. Other problems - fixed in foreign currency, high price, depend- ence on a foreign manufacturer. In addition, an analysis of specialized exhibitions showed that a licensed copy available on the Ukrainian market has limitations in terms of automatic search for objects. In tactical-level combat control systems, in particular [4]–[6], aircraft are used to shoot video and then watch it by the operator (similar to [1]), or as transport media for solving various technical problems. Ukrainian companies ([7]–[8] and others) are engaged in the sale of various types of UAVs for solving a wide range of tasks, but there is no list of tasks to automate video processing from the camera. The UAV, adopted by the Armed Forces of Ukraine - A1 SM Fury [9], DeViRo Leleka-100 [10] and others - contain LA, which is a video camera carrier, without the possibility of automatic (automated) processing video. "Blowfish A2" (China) [11] is a UAV with the ability to automatically perform certain tasks. Moreover, from the analysis of open sources it follows that the func- tionality of the automatic (automated) search for objects by video is missing. At the end of 2018, an agreement was signed on the supply of the Bayraktar TB2 UAV (Turkey) for the Ukrainian Armed Forces. From the analysis of open sources it follows that the functionality of automatic (automated) search for objects by video is also absent, even in the form of additional modules. An example of a UAV for automatically searching for targets by video is Project Maven (developed by Google by order of the US Department of Defense), which features the use of neural networks, which requires a significant amount of training samples, the formation of which is a separate labor-intensive and, in the case of the military sphere, dangerous task. 1.2. UAV with real-time tracking 1. DJI Phantom 3 Standard DJI Phantom 3 delivers great performance with Follow Me on. The drone uses GPS / GLONASS to accurately determine the position in Follow Me mode. Thanks to this system, the quadrocopter will freeze in the same position in the air, and the cam- era will monitor the object [32]. Follow Me is just one of the interesting features of DJI Phantom 3. From the quad- copter, you will also get smart modes such as Waypoints (a function that records a specific flight path), Point of Interest (the drone automatically rotates around the ob- ject), Home Lock (control carried out in relation to the position of the pilot) and head- ing lock (all flight controls are locked relative to the current heading). 2. 3DR Solo Solo has several intelligent control modes: Follow Me, Orbit, Cable Cam and Selfie. Intelligent computer system allows you to change the angle, distance and perspec- tive during the flight of the drone [33]. The result is a more detailed video that has smooth transitions and movement. 3. 3DR IRIS+ IRIS + has an enhanced Follow Me mode, and all settings can be configured using the tablet. In addition to following the subject, intelligent technology controls the suspension to reduce jitter and take clear photos and videos [34]. One small drawback should be remembered: in the drone there is no way to over- come obstacles. When using the "Follow me" mode, the selected course should be relatively free from high obstacles that the drone could potentially encounter. 4. Hubsan H501S X4 The built-in GPS system allows the quadrocopter to track the object [35]. This function can be enabled when both the quadrocopter and the transmitter are synchro- nized with at least six satellites. This safety measure is necessary for the correct posi- tioning of the quadrocopter. 5. Ehang Ghost Drone 2.0 The Follow Me feature is based on GPS positioning. Ghost 2.0 does not have an obstacle sensor: it is better to use it in an open area. The so-called G BOX is included in the package of delivery; the subject must be worn constantly to ensure more accu- rate tracking [36]. If you set the night mode, it will turn on the LED lights so you can better track it. All the above UAV models have a common drawback, namely a high price and a closed code, which makes it impossible to use these models in search and exploration systems. 1.3. Overview of Existing Computer Vision Techniques Since 2012, convolutional neural networks (CNNs) have reached an practically ac- ceptable level [12]. Their advantages are high search accuracy, low error types I and II, sufficient speed for a wide range of practical tasks when running on the x86_64 hardware in the presence of a GPU. The disadvantages are: 1) the need for a training sample of a significant amount; 2) low speed when running on hardware other than the above; 3) significant time spent on the training phase – e.g. in [13] the declared CNN training time is 14 hours, in [14] is about 240 hours. The disadvantage (1) is solved by methods specific to each specific subject area. The solution to the deficiency (2) is one of the main directions of scientific research in this area; for example, possible ways to increase the speed of CNN are given in [15]– [16], but there is currently no radical way to solve this problem. The results of analysis of other existing classes of methods for automatically searching for objects on video are as follows (taken from [17]). The Template Match- ing methods also require a reference base; in addition, some methods [18]–[19] are not robust to noise, and some [20] do not provide sufficient performance. The ap- proaches based on singular points and the classifier [21] require a training set and have a preliminary training stage. The use of segmentation methods [22]–[26] require manual setting of parameters; in addition, some of them have unacceptably low speed, some do not give acceptable results in noisy video frames or in textured areas. The class of Active Shape methods [27], statistical recognition methods [28], and methods based on imitating the emphasis of a person [29] also need a training sample and have a preliminary training stage. In [30], a description is given of an experimental sample of an automated target search system using UAVs, which provides the automatic generation of a list of sus- picious objects, and the selection of objects (s) of interest. The core of this system is actually the method [31] for automatically searching for suspicious objects on video from a UAV camera, which is designed to search for objects that are distinguished against the background and are not often found, do not require a training sample for training (self-learning on the first frames of the video) [31]. 1.4. Formulation of the problem Based on the foregoing, the purpose of this publication is to develop an automated complex based on an experimental sample [30] that would implement the search for potentially interesting objects, as well as the object-of-interest tracking. 2. Automated Complex Description System requirements. AU, being developed, should have the following functionali- ty: • flight in research mode; • synchronization with onboard autopilot; • search for potentially suspicious objects in automatic mode; • selection by the operator of objects of interest of an automatically generated list; • tracking objects selected by the operator Since the AS presented in this publication is based on AS [30], a brief description is given below with [30]. Hardware Technology The hardware of the AS consists of UAVs with target equipment on board, a ground station (NS) and a communication channel between them (Fig. 1). GPS Autopilot Camera Raspberry Pi Client app UAV Fig. 1. Composition of an automated system The target equipment on board the UAV in case of a problem, it is decided, is a video camera and a device for processing data. The last one is a single-board comput- er (OK). In the publication [37] for the class of similar tasks, it is recommended to use OK Raspberry Pi 3 Model B or DragonBoard 410c. It is proposed to use a laptop with the following minimum characteristics as an NS for processing data from UAVs and controlling flight mission or control: Intel Core i5 processor, 8 GB of RAM. The functional diagram of the speakers is shown in Fig. 2. Coordinates Wifi GPS Autopilot Camera Ethernet Sync/Control Client app Video Raspberry Pi Radio UAV Fig. 2 Functional diagram of the speaker Establishing high-quality communication between UAVs and emergency situations is a difficult problem, since all channels can be jammed or intercepted. The choice of the type of communication channel is a separate task that is beyond the scope of this publication. Search for potentially dangerous objects. To automatically search for suspicious (potentially dangerous) objects on board the UAV, a software implementation of the method is used [31]. The coordinates of suspicious (potentially dangerous) objects are determined on the basis of [38] or [39]. The structure of an automated system. The principle of the system as a whole is as follows: the client part (UAV with OK) processes the streaming video from the camera, determines suspicious objects, receives telemetry parameters - GPS coordi- nates, yaw angles, pitch, roll of the UAV - from the autopilot subsystem and sends it to the server (NA). The server part allows the operator to view the objects that were received from the UAV, the operator has the opportunity to select a specific object and send information about it back to the UAV, after which the UAV changes the flight mode: it switches to the search mode selected by the operator of the object. When the UAV finds it selected it begins to follow it [30]. The schematic principle of the system is shown in Fig. 3. · Send objects · Send UAVcoordinates Autopilot Camera · Get UAV coordinates · Get image · Detect objects · Track object Client app Raspberry Pi UAV · Send target object · Set flight mode Fig. 3. The structure of the complex: the relationship between the elements Thus, the tasks of the client side of the AS are: processing video from a UAV, searching for suspicious objects, searching for a target, tracking the target. Tasks of the AS server side: displaying suspicious objects, organizing the possibilities of se- lecting a specific object, changing the parameters of the algorithm for searching for suspicious objects, determining the coordinates of the UAV camera’s viewing area, displaying a UAV flight map [30]. In order to protect data from unauthorized viewing, it makes sense to encrypt it. Data encryption is also a separate task that deserves a separate publication. User interface. One of the main tasks of the server part (NS) is the interaction with the user (operator). For its organization, a graphical interface was developed as part of the server application. The structure of user interaction and the graphical interface is shown in Fig.4. Main window Objects Settings Selected object Operations View objects Select object User Main window Objects Selected object Settings Settings Objects Selected object Operations Operations View selected object Set settings Detect object coordinates Send settings to UAV Send object to UAV Fig. 4. The structure of user interaction and graphical interface [30] The procedure for working with the system is as follows: 1. Set up a Wi-Fi communication channel. 2. Run the server application on the emergency. Find out the IP address of the emergency. 3. Run the client application on the UAV. At startup, specify the IP address of the emergency. 4. Launch the UAV along the necessary path. 5. During the flight, view suspicious objects that the AS sends from the UAV to the emergency at a periodicity. If necessary, change the settings (algorithm parame- ters). 6. If an object of interest appears in the list, select it and instruct the UAV to per- form automatic actions in relation to it (for example, put the UAV in automatic track- ing mode) [30]. 7. In the mode of automatic tracking of the target, monitor the emergency process. In case of loss of the target object from focus, re-capture it and restart tracking mode. 8. After completing the task, landing the UAV, shutting down the client and server applications. Automatic UAV control. A single-board computer can send commands to the au- topilot in automatic mode, for this the MavSDK library is used. In fig. 5 shows the communication scheme. Autopilot MavSDK Mavlink Client app Fig. 5. Communication scheme Target tracking. The developed system implements an algorithm for searching and tracking targets. When the operator selects the object, the UAV returns to the coordinates of this object and starts the search algorithm, if the object was found, the UAV begins the tracking process. In fig. 6 shows an example of target tracking. Fig. 6. An example of target tracking The choice of the mathematical method of tracking is carried out taking into ac- count the features of the camera on board the UAV and the features of the target ob- jects from the point of view of computer vision. An example of a comparative analy- sis of trackers by the specified criteria is given in [40]. The test results of the developed AS at the test site showed high-quality results at a low speed of movement of the target object and a relatively simple background. 3. Conclusions Based on the experimental sample [30], an automated complex is implemented that provides the automatic generation of a list of suspicious (potentially dangerous) ob- jects, the selection of an object of interest from it, and tracking of the selected object. The system was tested on test objects at the test site. The test results are positive at a low speed of movement of the target object and a relatively simple background. Further research can be aimed at calibrating the developed system by choosing the best tracking method, optimizing the UAV control process. References 1. ComBat Vision, https://combat.vision/, last accessed 2019/10/16. 2. Warmate | WB Electronics, http://wb.com.pl/warmate-en/?lang=en, last accessed 2019/10/16. 3. WB Electronics Warmate — Wikipedia, https://en.wikipedia.org/wiki/WB_Electronics_Warmate, last accessed 2019/10/16. 4. Defence Software | Command and Control | Interoperability, https://systematic.com/defence/, last accessed 2019/10/16. 5. Warfighter Information Network-Tactical (WIN-T) - General Dynamics Mission Systems, https://gdmissionsystems.com/c4isr/warfighter-information-network-tactical-win-t/, last accessed 2019/10/16. 6. Force XXI Battle Command Brigade and Below (FBCB2), https://pdfs.semanticscholar.org/cb40/aeb3ca16cec0a0008c21409ebb9ff008084c.pdf, last accessed 2019/10/16. 7. Matrix UAV, http://muav.com.ua/, last accessed 2019/10/16. 8. Long-range multirotor UAVs | Tethered drones | Aeromagnetic survey, https://umt.aero/, last accessed 2019/10/16. 9. A1 SM “Furia”, https://www.facebook.com/media/set/?set=a.1620187274919672.1073741836.143418115 0186953&type=3, last accessed 2019/10/16. (in Ukrainian) 10. Unmanned Aerial System DEVIRO “Leleka-100”, http://deviro.ua/leleka-100, last ac- cessed 2019/10/16. (in Ukrainian) 11. Blowfish A2, http://ziyanuav.com/blowfish2.html, last accessed 2019/10/16. 12. Krizhevsky, A., Sutskever, I., Hinton, G.E.: “Imagenet classification with deep convolu- tional neural networks”. In: Advances in neural information processing systems, NIPS, Lake Tahoe, Nevada, USA, 2012, pp. 1097-1105. doi:10.1145/3065386. 13. Majumder, S., Balaji, N., Brey, K., Fu, W., Menzies, T.: “500+ Times Faster Than Deep Learning (A Case Study Exploring Faster Methods for Text Mining StackOverflow)”, May 2018 [online]. https://arxiv.org/pdf/1802.05319.pdf 14. Gu, X., Zhang, H., Zhang, D., Kim, S.: “DeepAPI learning”. In: Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering, 2016, pp. 631-642. 15. Zhang, X., Zou, J., He, K., Sun, J.: “Accelerating Very Deep Convolutional Networks for Classification and Detection”, Nov 2015 [online]. https://arxiv.org/abs/1505.06798 16. Accelerating Convolutional Neural Networks on Raspberry Pi, http://cv- tricks.com/artificial-intelligence/deep-learning/accelerating-convolutional-neural- networks-on-raspberry-pi/, las accessed 2019/10/16. 17. Prystavka, P., His, D., Chyrkov, A.: “Technique for Automated Target Object Search in Video Stream from UAV in Post-Processing Mode”. Ukrainian Information Security Re- search Journal, 2(21), 97–103 (2019). (in Ukrainian) 18. Lowe, D.G.: “Object recognition from local scale-invariant features”, In: Proc. of the 7th IEEE International Conference on Computer Vision, Greece, September 1999. https://dx.doi.org/10.1109%2FICCV.1999.790410. 19. Bay, H., Tuytelaars, T., Gool, L.V.: “SURF: Speeded Up Robust Features”, Computer Vi- sion – ECCV 2006. Lecture Notes in Computer Science, vol 3951, 2006. 20. Alsaade, F.: “Fast and Accurate Template Matching Algorithm Based on Image Pyramid and Sum of Absolute Difference Similarity Measure”, Research Journal of Information Technology, vol. 4, issue 4, pp 204-211, 2012. http://dx.doi.org/10.3923/rjit.2012.204.211. 21. Viola, P., Jones, M., “Rapid object detection using a boosted cascade of simple features”. In: Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001, USA, 2001. doi:10.1109/CVPR.2001.990517 22. Otsu, N., “A Threshold Selection Method from Gray-Level Histograms”. In: IEEE Trans- actions on Systems, Man and Cybernetics, vol 9, issue 1, pp 62-66, 1979. 23. Canny, J., “A Computational Approach to Edge Detection”, In: IEEE Transactions on Pat- tern Analysis and Machine Intelligence, volume PAMI-8, issue 6, pp 679-698, Nov. 1986. https://doi.org/10.1109/TPAMI.1986.4767851. 24. Lloyd, S.P.: “Least squares quantization in PCM”, In: IEEE Transactions on Information Theory, vol. 28, issue 2, pp 129-137, March 1982. 25. Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: “DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Con- nected CRFs”, Submitted on June 2016, last revised on May 2017. 26. Papandreou, G., Chen, L.-C., Murphy, K., Yuille, A.L.: “Weakly- and Semi-Supervised Learning of a DCNN for Semantic Image Segmentation”, Submitted on Feb 2015, last re- vised on Oct 2015. https://arxiv.org/abs/1502.02734. 27. Cootes, T.F., Taylor, C.J., Cooper, D.H., Graham, J.: “Active Shape Models – Their Train- ing and Application”, Computer Vision and Image Understanding, vol 61, issue 1, pp 38- 59, January 1995. https://doi.org/10.1006/cviu.1995.1004. 28. Gorelik, A., Skripkin, V.: Object Recognition Methods: A Studentbook. 2nd edn. Higher School, Moscow, USSR, (1984). (in Russian) 29. Gotovac, S., Papić, V., Marušić, Ž.: “Analysis of saliency object detection algorithms for search and rescue operations”, In: 2016 24th International Conference on Software, Tele- communications and Computer Networks (SoftCOM), Split, Croatia, September 2016. 30. Prystavka, P., Sorokopud, V., Chyrkov, A.: “Experimental Version of Automated System for Suspicious Objects Search on Video Stream from Unmanned Aircraft”. Weapons Sys- tems and Military Equipment, 2(50), 26–32 (2017). (in Ukrainian) 31. Chyrkov, A., Prystavka, P.: “Suspicious Object Search in Airborne Camera Video Stream”. In: Hu Z. et al. (eds) Advances in Computer Science for Engineering and Educa- tion. ICCSEEA 2018. Advances in Intelligent Systems and Computing, vol 754. Springer, Cham, Switzerland, 340–348 (2018). doi:10.1007/978-3-319-91008-6_34 32. Phantom | DroneUA, http://store.drone.ua/phantom/?gclid=Cj0KCQjwrrXtBRCKARIsAMbU6bHSc1ZuyG0bj mX2cK7lmPweWAnmqHW9j_W1ejf1QIwQ2j9St-173-gaAtm-EALw_wcB, last accessed 2019/10/16 33. Quadrocopter 3DR Solo + gimbal do Hero 3/4, https://led-expert.in.ua/p298730905- kvadrokopter-3dr-solo.html, last accessed 2019/10/16 34. 3DR IRIS+, http://quadrocopter.ua/quadrocopters_copters/3DR-IRIS 35. Hubsan H501S High Edition, https://hubsan.in.ua/product/hubsan-h501s-pro- x4/?gclid=Cj0KCQjwrrXtBRCKARIsAMbU6bFi8WuhgyuEYEKa5obtSG77hlvDiJdXrEk 75pPgH7nc18eO2KHFMmgaAiFwEALw_wcB, last accessed 2019/10/16 36. Ehang GhostDrone 2.0, https://ek.ua/EHANG-GHOSTDRONE-2-0.htm, last accessed 2019/10/16 37. Shevchenko, A., “Comparative Analysis of Microcomputers for Data Processing”. In: MPZIS 2016, Dnipropetrovsk, Ukraine (2016). (in Ukrainian) 38. Nichikov, E., Chyrkov, A.: “Information Technology of UAV Camera Field of Calcula- tion”. Problems of Informatizzation and Control, 4(52), 106–112 (2015). (in Ukrainian) 39. Buryi, P., Prystavka, P., Sushko, V.: “Automatic Definition the Field of View of Camera of Unmanned Aerial Vehicle”. Science-Based Technologies, 2(30), 151–155 (2016). 40. Chyrkov, A.: “Comparative Analysis of Object Tracking Methods in Video Stream from UAV Camera”. Problems of Informatizzation and Control, 1(53), 78–82 (2016). (in Ukrainian) 41. Hryshchuk, R., Danyk, Yu.: Fundamentals of Cybersecurity. ZhNAEU, Zhytomyr, Ukraine (2016). (in Ukrainian)