In military combat missions and/or rescue operations intelligence data have a significant role. Last time using unmanned aircraft vehicles (UAV) is an effective way for its obtaining. As practice shows, the main aerial intelligence tasks are object detection and object tracking, and these tasks are desirable for automation. Existing UAVs either have no admitted task automation functionality at all or have particular functionality for civil purposes. This publication describes automated complex with implemented potentially interesting objects search and the object-of-interest tracking functionality.
In modern armed conflicts, unmanned aerial vehicles (UAVs) have received active use at a tactical level, while their share is increasing annually. This trend is maintained due to the high efficiency of aerial reconnaissance missions [37].
As noted in [41], one of the basic requirements for cyber intelligence systems is the automation of typical tasks. Moreover, in practice, typical tasks are to search for suspicious (potentially dangerous) objects of a significant number of classes, and to monitor the target(s) object(s). So, automation of such tasks is an important issue.
Since the first half of the 2010s, there has been an active introduction of the UAV to solve a wide range of practical problems, primarily the agro-industrial sector, the security sector, as well as specialized dual-use and military systems. Moreover, the analysis of technologies, methods and tools (TMT) used in the UAV is not presented in open scientific publications. This is due to the fact that: 1) some TMT is military equipment or dual-use products, or 2) the vast majority of UHF are produced by private companies, therefore TMT used in them is a trade secret. So, to understand the current state of the industry, instead of analyzing scientific publications, it is necessary to analyze the open documentation of the UAV, the websites of manufacturers or sellers, relevant specialized exhibitions, and the like. The following is an overview of the UAV present in the Ukrainian market.
The most famous battle control system (of which the reconnaissance UAV is a
part) developed in Ukraine is Combat Vision [
One of the examples of UAV with the implemented functionality of automatic
(automated) monitoring, search, identification of objects of interest is UAV Micro
CUAS WARMATE (Poland) [
In tactical-level combat control systems, in particular [4]–[6], aircraft are used to
shoot video and then watch it by the operator (similar to [
Ukrainian companies ([7]–[8] and others) are engaged in the sale of various types of UAVs for solving a wide range of tasks, but there is no list of tasks to automate video processing from the camera.
The UAV, adopted by the Armed Forces of Ukraine - A1 SM Fury [9], DeViRo Leleka-100 [10] and others - contain LA, which is a video camera carrier, without the possibility of automatic (automated) processing video.
"Blowfish A2" (China) [11] is a UAV with the ability to automatically perform certain tasks. Moreover, from the analysis of open sources it follows that the functionality of the automatic (automated) search for objects by video is missing.
At the end of 2018, an agreement was signed on the supply of the Bayraktar TB2 UAV (Turkey) for the Ukrainian Armed Forces. From the analysis of open sources it follows that the functionality of automatic (automated) search for objects by video is also absent, even in the form of additional modules.
An example of a UAV for automatically searching for targets by video is Project Maven (developed by Google by order of the US Department of Defense), which features the use of neural networks, which requires a significant amount of training samples, the formation of which is a separate labor-intensive and, in the case of the military sphere, dangerous task.
1. DJI Phantom 3 Standard
DJI Phantom 3 delivers great performance with Follow Me on. The drone uses GPS / GLONASS to accurately determine the position in Follow Me mode. Thanks to this system, the quadrocopter will freeze in the same position in the air, and the camera will monitor the object [32].
Follow Me is just one of the interesting features of DJI Phantom 3. From the quadcopter, you will also get smart modes such as Waypoints (a function that records a specific flight path), Point of Interest (the drone automatically rotates around the object), Home Lock (control carried out in relation to the position of the pilot) and heading lock (all flight controls are locked relative to the current heading).
2. 3DR Solo
Solo has several intelligent control modes: Follow Me, Orbit, Cable Cam and Selfie.
Intelligent computer system allows you to change the angle, distance and perspective during the flight of the drone [33]. The result is a more detailed video that has smooth transitions and movement.
3. 3DR IRIS+
IRIS + has an enhanced Follow Me mode, and all settings can be configured using the tablet. In addition to following the subject, intelligent technology controls the suspension to reduce jitter and take clear photos and videos [34].
One small drawback should be remembered: in the drone there is no way to overcome obstacles. When using the "Follow me" mode, the selected course should be relatively free from high obstacles that the drone could potentially encounter.
4. Hubsan H501S X4
The built-in GPS system allows the quadrocopter to track the object [35]. This function can be enabled when both the quadrocopter and the transmitter are synchronized with at least six satellites. This safety measure is necessary for the correct positioning of the quadrocopter.
5. Ehang Ghost Drone 2.0
The Follow Me feature is based on GPS positioning. Ghost 2.0 does not have an obstacle sensor: it is better to use it in an open area. The so-called G BOX is included in the package of delivery; the subject must be worn constantly to ensure more accurate tracking [36].
If you set the night mode, it will turn on the LED lights so you can better track it.
All the above UAV models have a common drawback, namely a high price and a closed code, which makes it impossible to use these models in search and exploration systems.
Since 2012, convolutional neural networks (CNNs) have reached an practically acceptable level [12]. Their advantages are high search accuracy, low error types I and II, sufficient speed for a wide range of practical tasks when running on the x86_64 hardware in the presence of a GPU. The disadvantages are: 1) the need for a training sample of a significant amount; 2) low speed when running on hardware other than the above; 3) significant time spent on the training phase – e.g. in [13] the declared CNN training time is 14 hours, in [14] is about 240 hours.
The disadvantage (1) is solved by methods specific to each specific subject area. The solution to the deficiency (2) is one of the main directions of scientific research in this area; for example, possible ways to increase the speed of CNN are given in [15]– [16], but there is currently no radical way to solve this problem.
The results of analysis of other existing classes of methods for automatically searching for objects on video are as follows (taken from [17]). The Template Matching methods also require a reference base; in addition, some methods [18]–[19] are not robust to noise, and some [20] do not provide sufficient performance. The approaches based on singular points and the classifier [21] require a training set and have a preliminary training stage. The use of segmentation methods [22]–[26] require manual setting of parameters; in addition, some of them have unacceptably low speed, some do not give acceptable results in noisy video frames or in textured areas. The class of Active Shape methods [27], statistical recognition methods [28], and methods based on imitating the emphasis of a person [29] also need a training sample and have a preliminary training stage.
In [30], a description is given of an experimental sample of an automated target search system using UAVs, which provides the automatic generation of a list of suspicious objects, and the selection of objects (s) of interest. The core of this system is actually the method [31] for automatically searching for suspicious objects on video from a UAV camera, which is designed to search for objects that are distinguished against the background and are not often found, do not require a training sample for training (self-learning on the first frames of the video) [31].
Based on the foregoing, the purpose of this publication is to develop an automated complex based on an experimental sample [30] that would implement the search for potentially interesting objects, as well as the object-of-interest tracking. 2.
System requirements. AU, being developed, should have the following functionality: • flight in research mode; • synchronization with onboard autopilot; • search for potentially suspicious objects in automatic mode; • selection by the operator of objects of interest of an automatically generated list; • tracking objects selected by the operator
Since the AS presented in this publication is based on AS [30], a brief description is given below with [30].
Hardware Technology The hardware of the AS consists of UAVs with target equipment on board, a ground station (NS) and a communication channel between them (Fig. 1).
UAV UAV
GPS
The target equipment on board the UAV in case of a problem, it is decided, is a video camera and a device for processing data. The last one is a single-board computer (OK). In the publication [37] for the class of similar tasks, it is recommended to use OK Raspberry Pi 3 Model B or DragonBoard 410c.
It is proposed to use a laptop with the following minimum characteristics as an NS for processing data from UAVs and controlling flight mission or control: Intel Core i5 processor, 8 GB of RAM.
The functional diagram of the speakers is shown in Fig. 2.
Coordinates
Autopilot Sync/Control Raspberry Pi
Camera Video
Wifi Ethernet Radio
Client app
Establishing high-quality communication between UAVs and emergency situations is a difficult problem, since all channels can be jammed or intercepted. The choice of the type of communication channel is a separate task that is beyond the scope of this publication.
(potentially dangerous) objects on board the UAV, a software implementation of the method is used [31].
the basis of [38] or [39].
The structure of an automated system. The principle of the system as a whole is as follows: the client part (UAV with OK) processes the streaming video from the camera, determines suspicious objects, receives telemetry parameters - GPS coordinates, yaw angles, pitch, roll of the UAV - from the autopilot subsystem and sends it to the server (NA). The server part allows the operator to view the objects that were received from the UAV, the operator has the opportunity to select a specific object and send information about it back to the UAV, after which the UAV changes the flight mode: it switches to the search mode selected by the operator of the object. When the UAV finds it selected it begins to follow it [30]. The schematic principle of the system is shown in Fig. 3.
· Send objects · Send UAVcoordinates
Autopilot Thus, the tasks of the client side of the AS are: processing video from a UAV, searching for suspicious objects, searching for a target, tracking the target. Tasks of the AS server side: displaying suspicious objects, organizing the possibilities of selecting a specific object, changing the parameters of the algorithm for searching for suspicious objects, determining the coordinates of the UAV camera’s viewing area, displaying a UAV flight map [30].
In order to protect data from unauthorized viewing, it makes sense to encrypt it. Data encryption is also a separate task that deserves a separate publication.
User interface. One of the main tasks of the server part (NS) is the interaction with the user (operator). For its organization, a graphical interface was developed as part of the server application. The structure of user interaction and the graphical interface is shown in Fig.4.
User Operations View selected object Detect object coordinates Send object to UAV Objects
Selected object
Settings
Settings
Objects
Selected object Main window
Operations View objects Select object Objects
Settings Selected object Main window
Operations Set settings
Send settings to UAV
Automatic UAV control. A single-board computer can send commands to the autopilot in automatic mode, for this the MavSDK library is used.
In fig. 5 shows the communication scheme.
MavSDK
Mavlink
Fig. 5. Communication scheme
Target tracking. The developed system implements an algorithm for searching and tracking targets. When the operator selects the object, the UAV returns to the coordinates of this object and starts the search algorithm, if the object was found, the UAV begins the tracking process. In fig. 6 shows an example of target tracking.
The choice of the mathematical method of tracking is carried out taking into account the features of the camera on board the UAV and the features of the target objects from the point of view of computer vision. An example of a comparative analysis of trackers by the specified criteria is given in [40].
The test results of the developed AS at the test site showed high-quality results at a low speed of movement of the target object and a relatively simple background. 3.
Based on the experimental sample [30], an automated complex is implemented that provides the automatic generation of a list of suspicious (potentially dangerous) objects, the selection of an object of interest from it, and tracking of the selected object. The system was tested on test objects at the test site. The test results are positive at a low speed of movement of the target object and a relatively simple background. Further research can be aimed at calibrating the developed system by choosing the best tracking method, optimizing the UAV control process. 3. WB Electronics Warmate — Wikipedia,
https://en.wikipedia.org/wiki/WB_Electronics_Warmate, last accessed 2019/10/16. 4. Defence Software | Command and Control | Interoperability, https://systematic.com/defence/, last accessed 2019/10/16. 5. Warfighter Information Network-Tactical (WIN-T) - General Dynamics Mission Systems, https://gdmissionsystems.com/c4isr/warfighter-information-network-tactical-win-t/, last accessed 2019/10/16. 6. Force XXI Battle Command Brigade and Below (FBCB2), https://pdfs.semanticscholar.org/cb40/aeb3ca16cec0a0008c21409ebb9ff008084c.pdf, last accessed 2019/10/16. 7. Matrix UAV, http://muav.com.ua/, last accessed 2019/10/16. 8. Long-range multirotor UAVs | Tethered drones | Aeromagnetic survey, https://umt.aero/, last accessed 2019/10/16. 9. A1 SM “Furia”, https://www.facebook.com/media/set/?set=a.1620187274919672.1073741836.143418115 0186953&type=3, last accessed 2019/10/16. (in Ukrainian) 10. Unmanned Aerial System DEVIRO “Leleka-100”, http://deviro.ua/leleka-100, last accessed 2019/10/16. (in Ukrainian) 11. Blowfish A2, http://ziyanuav.com/blowfish2.html, last accessed 2019/10/16. 12. Krizhevsky, A., Sutskever, I., Hinton, G.E.: “Imagenet classification with deep convolutional neural networks”. In: Advances in neural information processing systems, NIPS, Lake Tahoe, Nevada, USA, 2012, pp. 1097-1105. doi:10.1145/3065386. 13. Majumder, S., Balaji, N., Brey, K., Fu, W., Menzies, T.: “500+ Times Faster Than Deep Learning (A Case Study Exploring Faster Methods for Text Mining StackOverflow)”, May 2018 [online]. https://arxiv.org/pdf/1802.05319.pdf 14. Gu, X., Zhang, H., Zhang, D., Kim, S.: “DeepAPI learning”. In: Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering, 2016, pp. 631-642. 15. Zhang, X., Zou, J., He, K., Sun, J.: “Accelerating Very Deep Convolutional Networks for
Classification and Detection”, Nov 2015 [online]. https://arxiv.org/abs/1505.06798 16. Accelerating Convolutional Neural Networks on Raspberry Pi, http://cvtricks.com/artificial-intelligence/deep-learning/accelerating-convolutional-neuralnetworks-on-raspberry-pi/, las accessed 2019/10/16. 17. Prystavka, P., His, D., Chyrkov, A.: “Technique for Automated Target Object Search in Video Stream from UAV in Post-Processing Mode”. Ukrainian Information Security Research Journal, 2(21), 97–103 (2019). (in Ukrainian) 18. Lowe, D.G.: “Object recognition from local scale-invariant features”, In: Proc. of the 7th IEEE International Conference on Computer Vision, Greece, September 1999. https://dx.doi.org/10.1109%2FICCV.1999.790410. 19. Bay, H., Tuytelaars, T., Gool, L.V.: “SURF: Speeded Up Robust Features”, Computer Vision – ECCV 2006. Lecture Notes in Computer Science, vol 3951, 2006. 20. Alsaade, F.: “Fast and Accurate Template Matching Algorithm Based on Image Pyramid and Sum of Absolute Difference Similarity Measure”, Research Journal of Information Technology, vol. 4, issue 4, pp 204-211, 2012. http://dx.doi.org/10.3923/rjit.2012.204.211. 21. Viola, P., Jones, M., “Rapid object detection using a boosted cascade of simple features”.
In: Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001, USA, 2001. doi:10.1109/CVPR.2001.990517 22. Otsu, N., “A Threshold Selection Method from Gray-Level Histograms”. In: IEEE Transactions on Systems, Man and Cybernetics, vol 9, issue 1, pp 62-66, 1979. 23. Canny, J., “A Computational Approach to Edge Detection”, In: IEEE Transactions on Pattern Analysis and Machine Intelligence, volume PAMI-8, issue 6, pp 679-698, Nov. 1986. https://doi.org/10.1109/TPAMI.1986.4767851. 24. Lloyd, S.P.: “Least squares quantization in PCM”, In: IEEE Transactions on Information
Theory, vol. 28, issue 2, pp 129-137, March 1982. 25. Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: “DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs”, Submitted on June 2016, last revised on May 2017. 26. Papandreou, G., Chen, L.-C., Murphy, K., Yuille, A.L.: “Weakly- and Semi-Supervised Learning of a DCNN for Semantic Image Segmentation”, Submitted on Feb 2015, last revised on Oct 2015. https://arxiv.org/abs/1502.02734. 27. Cootes, T.F., Taylor, C.J., Cooper, D.H., Graham, J.: “Active Shape Models – Their Training and Application”, Computer Vision and Image Understanding, vol 61, issue 1, pp 3859, January 1995. https://doi.org/10.1006/cviu.1995.1004. 28. Gorelik, A., Skripkin, V.: Object Recognition Methods: A Studentbook. 2nd edn. Higher
School, Moscow, USSR, (1984). (in Russian) 29. Gotovac, S., Papić, V., Marušić, Ž.: “Analysis of saliency object detection algorithms for search and rescue operations”, In: 2016 24th International Conference on Software, Telecommunications and Computer Networks (SoftCOM), Split, Croatia, September 2016. 30. Prystavka, P., Sorokopud, V., Chyrkov, A.: “Experimental Version of Automated System for Suspicious Objects Search on Video Stream from Unmanned Aircraft”. Weapons Systems and Military Equipment, 2(50), 26–32 (2017). (in Ukrainian) 31. Chyrkov, A., Prystavka, P.: “Suspicious Object Search in Airborne Camera Video Stream”. In: Hu Z. et al. (eds) Advances in Computer Science for Engineering and Education. ICCSEEA 2018. Advances in Intelligent Systems and Computing, vol 754. Springer, Cham, Switzerland, 340–348 (2018). doi:10.1007/978-3-319-91008-6_34 32. Phantom | DroneUA, http://store.drone.ua/phantom/?gclid=Cj0KCQjwrrXtBRCKARIsAMbU6bHSc1ZuyG0bj mX2cK7lmPweWAnmqHW9j_W1ejf1QIwQ2j9St-173-gaAtm-EALw_wcB, last accessed 2019/10/16 33. Quadrocopter 3DR Solo + gimbal do Hero 3/4, https://led-expert.in.ua/p298730905kvadrokopter-3dr-solo.html, last accessed 2019/10/16 34. 3DR IRIS+, http://quadrocopter.ua/quadrocopters_copters/3DR-IRIS 35. Hubsan H501S High Edition, https://hubsan.in.ua/product/hubsan-h501s-prox4/?gclid=Cj0KCQjwrrXtBRCKARIsAMbU6bFi8WuhgyuEYEKa5obtSG77hlvDiJdXrEk 75pPgH7nc18eO2KHFMmgaAiFwEALw_wcB, last accessed 2019/10/16 36. Ehang GhostDrone 2.0, https://ek.ua/EHANG-GHOSTDRONE-2-0.htm, last accessed 2019/10/16 37. Shevchenko, A., “Comparative Analysis of Microcomputers for Data Processing”. In:
MPZIS 2016, Dnipropetrovsk, Ukraine (2016). (in Ukrainian) 38. Nichikov, E., Chyrkov, A.: “Information Technology of UAV Camera Field of Calculation”. Problems of Informatizzation and Control, 4(52), 106–112 (2015). (in Ukrainian) 39. Buryi, P., Prystavka, P., Sushko, V.: “Automatic Definition the Field of View of Camera of Unmanned Aerial Vehicle”. Science-Based Technologies, 2(30), 151–155 (2016). 40. Chyrkov, A.: “Comparative Analysis of Object Tracking Methods in Video Stream from UAV Camera”. Problems of Informatizzation and Control, 1(53), 78–82 (2016). (in Ukrainian) 41. Hryshchuk, R., Danyk, Yu.: Fundamentals of Cybersecurity. ZhNAEU, Zhytomyr, Ukraine (2016). (in Ukrainian)