The paper considers the relevance of using a single software package for processing images made by unmanned aerial vehicles and coordinating the actions of search participants during operations to search for people and vehicles in the natural environment. An approach to the creation of such a package is proposed, its architecture is developed, the technologies used in the development and the main technical solutions are described. An algorithm for processing data for a search operation is developed and implemented. The package core, which recognizes search objects on aerial photographs, is built on a convolutional artificial neural network YOLOv5. The characteristics of the network operation depending on the structure of the training sample and the network parameters are studied. The experimental operation of the package by one division of the crisis management center of the Ministry of Emergency Situations of Russia in Bryansk region is described.
Search and rescue operations are one of the important activities of the Ministry of the Russian ORCID: 0000-0003-3584-9066
(D. Bulatitskiy); 0000-0001-8683-8629 (A. Selifontov); 0000-0001-6941-5595 (S. Shevchenko); ️©
2021 Copyright for this paper by its authors.
The traditional way to search in the natural environment is to scour the area. The search participants examine the area directly, visually, while going on foot in a coordinated manner at such a distance from each other as to ensure overlap of the areas by each of the participants and prevent unexplored areas in the scoured part of the search area. This distance strongly depends on the terrain and vegetation features. So let us assume that a person, when scouring a forest, meadow or field, walks at a speed of 3 km/h and provides visual control of a 10 m strip. Then the speed of scouring by one person will be 0.03 km2/h. This means that a square park with a side of 3 km will be scoured by one person for 300 hours (almost two weeks of continuous searches) or during 24 hours of continuous scouring 12 people will be able to do this. For scouring of a 10 km*10 km area per 24 hours 140 people will be necessary. At the same time, both the speed of a person's movement and the width of the strip that he controls are chosen quite optimistically.
Naturally, scouring is not very effective way of searching, so additional means are used (working
dogs, thermal imagers, bearing on a mobile phone, etc.) and field activities (for example, giving sound
and light signals in anticipation of a response from the lost person), as well as limiting the search area
and planning the sequence of examination of zones within this area [
With the advent of aviation, aircrafts, and later helicopters, also began to be used for search operations. But the use of traditional aviation is complicated by several circumstances.
Firstly, the cost of flights is so high that it is possible only when searching for large groups of people, very important persons and cargo, or other special cases.
Secondly, the classical scheme of such searches involves a visual inspection of the explored space by the pilot or passengers of the aircraft. At the same time, the height and flight speed of a helicopter or even more of an airplane do not allow to see the terrain in sufficient detail. Also, with a low degree of confidence that the search participant saw something similar to the search object or its traces from the air, the probability of returning to this place is extremely low due to the complexity of maneuvering.
Thus, we can say that the use of traditional aviation is effective only when searching a large area for clearly visible objects, which can include vehicles in distress, especially in cases of fire; people in bright clothes in open spaces, as well as people who are waiting for rescue and are able to identify themselves with special signs and means (beacons, bonfires, shots from a flare gun, etc.).
Equipping search helicopters and aircrafts with photo and video recording devices only partially removes some of the indicated difficulties, allowing operators on board or after returning to the base to study the footage more closely.
The appearance of unmanned aerial vehicles (UAVs) allowed not only to reduce the cost of flight
dramatically, but also to reduce the altitude and speed of flight, as well as to increase maneuverability.
The use of high-resolution cameras allows to see the explored surface in all details. Unmanned aircraft
quickly took a confident position in many sectors of the national economy: in cartography, in agriculture
[
There is information in the media about the development of UAV-based human search systems by
the Russian companies Beeline [
We can consider two variants for the use of unmanned aerial vehicles when searching for people in the natural environment. The first implies a permanent communication channel between the device and the operator. The operator controls the device, performing a planned search route as part of a flight of the search zone, and receives video from on-board cameras in real time. If the search object is assumed of being detected, the operator returns to the place of detection and examines it more closely. The second variant involves an autonomous UAV flight along a given search route with video and photography of the area. After the UAV returns to the base, the footage is transferred to experts for analysis.
Both of these variants can significantly improve the efficiency of the search operation, but they are also not without significant drawbacks or problems.
The first problem is related to the complexity of analyzing photo and video materials. In the first variant, the need to explore a large area requires long and high concentration of attention from the UAV operator. When the operator is tired, the attention is not so good and the probability of missing search objects or their traces increases. For example, when using a helicopter-type UAV (quadcopter), the examination of 1 km2 area takes about half an hour. Depending on the model of the device, it can continue the flight or needs to be recharged or refueled. But one way or another, it will take several hours to examine an area of several square kilometers, during which it is almost impossible to maintain the operator's concentration at the desired level. In the second variant, UAVs with greater autonomy are usually used. They fly over a fairly large area, taking photographs from once every few seconds to several times per second, depending on the altitude of the flight and other factors, and after 1-3 hours they return, bringing several thousand or even several tens of thousands of rather large photographs. It takes either too much time or a sufficiently large team of experts to process this material.
The second problem is related to the coordination of actions between the participants of search and rescue operations. Usually UAV operator and experts in the analysis of photographs are only a small part of the people involved in the operation. Experts are also required to collect additional information and interview witnesses; members of search teams operating in the area; drivers and pilots of vehicles for evacuation; medical personnel for first aid.
Even before the start of the active search operation, it is necessary to inform its participants about a new search, its goals (search objects) and their signs. If any traces or people similar to the search objects are detected from the air (with the help of both manned and unmanned aircrafts), the coordinator sends some participants or small search groups to inspect. At the same time, it is necessary to explain clearly where the group should go and what is found there. Then the group should report on the results of the inspection. Interaction is also complicated by the fact that often not only full-time employees of the Ministry of Emergency Situations participate in the search operation, but also volunteers of different levels of training and discipline.
A unified information system that automates the analysis of aerial photographs and provides tools for communication between search participants can help in solving these two problems. Search and rescue units of the Ministry of Emergency Situations and volunteer organizations use UAVs with their onboard and auxiliary software in search operations, a number of communication programs are also used, including both specialized departmental developments and public messengers and social networks, but there is no software package for information support of search operations aimed at solving problems of analyzing aerial photographs and communications.
This paper offers an approach to creating such a package, describes its architecture, and also considers the technologies used in the development and the main technical solutions.
The neural network kernel of the package, which recognizes search objects on aerial photographs, features of its configuration and training, is described separately.
When developing a software package for searching people and vehicles using an unmanned aerial
vehicle, we determine participants, stages and types of work for a search and rescue operation on the
basis of the guidelines of the Ministry of Emergency Situations [
The most in need of automation and information support is the active phase of the search operation, so the most attention was focused on it. The reception of data on missing persons, their signs and circumstances of disappearing, the report of the operational duty officer to the head, the decision of the head on the start of the search operation and the appointment of the head of search and rescue operations (HSSO), as well as the planning of the search operation is not considered within the package.
The main roles of participants in the active phase of the search operation are a head of search and rescue operations, a search coordinator, a member of the search group (MSG), an administrator, UAV operator, a data engineer.
After the start of the active phase of the search operation, HSSO appoints a search coordinator (SC) or performs this role himself. The coordinator registers the beginning of the search operation in the system and enters data about the search objects (SO). There may be several of them. For example, if a family went by car to pick mushrooms and disappeared, it is advisable to designate all the missing family members as an SOs, as well as their car, which can be an important reference point when searching for people. Then the coordinator selects the warning mode: broadcast - to all available potential search participants; or addressable – selected personally or by category (full-time employees, experienced volunteers, novice volunteers); and performs the warning.
Then during the search operation, information about the detected objects (DOs) appears, which can potentially turn out to be search objects. This information can appear as a result of the analysis of aerial photographs (then they appear in the system automatically from the image analysis subsystem), as well as the result of interviewing witnesses (then the coordinator himself enters these objects). Some detected objects that are clearly visible in the images and can be visually identified as not being the object of the search, can be immediately transferred by the coordinator to the "Wrong" status. The rest of the detected objects should be checked by the participants of the search groups operating in the search area. Also, the possible locations of the search objects (convenient shelters from bad weather, natural obstacles in the way of probable movement, and others) can be designated as objects for verification.
The unit for verification in the system is the detected object, but a member of the search group is given a mission that may contain one or more detected objects as a task for verification. The search coordinator manages the missions: creates missions, adds detected objects to them, appoints a performer (a member of the search group), and also cancels missions if necessary. If a member of the search group marks a search object with the status "probably detected", the search coordinator checks this information through other channels and transfers the search object to the status "detected" or returns it to the status "wanted".
The coordinator can distribute the detected objects into missions for manual verification. With a small number of them, this does not cause difficulties. But after the automatic processing of the next part of aerial photographs, quite a lot of detected objects may appear, and then the coordinator will have to spend considerable time on their distribution.
To save this time, an automated distribution mode is provided in the developed package. We considered it logical to entrust verification of several nearby detected objects to the same member of the search group who is the closest. When switching to the automatic distribution mode, the system first performs clustering of detected objects by k-mean method and groups the objects of each cluster into a mission, and then appoints the nearest MSG as the mission performer. By default, all detected objects with the "Requires verification" status that have not yet been distributed to missions and all MSG participate in the distribution, but the coordinator can exclude objects or participants from the distribution if he finds it necessary.
Potential members of search groups are all users registered in this status. As a result of selfregistration, the user receives the status of "potential MSG" with the category "novice volunteer". In the future, the system administrator can assign the user the category "experienced volunteer" or "fulltime employee". After receiving an invitation to participate in a search operation, a potential MSG accepts the invitation and becomes MSG of this operation or rejects it. Regardless of the invitation, a potential MSG can look through the list of active operations in which he could take part, choose one of them and become a member of the search group.
A member of the search group can always look through the list of operation search objects and information about them. Also, MSG sees a list of missions assigned to him, he can see the details of the mission: a list of detected objects for verification, information about each object, including coordinates and an image with the designated location of the object. Also, MSG can see missions of other participants or objects undistributed by missions.
While performing his mission, MSG passes the places where objects are marked for verification and sets the status "No object" (the object is not found on the spot), "Wrong" (the object is found on the spot, but is not the object of the search), "Probably wanted" (the object is found on the spot, and it is probably the object of the search). If the latter status is selected, the mobile application offers to select the search object from the list of the current operation and attach several photos of the detected object. When all the objects of the mission are checked, it automatically switches to the status "completed".
If MSG detects a search object in a place where the detected object was not previously indicated, then he selects the search object from the list and transfers it to the status "probably detected". In this case, the system automatically creates a detected object in this place and associates it with the search object, asking to add several photos.
UAV operator controls his device to perform aerial photography of a given section of the search area, sets flight paths and shooting modes, and then uploads the images to the system for further analysis.
During package usage, images can be accumulated that could improve the recognition ability of the system. The data engineer selects such images, adds them to training samples, performs training of the neural network and checks the training results. If the trained model turns out to be more successful than the previous one, the data engineer can decide to replace the old model with a new one. It is assumed that it is possible to save the previous models and switch from one to another. This can be useful, for example, when changing the season and, accordingly, the nature of the images.
The administrator has the ability to manage the accounts of other users of the software package.
Figure 1 shows a use-case diagram that shows the roles of users and the functions available to them in the software package being developed.
The system under development provides information support for the process in which users of various roles are involved significantly remote from each other. The participants of the search groups work in the search area, UAV operator is located quite close to the search area, and the coordinator can be located both in the operational headquarters in the field and in a stationary office at a regular workplace. In such conditions, it is convenient to organize data exchange between parts of the system via the Internet. The heterogeneity of the system, which defines it as a software package, is due to significant differences in data processing by different subsystems.
Users of different roles work with data in different ways. The coordinator needs to see the entire search operation on the map with all the missions and members of search groups, as well as enter quite a lot of text data, which means it is more convenient for him to use a laptop or a desktop computer. At the same time, it may be necessary to work at someone’s unprepared computer, so the user interface of the search coordinator is implemented as a web application that includes a visualization subsystem for displaying information about detected objects and search group participants on the map, a search operations management subsystem, as well as a subsystem for working with missions.
The interface of the search and rescue operations manager is also represented by a web application.
For UAV operator the system provides a web interface that allows, first of all, to download aerial photographs for analysis. At the first stage of the package development it is advisable to restrict to this, since interference in the on-board software of the aerial vehicle is associated with significant organizational difficulties. The variety of devices used also presents additional complexity for the development or modification of their onboard programs. At the same time, in the future it may be convenient to have a UAV control subsystem as a part of the operator interface, which will allow to generate a flight task based on the information about the search operation, as well as to receive images in real time. In some cases, UAV operator may not even have a laptop at hand, so an interface in the form of a mobile application is also provided for it.
The client part of the web application that implements user interfaces is built on React platform. Data for all types of clients, including mobile and web applications, is provided through the program interface (API) by the server part of the web application developed on ASP.NET Core technology. In its turn, the server of the web application accesses MS SQL Server database server to save and retrieve data. When processing requests for uploading images, the server part of the web application saves images on an external file storage and sends a new task to process a set of images to a queue based on RabbitMQ.
The choice of the mobile application form factor for the search group member's interface is due, firstly, to the fact that it is extremely inconvenient to carry even a laptop when working in the field; and secondly, to the fact that the phone provides the opportunity for geo-positioning and photographing, which may be needed in MSG work.
The image analysis subsystem is a separate application written in Python that constantly listens to the queue and retrieves and processes it when a new task appears. The task is a data structure in JSON. It contains ID and type of the task, the path to the directory with the source files and the path to the target directory where the recognition subsystem needs to send the marked-up images. The logging module leaves records about the extraction of the task and its processing in the local event log, and the module for sending diagnostic messages transmits information through a special program interface. The subsystem also sends the results as an array of JSON objects with the description of objects classes and coordinates through the appropriate methods to the server part of the web application. Since the task of recognizing objects in an image is quite resource-intensive and has a large computational complexity, it is solved using CUDA technology, which implements calculations on graphics processors, which can significantly increase the performance. Additionally, the package architecture includes the possibility of simple scaling for the image analysis subsystem: several applications can be launched simultaneously on separate computing capacities, which will listen to the task queue. The uniqueness of processing each task is achieved by means of RabbitMQ, and the convenience of convolution is provided by using a configured Docker container with an image analysis subsystem application.
The possibilities for a knowledge engineer are implemented as a set of utilities for image processing, the formation of a training sample and the training of recognition models.
The architecture of the software package for searching people and vehicles using UAVs is presented in Figure 2.
As it was mentioned above, automating the detection of search objects in UAV images is one of the key tasks of the package being developed. Among the typical tasks of image analysis, the most suitable for us is the detection task, which allows to detect objects of one or more classes in images. In this case, it is enough to solve a simpler modification of the problem, in which the edges of the rectangle describing the detected object are always made parallel to the sides of the image.
Convolutional neural networks have proven themselves well in solving such problems. The analysis
of scientific papers and the results of various competitions has shown that the most popular and effective
networks in many cases are YOLOv5, EfficientDet and RetinaNet [
So in article [12] application of convolutional neural network YOLO and Fully Autonomous UAV for Search and Rescue Operations on marine environment is described.
There are also examples of UAV images analysis for search operations based on image classification task [13] or combination of classification and segmentation tasks solved using Regionbased Convolutional Neural Network (R-CNN) [14, 15].
The problem is that the task of detecting people in the images is solved quite steadily when shooting at an angle: from above and from the side. At this angle, the human figure is well recognizable. This is how security cameras are installed in public places and in other cases when it is possible to choose the point of camera installation. This is how most of the models in the open access are trained. But when searching for people on rugged terrain or in places with high vegetation (trees, bushes and even just tall grass), this angle is very likely not to show a person in the photographed area at all, especially lying or sitting, since he is obscured by other objects. In such conditions, it is more appropriate to shoot an orthophotoplan or a combination of orthophotoplan with a perspective survey.
We decided to test how these networks will behave while solving the problem of detecting people and vehicles in the natural environment in UAV images. For this purpose, a training data set was formed, which was based on marked-up images from open sources, as well as made during various search and training operations of the group for the use of unmanned aircraft systems of the Control Center in Crisis Situations of the Main Directorate of the Ministry of Emergency Situations of Russia in the Bryansk region, which we marked ourselves. The sample was approximately equally filled with orthophotoplans and perspective images. The structure of the training set is presented in Table 1.
Total
3260
1086
3125
4211
To conduct the experiment, networks and their modifications were taken from open repositories [16, 17] and configured for operation. The criteria for comparison were the minimized inference time and the model size, as well as the maximized metrics: accuracy, completeness and F1Score Precision =
Recall = (TP + FP) ,
, 1 =
(TP + FN) 2 ∗ Precision ∗ Recall
, (Precision + Recall) where TP is the number of true-positive solutions (when the neural network detects the object, and it coincides with the reference markup), FP is the number of false-positive solutions (when the neural network detects the object, but there is no such an object at the reference markup), FN is the number of false-negative solutions (the neural network does not detect the object, where it is located at the reference markup).
In the problem being solved, the error in determining the position of the detected object does not matter much, as long as it does not exceed the size of the object itself, so the object is considered correctly detected if the Jaccard index for it and its reference exceeds 50%, that is, the areas of their location in the image overlap by more than 50%.
It should also be noted that convergence and resource intensity are very different in the compared networks. Therefore, testing by the same hardware configuration, we selected the maximum possible values of "batch size" parameter for each network, and then performed training until the network continued to learn significantly, without switching to overfitting mode. Thus, the number of epochs, batch size and learning time are not directly the selection criteria, but are given in the table for comprehensiveness.
The parameters and results of training models are shown in Table 2.
Testing has shown that the best recognition results in our sample are given by YOLOv5 in both modifications. At the same time, YOLOv5x is more resource-intensive and slower to output than YOLOv5s. Since the recognition quality was the most important criteria for us, the choice was made in favor of YOLOv5x. In addition, its speed is enough to process images by flying the area of 1 km2, which UAV does during half an hour in about 9 minutes.
If you need to recognize faster or use less resources, it will be advisable to use YOLOv5s, which has only slightly less recognition quality.
Object recognition using a neural network is an important, but not the only task of image processing in the package being developed. We can say that the image processing module is a pipeline consisting of several sections in which the image is subjected to different operations.
GeoTIFF files are sent to the module input. First, data on the spatial resolution and geo-coordinates of the image boundaries are extracted from them. These geodata are stored in the memory for future use.
Then the image is split. The fact is that the selected network configuration allows to process an image with the size of 832*832 pixels at a time, while the usual image size from the camera is 3840*2160 (by the way, other neural networks also work inefficiently with large images). Therefore, first the image is cut into tiles of the desired size and only then these parts are input to the neural network.
The neural network searches for objects of our interest. Based on the results of processing each tile, it outputs an array of rectangles specified by the relative pixel coordinates of this tile.
After that, the rectangles that fall on the cutting lines of the original image are patched together.
The resulting rectangles are applied to the resulting image, that is, the image is marked up. The color of the rectangle is selected depending on the class to which the detected object belongs.
Then the coordinates of the rectangles are converted to geo-coordinates, taking into account the location of the tile in the original image and the previously saved geo-data of the processed file.
Finally, the marked-up images and an array of rectangles in geo-coordinates are transferred as the output of the module.
Earlier, a similar scheme for processing aerial photographs was used by the authors of this paper in the project on recognition of buildings and structures [18].
No corporate computing power was allocated to develop the package, but the more clearly proposed architecture showed its stability. The server part of the web application and the database server were placed on the hosting provider's virtual machine, the file storage – on a personal desktop computer (CPU Intel Core i5-650 3.2 GHz, RAM 32Gb, GPU Nvidia GTX 1060 6Gb), and the image analysis subsystem and the task queue server were placed in separate Docker containers on the same computer.
A mobile application of a search group member was installed on several phones. All package components were connected to each other via the Internet. In this configuration, the package functioned normally.
Together with the GU of EMERCOM of Russia in the Bryansk region, a number of training search operations were conducted based on previously obtained material, and then a training operation in the field and the use of UAVs.
For the convenient work, most of the data is displayed to the coordinator on the map: here he sees the current position of the search participants (it is transmitted from the mobile application of the search group member), the position of the detected objects and their statuses. The function of filtering detected objects by missions and statuses is available. When you direct the mouse pointer to the label of the detected object, a window opens with a thumbnail of the image in which it was detected. When you click on the label or the window that opens, the marked-up snapshot opens to full screen. If objects with the same coordinates are found in several images, they are automatically combined into one object for verification, and in the window that opens to the full screen, you can scroll through all the images in which this object was found. A screenshot of the coordinator's web application is shown in Figure 3.
In the web application of a search group member, the main working screen is also based on a map, which shows the labels of this member himself, of other participants, as well as the labels of detected objects. Additionally, there are some screens to provide the rest of the functions. Screenshots of MSG mobile application are shown in Figure 4.
The developed package can be considered a successful pilot project, but its practical operation requires serious refinement in a number of areas, some of which were known at the development stage, some were identified during pilot operation. These areas include the following.
1. It is necessary to collect an extended training data set to prepare a universal model for recognizing images of all seasons and weather conditions, or some particular models. The current model provides recognition only in snow-free time. 2. It is necessary to implement the possibility of automatic filtering of objects detected in the images that correspond to the images of search participants and their vehicles. One of the ways of such filtering is recording the data about the location of search participants during UAV flight transmitted by the mobile application. 3. Good opportunities open up when implementing the recognition of additional classes of objects: vehicles by type, make and model; people by age and gender; as well as when providing a set of functions for parameterizing search objects, for example, by the predominant color. 4. At the current level of the package functionality, the coordinator is recommended to give some of the objects found in the images to special participants of the search group who are UAV operators; they will verify faster than MSG on foot. However, the development of the functionality for the automatic definition of detected objects that it is advisable to check with the help of UAVs can increase the efficiency of the coordinator's work.
[10] A. Ammar, A. Koubaa, M. Ahmed, A. Saad, Aerial Images Processing for Car Detection using Convolutional Neural Networks: Comparison between Faster R-CNN and YoloV3, 2019. URL: https://arxiv.org/abs/1910.07234. [11] G. Plastiras, C. Kyrkou, T. Theocharides, Efficient ConvNet-based Object Detection for Unmanned Aerial Vehicles by Selective Tile Processing, Proceedings of the 12th International Conference on Distributed Smart Cameras, New York, NY, 2019, pp. 1-6. doi:10.1145/3243394.3243692. [12] E. Lygouras, N. Santavas, A. Taitzoglou, K. Tarchanidis, Unsupervised Human Detection with an Embedded Vision System on a Fully Autonomous UAV for Search and Rescue Operations, 2019.
URL: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6720834. [13] Abdallah Zeggada, Abdelhamid Nouffidj, Farid Melgani, Mesay Belete Bejiga, Convolutional Neural Network Approach for Assisting Avalanche Search and Rescue Operations with UAV Imagery, 2017. URL: https://www.researchgate.net/publication/312874259_A_Convolutional_N eural_Network_Approach_for_Assisting_Avalanche_Search_and_Rescue_Operations_with_UA V_Imagery. [14] Asanka G Perera, Ali Al-Naji, Yee Wei Law, Javaan Chahl, Human Detection and Motion Analysis from a Quadrotor UAV, 2018. URL: https://iopscience.iop.org/article/10.1088/1757899X/405/1/012003/pdf. [15] Xiaoliang Wang, Peng Cheng, Xinchuan Liu, Benedict Uzochukwu, Fast and Accurate, Convolutional Neural Network Based Approach for Object Detection From UAV, 2018. URL: https://arxiv.org/ftp/arxiv/papers/1808/1808.05756.pdf. [16] Open Source Computer Vision Object Detection Models. URL: https://models.roboflow.com/object-detection. [17] YOLOv5 in Pytorch. URL: https://github.com/ultralytics/yolov5. [18] D. Bulatitskiy, M. Gavrilenkov, A. Buyval, Building Recognition in Air and Satellite Photos, Proceedings of the 29th International Conference on Computer Graphics and Vision, Bryansk, 2019, pp. 173-177. doi: 10.30987/graphicon-2019-2-173-177.