The article analyses the possibilities to use the unmanned aerial complexes in the system of decision making process for the crisis situations that require the object detection at aerial images received by the unmanned aerial vehicle under the conditions of atmospheric fog and smoke over the territories. For image sharpening we used Pansharpening method for injecting the dimensional details from panchromatic image to multispectral image. In order to increase the operational eficiency and accuracy of automotive vehicles detection at aerial images received by the unmanned aerial vehicles for more eficient use of received information in the system of decision making support it was selected Hybrid Task Cascade for Instance Segmentation model. This model is more appropriate for solving the tasks of small-sized object multiclass classification and detection at aerial image using the indirect signs.
Some ten even five years ago the unmanned aerial vehicles (UAVs) were regarded skeptically as the ex-pensive toys for entertainment – to film the landscapes, animals, make photos from a bird’s perspective over the reserved areas and so on. It was interesting only for quite few devoted people.
The contemporary situation all over the world concerning COVID-19 (SARS-CoV-2) epidemics
placed new demands on mankind for communication, behavior and living [
Another important UAV task was the drugs and other important items delivery to the people
on self-isolation (food, hygienic stuf, essential goods). There was also carried out the monitoring
and control over the fires and other hazardous objects in hard-to-access areas [
One of the most important elements afecting such tasks is the visualization system (information display) and information processing technologies. Often, one of the reasons making impossible using the visual control principles of UAV landing or information gathering concerning the objects at the Earth and the very Earth as bottoming surface is the atmospheric fog, smoke over the ground and imperfect (not adapted for such conditions) methods of object detection at the aerial photo-images received by the UAVs.
In the conditions of low possibility to take into account all factors concerning the UAV (visual confirmation) it may cause the task failure or flight safety violation. Accordingly, the key markers for delivery or situation monitoring using the UAVs in the conditions rough for visualization are the abilities to assess fast and reliably the area where the automatic object detection, recognition and classification means above the Earth ground may prove justifiable.
The contemporary visualization systems enable to represent huge information volumes from various sources of spatial basing: spaceships as the Earth surface optical-electronic monitoring and remote sensing, and UAVs. Usually, information from such sources does not contain the intermediate conclusions concerning monitoring that complicates the sequence of events forecast and executive decision making. To solve such problem in the automatic mode the gathered information processing is carried out – thematic aerial image processing. The thematic processing and data complexation from all aforementioned means enables the overall situation assessment in the given Earth area.
Such method of information gathering requires using system analysis and synthesis method of diferent time and parameter data from physically diferent means of information gathering. For qualitative incoming trafic transformation process of separated data from all the sources of spatial basing into a single final result fit for using under the complicated visual conditions it is necessary to determine the main components (phases) of thematic processing, logical links of various structural data complexation study along with determination of evolving problems and possible means of their solution.
In the framework of solution of the new tasks for noncontact communications using the UAVs it is necessary to search and develop an eficient (operative and suficiently reliable) detection method of fine-grained objects at aerial images received by the UAVs both in simple conditions and in the conditions rough for visualization.
The purpose of the article is to analyze the application of object detection neural network models for UAV image processing in conditions of atmospheric haze and smog, with their further improvement to increase the accuracy of localization and recognition of objects on the ground surface.
Based on the analysis of atmosphere transmission over Ukraine in 2019 given in table 1 as for classical visualization of image results at aerial photo-images from various sources it is possible to conclude that depending on the season 35 percent of daytime per year is clouded and require special methods of object detection at the aerial images received under such conditions. r e b m e t p e S r e b o t c O r e b m e v o N r e b m e c e D iegonR rcahM lirpA ayM Jeun lJyu AR Crimea Vinnytsia
Volyn Dnipropetrovsk
Donetsk Zhytomyr Zakarpattia Zaporizhzhya Ivano-Frankivsk
Kyiv Kirovohrad
Lugansk
Lviv Mykolayiv
Odesa Poltava Rivne Sumy Ternopil Kharkiv
Kherson Khmelnytskyi
Cherkasy Chernivtsi Chernihiv
UKRAINE Over 70% of time the sky was cloudy during that month From 25% to 70% of time the sky was cloudy during that month
Up to 25% of time the sky was clouded, and 75% it was clear during that month
Smoke is one of the emergency situations factors, which excludes the possibility of using
detectors for processing aerial photographs from UAVs. The Pansharpening method, which is
based on the use of spatial details injections from panchromatic image to a multispectral image,
showed better results for improving the original image in the presence of atmospheric haze or
smoke during fires. Currently, the following injection models can be distinguished: the
GramSchmidt projection model of orthogonalization, which was underlined the spectral sharpening
[
For the task solution of eficient object detection and recognition at images there have been used the methods of image semantic and instance segmentation which are developing in parallel and which have their peculiarities, ad-vantages and disadvantages.
The methods of semantic segmentation that use convolution neural networks (CNN) solve the
task of detection and recognition from their multilevel aggregation or from through structural
prediction [
Instance segmentation allows solving the tasks of actual semantic class object identification
related to an aerial image pixel. Starting from the regional CNN (R-CNN) [
Background object classifiers that use semantic segmentation methods usually built on FCN
with extensions [
To increase reliability of fine-grained objects detection two-stage detectors have been
developed [
One of the ways to increase the detection credibility and object recognition is usage of cascade structure of neural network structure. In particular, Cascade R-CNN [32] is composed of several stages where the previous stage sends the data to the next one with metrics IoU threshold values increase to increase the quality of data processing trainings. Direct combination of Cascade R-CNN and Mask R-CNN provides an insignificant improvement due to mask foresight at further stages that receive higher accuracy of detection and recognition only from more qualitatively localized bounding boxes without direct combination. So, the creation of multi-stage conveyer of aerial image pro-cession that uses the combination of detection, instance segmentation and semantic segmentation for receiving the context, will enable to increase the accuracy of object detection and recognition.
Information from various sources of spatial basing will stipulate complexation of various structural data within onetime interval (during the first day half). So, the data about the same object are received by UAV – aerial image in visible range, and from spaceship – multispectral image. Such approach will enable the connection with spatial and spectral analytical models and in case of the library of spectral etalons availability it may enable to use spectral-spatial (sub-pixel) analysis which should result in automatic ground object identification at the Earth surface (table 1).
It is also important to harmonize the images in one format, so that they had the same resolution. Then, the main principle of data acquisition construction about the object of monitoring may become the optimal efort resolution among the means of various spatial basing sources. In this case it is necessary and suficient is the task of multi-criteria task solution for choosing suficient means of intelligence data gathering and sequence of their use determination. The optimization approach stipulates mathematical models use and optimization criterion explicitly. The basis for such task solution is the best alternative search by some criterion. Such approach enables to increase the solution quality through such factors: • enables to find the variants of task solution at various values of real limits to variables and various initial conditions; • enables to simplify the best solution selection procedure thanks to using the analytical criteria; several criteria may be used simultaneously; • presence of multitude of methods of dynamic optimization task solution enables to select the best alternative.
Pansharpening methods synthesize images with the same number of spectral channels as the input multispectral image and resolution as in the input panchromatic image. After interpolation from the multispectral image into the panchromatic space, elements are extracted from it and added to the corresponding bands of the multispectral image using the injection model. Panning is pre-selected with a histogram, that is, radiometrically transformed by constant gain and ofset. The injection model defines the combination of the multispectral image low frequency image with the spatial details of the panchromatic image. This approach is applied to each resampled band of the multi-spectral image and the low-frequency version of the panchromatic image. In this approach, the bandwidth of the panchromatic image covers four spectral bands (figure 1). This provides the advantage that the removal of the estimated path radii for the calculation of the injection model is more consistent in terms of spectral quality (color hues) in relation to spatial characteristics [33]. This approach is the basis for the decision regarding the survey of infrastructure objects in the epicenter of the fire to improve image quality [34].
Based on the results of the detectors application [
As the model basis CNN ResNeXt [29] in BiFPN [
To increase the credibility of object detection and recognition through object image
localization increase at the aerial image and bounding boxes adaptation to the object forms the guided
anchorage regional proposal network is used (GA-RPN) [
Common communication use between the bounding box detection and masks gives limited prize, so their cascade application for improvement of detected object localization and their recognition is more eficient solution. The cascade procedure is applied during the conclusions of each stage that enables to coordinate the hypotheses more accurately. The cascade use enables to decrease the network retraining as a result of exponentially vanishing positive samples and stage conclusion non-conformity for IoU value, for which the detector is optimal, to incoming hypotheses. But there is a rupture in information flow between the branches of cascade various stages that results in mask separation at later stages and gives prize only in better localized bounding boxes [32].
To overcome the rupture between the stages the hybrid task cascade is used for instance
segmentation [
To detect the objects, the scene context provides useful recommendations for semantic
branches combination for receiving the categories and scales. Received from each BiFPN layer
feature cards of various levels transform into pyramidal phalanx module PPM [
This approach difers from the existing cascade solutions by regression of bounding box sequence and mask prediction instead of their processing in parallel, inclusion of direct way to augment the information flow between the mask branches, delivery of previous stage peculiarities to the mask, direction for study of more contextual information of additional semantic segmentation branch and its alignment with bounding box and masks branches (figure 2). Using the detector sequence that passed the training with the threshold values increase of IoU metrics to be consistently more selective against the close faulty actuations. The sequence of information passing among the cascade stages is displayed by the formulae: = (, − 1) + ((), − 1), = (, ) + ((), ), = (),
, −− 1)),
= ( (
(1)
(2)
(3)
(4)
(5)
(8)
(9)
(10)
where , – detected by bounding box and feature masks; (, − 1) – align operation
RoI Align [
Training of suggested cascade includes the class predictions, bounding box and mask regression and it is per-formed in the mode from beginning till the end. The general loss function takes the form of multi-task training at each iteration and looks like this:
= ∑︁ ( + ) + ,
=1 (, , , , ˆ, ˆ, , ˆ) = (, ˆ) + (, ˆ) + 1(, ˆ) + ℎ(, ˆ), ˆ (6) (7) (, ˆ) = (, ˆ),
= (, ˆ), where – general loss function;
, – loss of bounding box prediction and mask at stage ; , – loss of classification prediction and object image regularization; , ℎ – losses of anchor localization and anchor form prediction; – loss of semantic segmentation prediction; – loss function of cross entropy; – loss function of binary cross entropy.
While creating the training selections for each class of objects by their images for the new dataset from the aerial images a misbalance of classes arises because of lack of suficient number of object images. When using the loss function of cross entropy during model training at such datasets the scale ratio goes to zero because confidence in correct class grows. To solve this problem various methods are used as resampling. According to the results of re-searches held this solution ofers to modify the focal loss method designated to improve the model training at the original non-balanced data. So, instead of cross entropy loss function:
() = − (),
very often the function of focal loss [
() = − (1 − )(), where – focal loss; – loss function of cross entropy; – probability of credible class; – focusing value.
The focal loss minimizes the input of well classified samples and directs the focus at complicated samples. The function of focal loss is elaborated to solve the object determined detection scenario where an extraordinary balance exists between the full and sparse classes. But it does not show better results for two-passage detectors which separate the background at the ifrst stage. It is ofered to modify the focal loss function to soften the reaction for the loss functions to complicated samples. Accordingly, the same weights are used for positive samples with probabilities less than certain threshold as well as for minimization of well classified samples influence the focal loss approach is pre-served which scale reflects the threshold. The aforementioned may be described next way: where (, ℎ) – rejection ratio that scales the loss function by next formula: () = − (, ℎ)(), {︃ (1− ) 1 ℎ : < ℎ : ⩾ ℎ (11) (12) where ℎ – probability of fundamental truth class.
The focal loss modification function helps to improve the average accuracy of object detection mAP for sparse classes, however, mAP is decreased a little for well flooded classes. Function of modified focal loss application decreases the action of class misbalance factor in the process of model training.
For approbation of improved model of hybrid segmentation cascade and in order to study the process according to the task DataSet with Vehicle Detection in Aerial Images was used. It contained 10 photos at height 1595-1600 m with resolution 5616x3744 pixels. As a result of object distribution 10 classes of transport vehicles were formed. The object class set is not balances (number of object images in the classes varies from 7 to 2454), transport vehicle images difer much by dimensions, aspect ratio, distribution by brightness and color density.
Online augmentation was used for enlargement of object images taking into account the executing condition of photographing from UAVs (turns to 0∘ , 90∘ , 180∘ , 270∘ , adding Gaussian noise, contrast, sharpness, color density change). Transfer Learning approach was used through the trained models at COCO Detection dataset.
For the model work assessment metrics mAP was used that calculates mAP average score value for variables IoU to fine a great number of bounding boxes with incorrect classifications and it enables to avoid the maximum specialization in several classes at the account of weak projections in others.
To adapt the target function presentation for the object congfiuration the deforming convolution at BiFPN top was used that applies high level of feature synthesis; for fewer anchors use and taking into account of their possible form and size the guided anchorage method is applied; for further information loss reduction in the context among various sub-regions the hierarchical global previous content is applied – PPM module enables to combine the features from five various FPN scales.
To improve the model operation quality the approach of triple increase of testing time for aerial image pre- and post-processing (image compilation with resolution 600x600, 700x700 and turn (0∘ , 90∘ , 180∘ , 270∘ ), with augmentation to 800x800, 900x900, 1000x1000).
Model training was conducted from the end to the end of 18 epochs. The results obtained are shown in table 2.
As a result of Hybrid Task Cascade model improvement along with image set growth and post-processing the mAP accuracy was improved by 3%. It enables to increase the small-sized object detection credibility at aerial photos received by UAVs. As far as this approach has a little calculation complexity it enables to implement it on UAV board.
The ofered approach based on the results of existing approaches analysis of atmospheric correction based on injection model application of spatial details based on contrast highlighted from panchromatic image into interpolated multispectral band. For the outgoing image correction to solve the infrastructural object analysis task, cars in the fire epicenter enables to reduce the atmospheric fog or smoke influence on the quality of incoming image of aerial photo processing systems for suficient level to actuate the object detector.
As a result of image automatic processing method analysis during neural networks exploration to solve the task of scene analysis, aerial images review the influence of object deformations, occlusions was identified while receiving the aerial image and background change, where the object is located. It reduces the accuracy of object detection and recognition. Based on the review of contemporary neural network models in the framework of the task it was selected Hybrid Task Cascade for Instance Segmentation. It improves the information flow through cascade inclusion and multi-tasking at each stage that uses indirect signs from topographic elements of terrain to increase cred-ibility of object detection and recognition.
Further research should be directed at increasing the possibilities to use the UAVs in the complex conditions of the crisis situation and complex spatial orientation. in: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, pp. 8759–8768. doi:10.1109/CVPR.2018.00913. [24] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, S. Belongie, Feature Pyramid Networks for Object Detection, in: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 936–944. doi:10.1109/CVPR.2017.106. [25] J. Dai, K. He, Y. Li, S. Ren, J. Sun, Instance-Sensitive Fully Convolutional Networks, in: B. Leibe, J. Matas, N. Sebe, M. Welling (Eds.), Computer Vision – ECCV 2016, volume 9910 of Lecture Notes in Computer Science, Springer International Publishing, Cham, 2016, pp. 534–549. doi:10.1007/978-3-319-46466-4_32. [26] R. Girshick, Fast R-CNN, in: 2015 IEEE International Conference on Computer Vision (ICCV), 2015, pp. 1440–1448. doi:10.1109/ICCV.2015.169. [27] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, A. C. Berg, SSD: Single Shot MultiBox Detector, in: B. Leibe, J. Matas, N. Sebe, M. Welling (Eds.), Computer Vision – ECCV 2016, volume 9905 of Lecture Notes in Computer Science, Springer International Publishing, Cham, 2016, pp. 21–37. doi:10.1007/978-3-319-46448-0_2. [28] J. Redmon, S. Divvala, R. Girshick, A. Farhadi, You Only Look Once: Unified, RealTime Object Detection, in: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE Computer Society, Los Alamitos, CA, USA, 2016, pp. 779–788. doi:10.1109/CVPR.2016.91. [29] S. Xie, R. Girshick, P. Dollár, Z. Tu, K. He, Aggregated Residual Transformations for Deep Neural Networks, in: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 5987–5995. doi:10.1109/CVPR.2017.634. [30] S. Gidaris, N. Komodakis, Attend Refine Repeat: Active Box Proposal Generation via In-Out Localization, in: British Machine Vision Conference, York, 2016. doi:10.48550/ arXiv.1606.04446. [31] B. Yang, J. Yan, Z. Lei, S. Z. Li, CRAFT Objects from Images, in: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 6043–6051. doi:10.1109/ CVPR.2016.650. [32] Z. Cai, N. Vasconcelos, Cascade R-CNN: Delving Into High Quality Object Detection, in: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, pp. 6154–6162. doi:10.1109/CVPR.2018.00644. [33] S. Lolli, L. Alparone, A. Garzelli, G. Vivone, Benefits of haze removal for modulation-based pansharpening, in: L. Bruzzone (Ed.), Image and Signal Processing for Remote Sensing XXIII, volume 10427, International Society for Optics and Photonics, SPIE, 2017, p. 1042707. doi:10.1117/12.2279086. [34] S. Kovbasiuk, L. Kanevskyy, I. Sashchuk, M. Romanchuk, Object Detection Method Based on Aerial Image Instance Segmentation in Poor Optical Conditions for Integration of Data into an Infocommunication System, in: 2019 IEEE International Scientific-Practical Conference Problems of Infocommunications, Science and Technology (PIC S&T), 2019, pp. 224–228. doi:10.1109/PICST47496.2019.9061496. [35] N. Tijtgat, W. Van Ranst, B. Volckaert, T. Goedemé, F. De Turck, Embedded Real-Time Object Detection for a UAV Warning System, in: 2017 IEEE International Conference on Computer Vision Workshops (ICCVW), 2017, pp. 2110–2118. doi:10.1109/ICCVW.2017. 247.