The article presents the drone positioning technology in a multi-camera system by using the detection algorithm. Paper describes positioning system and algorithm for calculating 3d drone coordinates based on its image position, detected on images of stationary video cameras. Positioning enables automatically control the drone when precise data from satellite navigation systems are not available, for example, in closed hangars. The developed technology is used to create a complex of automatic visual control of aircraft. The ways of adaptation of neural network detection algorithm to the problem of drone detection are presented. The main attention is paid to the methods of training data preparation. It is shown that high accuracy can be achieved using synthesized images without any real data or manual labelling.
Currently, due to the increase in the aircraft flow in the airspace, the complexity of their timely and high-quality visual inspection during the maintenance at the airport has increased significantly. Significantly increased the total downtime of aircraft during unscheduled inspections, caused, for example, the impact of atmospheric electricity on the surface of the fuselage of the aircraft in flight. External human inspection of hard-to-reach areas of the aircraft, such as the upper fuselage or tail, aimed to identify the effects of lightning today takes a significant time, leading to downtime of aircraft or even flight delays. For companies which have a fleet of more than 200 aircraft, such as Aeroflot, such an event is not uncommon: according to the company, it occurs about 300-400 times a year, leading to significant time and financial losses. Large companies such as Airbus, Lufthansa, EasyJet, American Airlines start applying drones to solve the problems of accelerating the visual inspection of the aircraft. However, currently, the use of drones is carried out in manual mode, which does not allow to completely reveal the potential of the technology. According to experts, the use of programmable drones will significantly reduce the time of inspection of the aircraft and, no less significantly, make the technology itself completely digital.
The article proposes an approach to the creation of automated drone control technology based on its real time positioning using a system of stationary cameras. This technology is necessary to ensure the functioning of the drone control system in enclosed spaces such as aircraft hangars. The development of a special positioning technology is necessary, since the signals of global satellite navigation systems (GPS, GLONASS, etc.) may be partially or completely inaccessible in the hangar where aircraft maintenance is carried out. At the same time, the inertial navigation system of the drone can’t provide sufficient accuracy throughout its flight. Due to the fact that the flight of the drone must be carried out at a short distance from the aircraft (no more than 1.5 meters), ensuring the accuracy of the trajectory is a critical aspect for the safety and applicability of the technology. Visual positioning system is the most preferable in the described conditions, as it is able to provide sufficient accuracy, it does not require the installation of additional equipment on the drone, it is passive, so, it does not emit any radio or other signals except Wi-Fi.
During maintenance, the drone flies over the aircraft on a
programmed trajectory and makes a high resolution video of the
surface of the fuselage and wings (Fig. 1). Based on the
coordinates obtained from the visual positioning system, the
onboard drone control system monitors compliance with the
choosen trajectory. By results of the automatic analysis of the
received videos the decision on existence of damages on a
covering of aircraft is made. This technology allows complete
automating the process of visual inspection of aircraft [
The paper describes the features of creating such a technology in terms of positioning drones through the use of CNN-based detectors.
Fig. 1. The drone flight over the aircraft during the tests.
The proposed technology is based on an algorithm for detecting objects in images (namely, video frames). The most advanced detection algorithms today are algorithms based on deep convolutional neural networks (CNN). Neural network architectures for detection are divided into two main types: single-stage and two-stage. In two-stage approaches, the task of detecting objects is divided into two steps: identifying areas of interest, then classifying the class of object in the area, and predicting the parameters of the bounding box.
The two-stage approach was first introduced in 2014 by
Girshik [
In single-stage approaches, there is no stage of finding regions of interest, the regression of bounding boxes and the classification
of candidates in anchor areas is performed
directly.
[
A slightly different approach is used by CenterNet [
map. The parameters of the bounding
rectangle are then regressed for the detected centers. Corner Net
[
calculated coordinates of the object are transmitted to its onboard control system via Wi-Fi channel. The scheme of the proposed navigation system is shown in Fig. 2.
To calculate the three-dimensional coordinates of the object
based on its position in the images, a method is used, which is a
special case of block triangulation by the method of ligaments
[
1( − )+ 1( − )+ 1( − ) = − 3( − )+ 3( − )+ 3( − ), 2( − )+ 2( − )+ 2( − ) (1) (2) where ( ,
, ) are camera positions, ( , , ) is 3D object position, (xi,yi) is its projection on image i, fi is focus distances, 1 2 3 = ( 1 1 2 2 3) 3 is rotation matrix for camera i.
ATA ΔX + ATB = 0,
This is a well-known problem, which is solved by the method of iterative approximations. Each increment step of the three-dimensional coordinates ΔX is determined from the solution of the system of equations: where A is the matrix of partial differential of projection equations (1),(2) by drone coordinates over all cameras (size 3*3*number of cameras in the system), B is the discrepancy vector (size 2*number of cameras), containing deviations of object projections from real positions on images.
As the detection algorithm YOLOv2 [
The network receives a three-channel image as input, and outputs a tensor of size X×X×Y, where X is the number of cells in the input image. The length of the tensor Y depends on the number of classes detected and the number of anchor regions in the cell. For each anchor area, 5 basic parameters are calculated: the coordinates of the upper left corner of the rectangle, the width, the height, and the probability that this rectangle contains any object. In addition, the probability of the object belonging to each selected class is determined. The hyperparameters of the algorithm are the number and size of the anchor areas and the size of the input image.
The image size determines the number of cells for which the features are calculated, since the cell size is fixed and is equal to 32x32 pixels. Therefore, it directly affects the performance and accuracy of the network, as the number of cells increases the number of network filters. On the other hand, if there are more cells, each of them contains fewer objects; the features calculated in it correspond more accurately to each object and allow to build a more reliable prediction. The plot in number of anchors is set to 5, as the higher number of anchor areas decreases performance. K-means clustering of bounding rectangles on our training data set was used to determine anchor sizes.
Since the work uses AI detection algorithms, training data is required to learn them. In our case, data are images with annotation: the type of object and its coordinates (bounding box) in the image. The CNN detectors used are very flexible but have a very large number of parameters. In this regard, a lot of training data is required. In order to avoid time-consuming manual data labelling, automatic synthesis of images was used for training and testing the algorithm.
The data were synthesized based on the rendering of the
existing three-dimensional model of the drone (Fig. 4). Special
3D environments were not used during data endering, as their
preparation requires additional manual labor of designers.
Instead, the process was structured as follows. The model of the
drone in different angles was rendered in a 3D modeling system
on a uniform-colored background. The object in the image was
automatically cut out, and its mask was built. Then the image
and mask were subjected to random transformations: rotation,
scaling, displacement, reflection, perspective transformation,
blurring, salt/pepper noise, shift of color channel values (Fig.
5). After that, the image of the object on his mask was ovelayed
on arbitrary backgrounds. Both random images and images
from the test hangar where the subsequent testing was carried
out were used as backgrounds. In order to make such insertion
look natural and the network did not remember overlay artifacts
as informative features of the object, local smoothing of objects
with a Gaussian filter with randomized intensity was performed.
In addition, objects from the Coil-100 collection were added to
the images to increase the discriminative ability of the network
[
To prepare the test data and expand the training base through real images, the real drone flight and video capture in the test hangar were carried out (Fig. 6). To get rid of manual annotation video files, the following automatic labellng algorithm was used. Optical flow maps were calculated for each video frame. The area with the maximum magnitude of the optical flow was selected on the maps. Since normally there were no other moving objects in the experimental scene, this area was thought to correspond to a drone. Sometimes due to the presence of foreign moving objects and shadows, as well as inaccuracy of segmentation, such labelling contained several errors. An experimental study was devoted to the estimate of the influence of different types of training data on the detection results.
The accuracy of the detection algorithms was tested in a series of computational experiments on various training collections. We had three main data collections: synthetic, where images of drones were obtained by rendering 3D models, and the backgrounds are taken arbitrarily; semi-synthetic, where backgrounds for rendering was real images of the hangar in which the experiment was carried out; and autolabelled real, obtained by automated labelling drone videos (using optical flow). Incorrectly labelled data was manually deleted. Testing data was the part of the real data collection that was not used for training. The obtained precision, recall and IoU for different sets of training data are shown in table 1.
According to the results of the experiments, the following conclusions can be drawn. First, it is possible to train the algorithm with high accuracy on fully synthetic data, which was the purpose of the work. Secondly, the smoothing of objects when overlaying them on the background image plays a crucial role. Without smoothing, artifacts at the boundaries of objects become too important feature for the neural network, and it overfits to detect only artificial objects. Third, the use of a large number of random backgrounds was better than the use of a small number of real backgrounds from the test hangar. Despite the fact that the background images on the test data were similar to the training ones (but not the same), the network overfits, that means it has a low generalizing ability and does not cope with new scenes. Fourth, the inclusions of random objects (distractor) in the training images allowed significantly improve the accuracy of the work. Although these objects are not labelled in the test data, the network has learned to better distinguish drones from any other objects (see table 1).
In addition, during the experiments it was found that when training the network on the data obtained by the abovedescribed autolabelling method, the accuracy was worse than on synthetic data. This is due to the fact that optical flow map is blurred, and the resulting bounding box is greater than the real object bounding box (Fig. 6). Also, the available real data are not sufficiently diverse. 7. Conclusion 100% 27.53% 18.25% 41.13% 45.56% 33.92%
Recall 98.69% 97.71% 86.60% 92.48% 93.32% 37.91%
IoU 98.65% 97.63% 86.41% 91.58% 98.64% 37.89%
The paper describes the indoor drone positioning technology based on stationary visual sensors and the algorithm of drone detection. Given camera orientation and detection results the 3D position is reconstructed using a special algorithm of iterative minimization of the total reprojection error. The ways of adaptation of the CNN-based detector to the subject area were investigated. Both the automated process of creating training data and hyperparameter tuning are described. The influence of the data generation methods on the result is studied, in particular the inclusion of distracting objects in the data, artifacts of object overlay, the use of various background images. Conducted experiments showed that it is possible to train high accuracy detector exclusively on automatically synthesized images obtained using the renderings of a threedimensional model of the drone without any real samples.
This work was supported by the Russian Foundation for Basic Research, project no. 17-08-00191 a.