Current neural network-based algorithms for object detection require a huge amount of training data. Creation and annotation of specific datasets for real-life applications require significant human and time resources that are not always available. This issue substantially prevents the successful deployment of AI algorithms in industrial tasks. One possible solutions is a synthesis of train images by rendering 3D models of target objects, which allows effortless automatic annotation. However, direct use of synthetic training datasets does not usually result in an increase of the algorithms' quality on test data due to differences in data domains. In this paper, we propose the adversarial architecture and training method for a CNN-based detector, which allows the effective use of synthesized images in case of a lack of labeled real-world data. The method was successfully tested on real data and applied for the development of unmanned aerial vehicle (UAV) detection and localization system.
Object detection is one of the key tasks of computer vision. The main purpose of detection is to locate, identify, and localize all objects of certain classes in an image. The well-established approach is using convolutional neural networks of various architectures (CNNs). Currently, a number of detectors have achieved the ability to work in real-time with fairly high accuracy.
While excellent performance has been achieved on large public datasets, real-world object detection still faces great difficulties. One reason for this is the lack of sufficient annotated real-world data to train detection algorithm for a specific task. Another reason is the difference between filming conditions at the training and the execution stages. View angles peculiarities, object appearance, background, illumination, image quality determine as so-called “domain” of data. The development of domain-invariant recognition methods or image domain transfer methods is a complicated scientific task. Manual target dataset creation and annotation is an expensive and time-consuming problem, ? Publication is supported by RFBR grant 19-07-00844 which researchers try to bypass in different ways. In particular, the design of a domaininvariant object detection algorithm is an important and prospective task.
In this paper, the domain adaptation problem is investigated in relation to the object detection task for mini-UAVs (drones). The source domain is the data obtained by rendering 3D models of the object (artificial data), the target domain is a limited number of real-world images. Our main achievement is the application of domain transfer by adversarial training technique between synthetic and real data, which allowed us to achieve high detection accuracy provided an extremely limited amount of real data. Such results are not achievable by conventional training methods as we demonstrated in experiments.
The drone detection algorithm is a key component for the development of a
passive indoor drone positioning system (where there is no signal from satellite navigation
systems available) using a set of stationary surveillance cameras [
The idea of automatic object detection on images has been around for a long time.
The first successful attempts were based on detection of low-level image features, for
example, the Canny edge detector [
Domain adaptive methods have been developed last years. They have made it possible to expand the neural network algorithm’s applicability and partially solve the lack of annotated data problems for applied tasks.
The domain adaptation concept based on adversarial training was proposed in [
The A-Fast-RCNN detector [
Neural networks require lot of annotated training samples to achieve outstanding accuracy. Those algorithms have large number of tunable parameters that determines their high flexibility. Therefore, a large number of annotated images are required to train these CNN-based algorithms. In case of detection, annotation indicates the object class and its coordinates (the bounding box) on the image. To avoid time- and resourceconsuming manual data labelling, artificial creation of training and testing images was applied. Images were created by rendering the 3D model of target objects.
During the data creation process, in the 3D modeling system the drone model was
drawn on a homogeneous background. Additionally, the object mask was drawn. Then
the image and mask were randomly transformed by rotation, scaling, shifting color
channels, reflection, perspective transformation, blurring, adding salt and pepper noise.
After that, the object image by its mask was placed on background images. Large
collection of arbitrary images was used for backgrounds. Local smoothing with a Gaussian
kernel were performed at the object borders in order to make such pasting look natural.
Distracting objects from the Coil 100 dataset [
The drone was flown and surveyed from 6 high-resolution video cameras in the test hangar to prepare the test samples and expand the training samples (Fig. 3). Data from all 6 cameras were annotated by hand.
Faster R-CNN architecture was used as the base detection algorithm [
The problem of most algorithms is underperformance on real data, while training on
artificial images. The unlimited synthetic data was available because of 3D rendering.
The updated algorithm main goal is training on synthetic data effectively, and then show
the good performance on real data. Object artifacts in synthetic images are perceived by
the detection algorithm as highly informative features. The domain adaptation method
was applied using adversarial training, following the approaches [
The following approach was applied in order to train the network to create more invariant object representations. Special domain classification branch (head) was added after backbone, in parallel with detection branch. The network was trained to perform object detection and data domain classification simultaneously (synthetic or real-world) based on the attributes computed by the backbone. The new domain classification head was used for network to “forget” the domain. The correct domain classification was penalized by gradient propagation of the respective loss function with the reverse sign. In other words, the network learned to create object representations invariant to the image domain. The modified detection algorithm architecture (Faster R-CNN+AT) is presented in the Figure 4.
The main role in training of the proposed network is a special loss function. It provides a mathematical formulation to compare predicted and ground-truth and significantly effects on training time and reaching a required accuracy. The minimizing loss function is set as follows: min LO(fO(fT (YT )); YO) fO;fT
LN (fN (fT (YT )); YN ); min LN (fN (fT (X)); YN ); fN where O - object detection task, N – domain classification task, T – feature extraction, fO;N;T – task modules, respectively, YO;N – training samples for the task, respectively, LO;N – losses functions, respectively, – adversarial coefficient.
(jxj; if jxj > a l1;smooth(x) = jxa2j if jxj < a (2)
Smoothed l1 (2) is taken as a loss function for the detection task. Often softmax loss
is used in classification tasks, but it causes gradient explosions in adversarial training
according to [
So, the final loss function looks like: min LO(fO(fT (YT )); YO) + fO;fT
Lne(fN (fT (YT ))); min LN (fN (fT (X)); YN ) fN
Training this model is similar to training GANs [
Algorithm 1 Faster R-CNN+AT with adversarial training for a detection task. INPUT: pre-trained fT ; fO; fN k - iterations for dumping fN weights. 1: procedure TRAINING FASTER R-CNN+AT(fT ; fO; fN , k) 2: for epoch in range(epochs) do 3: Sample a mini-batch of n examples [X1; ; Xn] 4: Update fT (wT ) and fO(wO) with gradients: 5: rwT ;wO n1 iP=n0 LO(fO(fT (YTi )); YO) + Lne(fN (fT (YTi ))) 6: while predictions from fN have training accuracy 6 0.9 do 7: Update fN (wN ) with gradients: 8: rwN n1 Lne(fN (fT (YTi ))) 9: end while 10: Restart fN for every k iterations and repeat the Procedure again. 11: end for 12: end procedure (1) (3)
5
The accuracy of the proposed algorithm (Faster R-CNN+AT) has been tested on different datasets to demonstrate its advantage over the base Faster R-CNN, provided different ratios of synthetic and real images. Synthetic data were generated by rendering and automatically labeled. The real data were annotated manually. Data from 5 cameras (800 images) and synthetic data was involved in the training phase. Data from the 6th camera (124 images) was used for testing, which did not participate in the training phase. Obtained accuracy (precision and recall) at fixed cutoff thresholds by probability, as well as standard average metrics (average precision and mean average precision) in Tab. 1.
Two main conclusions can be inferred from the experiments results (Tab. 1). 1. The results demonstrate that the proposed method provides the most effective way (among other approaches) to use synthetic data along with real data than the amount of real data is fixed. Although the use of pure synthetic data is inefficient (first row in Tab. 1), the amount of real data is insufficient to train detector solely on real images (second row in Tab. 1), adversarial training achieves perfect results (last row). 2. The developed method outperforms the base one due to the adversarial learning application. The advantage is observed regardless of what real-world and synthetic images ratio was used. It is determined not only by the complexity of the neural model (the feature extraction network is identical) but by the learning principle. 3. Experiments prove that the proposed method is more effective than the widespread fine-tuning technique, the results of which are shown in the penultimate row of table 1. Fine-tuning is training the model first exclusively on a large available collection of relatively relevant data (in our case, synthetic), then further training on target data (in our case, real). This allows one to get better results than simple data mixing, but still less accurate than the results of the proposed Faster R-CNN+AT method. 6
The CNN-based object detection training algorithm and architecture based on the adversarial technique is proposed. By its application, we solved the problem of the lack of annotated target training data. The training algorithm enforces the detector’s encoder subnet to generate domain-invariant image features. The proposed algorithm has the ability to train detectors mainly on synthetic images (obtained by 3D objects rendering) and a limited number of real-world data but shows high accuracy on target real images. We have shown the superiority of the proposed training scheme for training object detectors in a set of experiments using different real and synthesized image ratios in the training set. The method could be applied to arbitrary detector architectures.