The paper introduces how multi-class and single-class problems of searching and classifying target objects in remote sensing images of the Earth are solved. To improve the recognition efficiency, the preparation tools for training samples, optimal configuration and use of deep learning neural networks using high-performance computing technologies have been developed. Two types of CNN were used to process ERS images: a convolutional neural network from the nnForge library and a network of the Darknet type. A comparative analysis of the results is obtained. The research showed that the capabilities of convolutional neural networks allow solving simultaneously the problems of searching (localizing) and recognizing objects in ERS images with high accuracy and completeness.
Today, there is an upsurge of activity in the field of Earth remote sensing (ERS) data processing: new software systems are being created, high-resolution image processing methods are being modernized. The current situation is characterized by the improvement of the equipment of spacecraft (SC) and ground control stations, the expansion of the functionality and spectrum of the image processing tasks performed. The scope of application of these spacecraft includes monitoring of forest, agricultural and arctic zones, analysis of natural disasters, environmental protection, public safety, etc. The growing volumes of evolving ERS data have significantly increased the requirements for speed and quality of information processing. Recently, artificial neural networks (ANN) and high-performance computing technologies have been increasingly used.
The analysis of modern work on the application of ANN has shown that neural networks are
mainly used for searching and recognizing targets that are related to the category of nonrigid [
Two types of CNN were used to process ERS images: a convolutional neural network from the
nnForge library [
A special tool for the automated preparation of training samples has been implemented to improve the quality of the classification. A human expert prepares preliminarily some images with a transparency marker set, where background pixels are set invisible using alpha channel controls. Figure 1 shows the original fragment of the ERS image, on the right: the same fragment is shown after the alpha channel change. For convenience, images with objects of different classes are sorted to different directories.
Various settings of the ANN from the nnForge library (configurations and characteristics of the layers) can be customized with scaled copies of these images. The scaling factor is chosen so that each target object is placed in a separate scanning window, the size of which coincides with the size of the input ANN window. The experiments were conducted on military equipment images of 6000x4000 pixels, made from a height of 300 meters at the Russia Arms Expo – 2015 (RAE-2015) international exhibition.
The following CNN architecture from the nnForge library was experimentally chosen:
− contrast extraction layer with a 9x9 pixel Gaussian window [
− convolution layer with the 6x6 feature maps (total 136 maps), the hyperbolic tangent module is used for normalization, the window size after processing is 34x34; − average subsampling layer with a 2x2 mask, the window size after processing is 17x17; − convolution layer with the 6x6 feature maps (total 272 maps), the window size after processing is 12x12; − average subsampling layer with a 2x2 mask, the window size after processing is 6x6; − convolution layer with the 6x6 feature maps (total 544 maps), the window size after processing is 1x1;
− dropout layer with an adjustable probability of disabling connections between neurons (experimentally set to 0.05);
− convolution layer with the 1x1 feature maps, the number of feature maps corresponds to the number of distinguished classes, the hyperbolic tangent type activation function is used.
The scanning window during the recognition moves through the image in increments of one pixel and is processed by the neural network. The sequential processing of the entire image results in a colored map where the target objects are separated from the background. Figure 2 shows an example of the original image and the result of its processing.
3.626 million objects automatically extracted from 73 images were used for training. The following results have been achieved: Classification completeness: background – 0.9976, materiel – 0.9354. Classification accuracy: background – 0.9392, materiel – 0.9974. Training time: 24 hours on one Nvidia Geforce GTX 1060 and using a single CPU core Intel Core i7 6850K (of the existing 6 cores, 3.6 @ 4.0 GHz). Processing time of ten panoramic images on one GPU: 3051 s; on two GPUs: 1640 s.
As a result of numerous experiments, the developers of the Darknet network have selected a very
successful architecture, such that it works on different training / test samples [
The software tool developed as part of this study includes the programs YOLORotate, YOLOAnchors and YOLOGetObjects.
The YOLORotate program is designed to convert images into a format suitable for training the
YOLO v2 type ANNs. The data needed for the preparation of a series of our experiments include
many panoramic images taken from the UAV. They have four classes of target objects: IFV (infantry
fighting vehicles), Military vehicles, SPG (self-propelled gun mounts) and Tanks; information about
the coordinates and sizes of each of the objects is pre-assembled and stored in text files. Each such
image has a size of not less than 832x832 pixels, where all target objects occupy a relatively small part
of the image. YOLORotate rotates images from a training sample with a given step, for example, 15
degrees. At the same time, the maximum possible number of fragment images with target objects is
cut out. A total of 4361 fragments for the training sample and 1173 fragments for the test sample were
automatically created. It is guaranteed that on each such fragment there is at least one target object. In
order to ensure that the target objects are located in random positions of the received fragment images,
a pseudorandom number generator is used. The analysis showed that with the selected size of the
source data of 832x832 pixels, up to 20 targets fall into the frame. The YOLOAnchors program uses
the k-means method [
The program YOLOGetObjects segments the images into fragments of 832x832 pixels while providing a partial intersection, and each next window captures a quarter of the previous one (both horizontally and vertically). In total, there are 70 fragments per panoramic picture. Further, all fragments are independently processed using a GPU and a general-purpose processor. The next step is to combine information about all the target objects found. Figure 3 shows an example of the result of using a trained ANN. Share of target IFV objects found Military cars
SPG
Tanks Average share of target objects found Completeness IFV
Military cars SPG
Tanks Normalized accuracy IFV
Military cars SPG Tanks IFV Military cars SPG
Tanks
The batch training where the next step of adjusting the weighting factors is based on information
about the results of processing a limited group of images of the training sample was used in all
experiments. Each group of images on the new training period is formed randomly; preference is given
to the groups with representatives of all classes of target objects. Using batch training allows improve
the quality of the neural network and abandon the resource-intensive dropout layer [
Tables 1-3 show the refinement characteristics and the results of the experiments performed in solving a multi-class problem — simultaneous search and recognition of objects of four classes.
Table 4 shows the comparative results of processing the test sample when solving single-class and multi-class problems. The training time of the selected network configuration on each of the four classes of military equipment was 6.84, 13.6 and 26.8 hours when working with groups of 28, 56 and 112 images. In the single-class case, the ANN works with only one class; the user has the opportunity to choose the best option – the network trained using a group of images of the optimal size. In the last column of table 4, the best coefficients for mixed mode are collected, when a network trained for individual classes is used. For instance, a network trained on a package of 56 images is used for IFV, and for the Tanks class – a network trained on a package of 112 images. The average processing time of a 6000x4000 pixels image in one separate single-class neural network is the same as that of a multiclass neural network, that is, four seconds.
Class of objects
IFV Military cars
SPG
Tanks Average share of target objects found Training time of one separate neural network, hours
The results of the experiments confirmed the effectiveness of the use of single-class neural networks. However, training a complex of such networks requires more computing resources than those used for training of one multi-class network. An increase in the complexity of the task with the same number of feature maps leads to a decrease in the average share of the found target objects for a single-class ANN.
The article presents the results of research related to the use of modern convolutional neural networks for processing panoramic full-color aerial ERS images. Using the nnForge and Darknet CNN, multiclass and single-class problems of searching and classifying targets are solved. Some methods for preparing training samples, optimal configuration, and the use of high-performance computing have been developed to improve the recognition efficiency. A comparative analysis showed that the oneclass approach has an advantage in recognition quality but loses in operation time. In general, it should be noted that the capabilities of convolutional neural networks allow solving simultaneously the problems of searching (localizing) and recognizing objects in ERS images with high accuracy and completeness.
This work was supported by the Russian Foundation for Basic Research (projects No. 18-29-03011-mk Research and Development of New Methods and Technologies for the Tasks of Intellectual Analysis and Optimization of Processing Large Data Streams of the Earth Remote Sensing and No. 17-29-07003-ofi_m Development of Methods and Models of Dynamic Behavior Planning and Hierarchical Intellectual Motion Control of Unmanned Aerial Vehicles in an Uncertain Environment with Computing Resources Constraints).