Recognition of terrain elements and objects from aerial image data wich obtained from aircraft air surveillance cameras, UAVs and high-resolution satellite images is a complex task and requires numerical factors to be taken into account. The created training set for intelligent systems AERIAL SURVEY allows solving a wide range of scientific and practical problems in the field of machine learning and neural network recognition of terrain elements. The purpose of this paper is to show the dynamics of the conducted research on the creation of the training set AERIAL SURVEY, identify problem areas and identify promising vectors for further development of this area. The training set already now allows solving a number of video analytics tasks, including dual-use ones: various aerial monitoring of roads, agricultural fields, forests, water bodies, aerial reconnaissance with the search for targets in the combat zone, use in tracking and target designation systems. In the course of further research, the aerial survey data set will allow solving such problems as information filtering of excessive information content of terrain images, automated formalization and description of new unlabeled classes, the transition from recognition of individual terrain elements to a structural description of the aerial survey area in order to automate its georeferencing and many others.
The success of the widespread introduction of artificial intelligence technologies over the past decade is largely due to the development of machine learning methods, in particular, such a component as deep learning. That is why the value of data sets, on the basis of which the creation and training of intelligent systems for various purposes, is increasing. In recent years, issues related to data collection and their preparation for learning have already formed into an integral part of a separate area - machine learning engineering. The result of its deployment and implementation depends on how adequate these goals are for the sake of which a machine learning system or model is created. At any time, with machine learning, the researcher is not limited to a linear data processing algorithm. The learning model can be changed, improved depending on the purpose and circumstances of the applied task, and therefore requires the introduction of additional hyperparameters or the involvement of additional data. Of course, we are talking about complex applied problems, the solution of which is faced with the need to solve scientific and technical problems.
One of these tasks is the recognition of terrain elements and objects from aerial photography data obtained from aircraft surveillance cameras, UAVs and high-resolution satellite images as a special case of Earth remote sensing data. The complexity of this task is due to a large number of factors, so here are the most important ones (the order of mention is not critical):
2022 Copyright for this paper by its authors. • Various shooting heights lead to the necessity to recognize objects at different scales and with different levels of detail;
• Recognition of objects during different seasons, under meteorological conditions and period of the day; • Different shooting angle of objects creates perspective distortion of recognition targets; • Interference that occurs when shooting: from aberration or micro-movement of the fixation camera on board the aircraft to distortion from ground-based means, such as electronic warfare; • The necessity to identify not only objects in the image, but also often the solution of related tasks, such as setting the location of the shooting frame, the speed of processing a large amount of information per unit of time, preferably in near real time, etc.
In the complex solution of the problem of recognition of aerial data, special attention is required to the formation of a high-quality training set, on the basis of which it is already possible to form a problem statement for the creation of various machine learning systems, both surface and deep.
The relevance of the work on creating a training set of aerial survey data is especially important for Ukraine, as an aviation State that actively develops and uses various unmanned arial vehicles. With the outbreak of war with the Russian Federation, the use of recognition systems in the processing of aerial reconnaissance and air surveillance data in the combat zone acquired particular importance. According to the order [1] of the Cabinet of Ministery of Ukraine dated 08.30.2017 No. 600-r “Some issues of the development of critical technologies in the field of production and military equipment” and its Adition 1 as amended on 23.02.2022, in the list of critical technologies in the field of production of weapons and military techniques include but are not limited to:
• Technologies for figurative interpretation, selection and classification of targets for homing systems of high-precision weapons; • Technologies for controlling robotic platforms, identifying, recognizing and tracking targets; • Technologies of machine learning, artificial intelligence, neural networks for the design, production and operation of military and special equipment;
• Technologies for coding, transmission and receipt (automatic recognition, processing, analysis, generation, visualization) of information.
As you can see, it is difficult to overestimate the importance of creating, filling and using a training set for recognition of data from aerial surveillance cameras (aerial photography data).
That is why, starting from 2018, the Department of Applied Mathematics of the National Aviation
University [
Consider, according to open sources, the existing developments in the field of training sets of aerial observation data (or may be suitable for such processing) and pay attention to deep learning technologies that have the prospect of being used when working with the mentioned sets.
Conventionally, the formation of sets can be divided into two options: 1. Formation of the most universal set for a wider coverage of possible tasks. 2. Formation of a dataset for a specific task.
Of course, the first option is a priority in case of an excess of time, and the second - in case of its
absence. So in [
In general, a large amount of data is currently freely available. For example, since 2006, the
Environment Agency has been publishing sets collected on a plane from a few square kilometers to
hundreds. Images are provided as a raster dataset in ECW (extended compressed wavelet), as true color
(RGB), near infrared (NIR), or 4-band (RGBN) dataset. If the image was taken under incident response
conditions and/or lighting conditions not be optimal, it is identified by the IR prefix. This set is
constantly updated (last updated March 28, 2022) [
To work with a data set, those types of neural networks that are typical in image processing are
suitable: convolutional (CNN), with an autoencoder (autoencoder), generative competitive networks
(GAN). Thus, it is relevant to process the set using the VGG-16 network, which is an improved version
of AlexNet [
The YOLO network [
At the moment, the images in the “Aerial survey” dataset, which are the subject of this publication, are divided into 16 classes and have the structure shown in Fig. 1.
In total, the dataset consists of almost eighty thousand 64x64 pixel images. In 2018, the first iteration of the formation of the dataset and work on image processing on it was carried out. At that time, the set consisted of about two thousand images with a 64x64 pixel extension, divided into four classes: water, road, building, target class [18, CNN-technology for recognition of objects for aerial photography data]. The study used a convolutional neural network consisting of two convolutional blocks and a multilayer perceptron. Three experiments were carried out in which the dataset was divided into training and test parts of different sizes, and the neural network was trained for a different number of epochs (50, 60 and 80). In each of them, the training and test data were shuffled randomly five times. The summarized results are presented below in Tab. 1.
As of 2019, the dataset has been updated to 16,000 images divided into eleven classes: buildings, forests, fields, water bodies, roads, vehicle traces, civil engineering, poles, and three target classes. In addition, the study was also carried out on a new type of networks - a convolutional autoencoder with a classifier [18, Automated object recognition system based on aerial survey]. The results of the classifier, in the most general form, are presented in the diagram (Fig. 3) The results of the classifier are presented in more detail in the diagram (Fig. 2).
Also, work continued with the convolutional network - the classifier. The number of images was additionally added and augmented (up to 16 thousand), and the classes were redistributed as follows: buildings, buildings between trees, forest, roads, roads between buildings, roads between trees, vegetation field, non-vegetation field, waters between trees, technics footprints, trenches, military and civil technics[18, Neural network recognition of image classes from aerial survey]. The work done, in general, contributed to the quality of recognition (Fig. 3).
In 2020, the number of images in the studies was increased to 22,000. In addition, as the analysis of the results of previous recognition works (Fig. 2, 3) shows, the “technics footprints” class, which is closely related to the recognition of roads, in particular dirt roads, required special attention. and different paths. Therefore, in [18, Information technologies for recognizing road classes based on aerial survey data], new elements of the dataset with different types of roads were created, about 40 thousand of the total number of images. Subsequently, a more detailed analysis of representations was carried out for architectures based on the autoencoder model. In the same work, for the representations obtained from the encoder, the principal component method was applied, according to which it turned out that the first three components contain more than 50% of the variability, and the first of them accounted for 28.9%, the second 12.3%, the third 8.9%.
Through the application of the principal components method, a two-dimensional and threedimensional model of data variability was built (Fig. 4). In the course of testing the classifier on different types of roads, the histogram of the top and bottom transitions was taken away (Fig. 5). In addition, the possibility of presenting road classes in the representation space to the autoencoder in the form of actually linearly separated sets was demonstrated (Tab. 2) “Other” - blue color; "Roads between buildings" - pink color. "Roads between trees" - a shade of green; "Narrow dirt road" - green color; "Roads between buildings " - purple color. "Other" - blue color; "Roads between trees" - a shade of green. "Roads between trees" - a shade of green; "Asphalt roads" - red color; "Roads between buildings " - purple color. "Roads between the trees" - a shade of green; "Trails" - yellow color; "Asphalt roads" - red color.
An interesting experience was the study of the generative network model (GAN). About twenty thousand images were processed by the network, after which a new sample was generated [18, Technology of neural network generation of a training set of digital images]. Both samples during the research served as both training and test for the convolutional network of the classifier. As can be seen from Table 3, the classifier showed the best recognition accuracy when trained on data generated by the GAN.
In addition, the possibility of presenting road classes in the representation space to the autoencoder in the form of actually linearly separated sets was demonstrated (Tab. 2).
In addition, studies of individual classes were carried out. For example, the analysis of the recognition of specific forest plantations and individual trees [18, Automation of filling the training sample according to aerial observation data]. This work is interesting because the qualitative recognition of just such a class makes it possible to automate the calculation of forest plantations, as well as more reliable recognition of "mixed" classes, for example: buildings among trees or trees (forest plantations) along roads. In 2021, the data from the set was used to train modern architecture - MASK R-CNN. The network was used to detect buildings on live video. Positive results were obtained, even with the degree of Jaccard (IoU) 0.8, accuracy and completeness of recognition reached approximately 0.6.
Also at this stage, studies were carried out to identify data that were not formalized in the training set. That is, images with objects that were not present in the training set were attached to the test sets [18, Identification of aerial survey data classes not formalized in the training set]. The result of 36 experiments was demonstrated to add a fourth (unknown) to the neural network classifier trained in three classes. On average, for the case of adding one non-formalized class to three, its identification was carried out with an accuracy of ~68%. And in the case of adding one formalized class to four classes, approximately 63%.
Research continued on the representations obtained by using the encryption part of the autoencoder. In particular, the representations were subject to analysis, in which they were clustered by the k -means algorithm into a different number (from 10 to 17) of newly formed classes, compared with the initial eleven [18, Information technology for modeling the initial datasets of aerospace surveys based on neuromeasurement architectures with an autoencoder]. As an encoder architecture that solves the problem of data dimensionality reduction, the following network was built (Fig. 6).
The refore, using the PCA, the following representations were obtained (see Fig. 7). After applying the PCA, it was found that the first 10 principal components account for 74% of the data variability. The use of PCA made it possible to carry out cluster analysis in the feature space. An analysis of the dependence of the choice of the optimal clustering parameter and repeated retraining of the network on the resulting clusters was carried out. It is shown that using this approach, we obtain a comparable level of recognition - up to 97% on the test sample, and this opens up the possibility of researching approaches in the field of building self-learning systems.
In several works, attention was paid to automating the filling of the training set of images. In the master's work [18, Information technology for classification of terrain types and object search based on aerial survey data], the idea was to carry out the initial segmentation of images, after which the selected clusters were used to automatically generate new classes for the training set. The disadvantage of this approach is that the training data obtained in this way must be reviewed by a human specialist and assigned a name to the class. However, as the practice of applying this approach has shown, in many cases this approach has significantly increased the number of images for such types of images: forests, fields, buildings, large water bodies. A more adequate automation of data set filling was demonstrated in a thesis [18, Information technology for automating the filling of the training set for neural network recognition]. In this case, an automated application was created that used the results of neural network training to process aerial data. Recognized image fragments were automatically marked up and could be added to an existing set of training images. Testing has shown that this approach reduces the cost of data labeling and improves the quality of recognition after training on the updated training set.
As even a brief review of the work performed shows, the availability of the Aerial Survey training set allows a wide range of scientific and practical research in the field of machine learning and neural network recognition of terrain elements. On the applied side, the presence of such a set already now allows solving a number of video analytics tasks, including dual-use ones: various aerial monitoring of roads, agricultural fields, forest plantations, water bodies, conducting aerial reconnaissance with the search for targets in the combat zone actions, use in tracking and target designation systems.
In addition to the indicated practical, the authors of this work also have a number of purely scientific interests and plans for working with the set. Let us outline some of them and a general perspective for further research.
If we strive for the universality of this training set, then we should take into account the need to label data for different seasons, which will lead to significant data heterogeneity within some classes, for example, trees, roads, fields, etc. In addition, even within Ukraine, the structure and elements of the landscape, vegetation, and even buildings are noticeably different. Increasing the class options presented in the set requires research into the impact on the quality of training and recognition. Perhaps a separate study will be the selection of some "core" of base classes within the overall architecture of the training set, using for "additional learning" classes of images characteristic of a particular season or type of terrain. In this context, the experience of automatic replenishment of the dataset, which can be performed on the basis of a stable training “core”, with adjustment to a specific monitoring task, is useful. Another area of research may be work on the generation of artificial data, on the principle of generative networks, or on the basis of research and use of representations in the autoencoder space. Working with low-dimensional representations also allows the study and primary analysis of the distributions of data presented in individual classes. The influence of heterogeneity or the presence of anomalies on the learning process is quite an interesting and promising area of research.
Information filtering of excessive information content of terrain images, automated formalization and description of new, unlabeled classes, the transition from the recognition of individual terrain elements to the structural description of the aerial survey area in order to automate its georeferencing this is not a complete list of tasks that can be performed with the “Aerial survey” dataset in further studies.
The authors express their gratitude to the Aerorozvidka public association [