Investigation of informative representations of complex data is a rapidly developing field of research in machine learning. In this work we present a process of production and analysis of informative low-dimensional latent representations of real-world image data with neural network models of unsupervised generative learning. A model of convolutional autoencoder based on VGG-16 architecture was used to produce low-dimensional latent representations of aerial image data and the characteristics of distributions of several higher-level classes of terrain types were studied. The analysis of distributions demonstrated a landscape of compact concept clusters for most studied types of terrain with good separation between concept regions. The results of this work can be used in developing methods of effective learning with minimal labeled data based on the emergent concept-sensitive structure in the latent representations. Artificial Intelligence, unsupervised machine learning, clustering, image recognition, image Related Research Informative representations obtained with models of unsupervised generative self-learning were used in a number of applications to identify the concepts or classes of interest in the observable data. Artificial neural networks have strong potential in such problems and applications due to their capability of universal approximation [1,2], making them suitable for processing data of virtually any type and complexity including live image data recorded in aerial surveillance.
classification
Classification of complex data such as images represents a significant challenge in the areas with severe deficit of labels. In these problems and applications, unsupervised machine learning methods such as processing data with models of unsupervised generative self-learning has shown to be effective in identification and selection of in-formative latent representations that can simplify subsequent classification and significantly reduce the label requirement. The motivation of this work was to apply these methods to the practical task of aerial image recognition where a specific set of classes combined with a strong deficit of labels make application of standard methods of supervised classification challenging. 1.1.
2021 Copyright for this paper by its authors.
is further complicated by poor or inconsistent between different studies formalization of classes and
severe deficit of labelled samples [
Hierarchical representations of observable data were obtained in a completely unsupervised training
process with Restricted Boltzmann Machines (RBM) and Deep Belief Networks (DBN) [
Previous results in unsupervised representations with generative self-learning neural network
models include applications of deep autoencoder models of different architectures such as sparse,
variational, convolutional and others to create informative representations of image [
The motivation of this work suggested by earlier results [
To solve the problem of strong label deficit methods of creating informative representations with deep neural network models of unsupervised generative learning were applied, that are capable of learning essential patterns in the observed data in the unsupervised mode without any labeled data. The novelty of the proposed approach is a successful application of generative models to real-world data such as aerial images, allowing eventually to perform processing on board of an autonomous vehicle; secondly, designing and demonstrating the methods of evaluation and measurement of latent representations, including entirely unsupervised ones.
The dataset of images recorded in real surveillance of terrain from an aerial vehicle was chosen for two main reasons: first to demonstrate that the methods developed in this work can be applied to realistic complex data types; and not in the least, because the tasks aerial image classification and interpretation are becoming increasingly common in many practical applications.
This section contains the description of the model, data and methods used to produce and analyze the distributions of the characteristic classes of data in the input image dataset. 2.1.
The models were of the type of deep convolutional autoencoder neural networks [
Generative neural network models were used as described further in this section to produce compressed latent representations of aerial terrain observation images represented in the dataset to evaluate distributions of classes identified by a type of terrain the latent space. Methods of geometrical analysis were applied to samples transformed to latent representation to identify characteristic parameters of distributions, such as compactness and separation between distribution regions of different classes. 2.2.
The architecture diagram of the models used in this work is given in Figure 1. It can be described as a deep convolutional autoencoder neural network that contained the encoder model with several convolutional blocks, activation and normalization layers producing a flattened numerical representation with dimensionality 8 x 8 x 256; and the generator model with up-sampling blocks with the resulting output layer of the same dimensionality as the input layer.
The models were trained in the unsupervised mode, with unlabeled raw image da-ta, to reduce the generative error, i.e. the mean deviation of input images in the training dataset from their regeneration by the model as:
= (| ( ) − |) → where S is the training sample, G(S), the output of the model on the training sample.
The architecture of the model is shown in Figure 1. (1)
A trained model can produce encoded representation of the input sample X in the observable space by transforming it to the latent representation space defined by activations of the neurons in the latent layer of the model as:
= ( ) (2) where E(X) is the generating stage of the model, from observable input to the latent representation layer (Figure 1).
It needs to be noted that in a trained generative model the encoding transformation (1) is defined in completely unsupervised process and does not require for training any samples labeled with known categories of the observable data. 2.3.
The dataset of images was obtained in live aerial surveillance of the terrain with preprocessing of scaling to the standard size (64 x 64) and augmentation by rotation. Images in the dataset were classified in semi-automatic process into classes representing characteristic types of terrain with significant representation in the dataset. In the rest of the study, classes or categories of images were denoted with a symbol, such as “T” for transport tracks, “W” for wooded areas and so on.
The detailed composition of the dataset is described in Table 1.
W H F D O
The models were trained in unsupervised process with minimization of the deviation between the training set of images and their regenerations by the model. Cost functions used for unsupervised training were Mean Squared Error (MSE) and binary cross entropy (BCE), both showing strong improvement in the process of training (Figure 2).
Strong reduction of generative error in the process of unsupervised training indicated that the latent representations created by the models contained sufficient information to regenerate observable data, because in a feedforward neural network of the type used in the study the output has to be generated entirely from the information contained in the latent representation.
In this section we present the results of measurement and analysis of distributions of higher-level concepts in the original (observable) dataset in the latent representations produced with generative training of unsupervised autoencoder models as described in the previous sections. 3.1.
The characteristics of the general representative set of samples in the latent representation, without breakdown by higher-level concepts were as follows:
The analysis of principal components [
These results indicated the possibility of strong redundancy reduction in the latent representation without significant loss of information. Based on these results in the analysis of class distribution three principal dimensions with the highest variation were used, which allowed to produce direct visualizations of concept distributions in the latent representation. 3.2.
In this section the results of the measurement and analysis of concept distributions in the latent representation of generative models are presented. The parameters of distributions such as the characteristic size, standard deviation and density of the concept distribution regions in the latent representation coordinates are given relative to the maximum dimension of the overall latent dataset, and the uniform density.
For most concepts with significant representation in the dataset, a compact and well-defined character of latent concept distributions was observed with the density of the concept region (i.e. the region of distribution of samples associated with the studied concept in the latent representation space) significantly higher than uniform.
These results confirm the earlier observed effect of correlation between unsupervised latent
distributions and higher-level concepts with strong representation in the observable dataset [
In this section, measurements related to categorization capacity of the models are presented and discussed. Shown in Table 3 are visualizations of intersections of concept regions, with the highest relative volume. A cross-concept intersection matrix can be defined to indicate the degree of disentanglement of concept regions in the representation, as the ratio of the latent volume of the overlapping region between concepts A and B, Oa,b to the volume of the concept region A, Ha: ( , ) (2) , =
( )
As can be seen from the results in Table 3, a good separation of concept regions was observed for most categories of images with significant representation in the dataset, indicating strong categorization achieved by the models in the process of unsupervised generative learning.
Unsupervised categorization or decoupling of higher-level concepts in the unsupervised latent
representations is the effect observed in a number of experiments with unsupervised self-learning
models [
Accordingly, categorized distributions tend to minimize the volume and consequently, maximize the relative density of latent concept distributions while minimizing the overlapping between the distributions of different concepts. 3.4.
Models that were successful in identifying characteristic patterns or concepts in the observable data as a result of generative self-learning can be expected to be able to regenerate input samples with close resemblance / low deviation from original input samples. To evaluate generative ability of the models, experiments were performed with samples of the concepts present in the training dataset, as well as those that were not classified into concepts (i.e. general background). Examples of generative results of the models are shown in Table 4.
Table 4
A successful generative ability across multiple classes of data indicates that the distributions in the latent representation were correlated with characteristic patterns in the original (i.e. observable) data. It follows from the architecture of feed-forward artificial neural networks that all the information necessary for generation of the output of the model must be contained in the latent representation layer and there-fore the process of unsupervised generative training was able to produce informative representation with substantially reduced redundancy in the observable data represented in the training dataset.
Generative ability of self-learning models such as those studied in this work can be used in augmentation of training datasets for models of supervised learning, to improve the accuracy and extend classification ability to classes of data under represented in the datasets. It will be investigated in more detail in another study.
The results presented in this work are in agreement with the growing number of reports with
observations of the effects of concept-correlated latent representations emerging in unsupervised
generative learning and provide strong additional arguments in support of the general character of this
effect. Given the wide range of models and types of data, from lower complexity [
It was demonstrated that under certain constraints such as generative accuracy and information compression or redundancy reduction, low-dimensional latent representations of models learning to generate input distributions in a completely unsupervised learning process can produce distinct structure that is correlated with principal higher-level concepts in the observable data. The results in Sections 3.2, 3.3 on measurement of latent distributions of main concept regions appear to support this conclusion.
Identification and description of unsupervised latent structure that emerges in generative learning can be a valuable instrument in the analysis of general data, in particular, of less known origin where massive prior knowledge such as large labeled datasets used in supervised machine learning may not be available.
A number of recent results indicated that similar low-dimensional representations can play an
important role in processing of sensory data by humans [
Based on the results reported in this work, several essential observations can be made on the process
of unsupervised generative learning with real-work image data and the conceptual structure in the latent
representations of data produced by such models:
1. The models of generative unsupervised learning used in the study were capable of producing
well-defined categorized representations correlated with the principal (i.e. strongly represented)
higher-level concepts in the training dataset.
2. The observed latent representations showed good categorization and separation of principal
concepts and appear to support the hypothesis [
Overall, the observed latent representations showed good categorization and separation of principal
concepts and appear to support the hypothesis [
The methods of evaluation of latent distributions of data classes, demonstrated and verified in this work are of general nature not limited to a specific type of data and can be instrumental in evaluation of the learning capacity and performance of generative models. The unsupervised latent structure demonstrated in this and other works can be used to enhance learning ability of the models in the environments with strong deficit of labels. These findings can therefore be instrumental in development of learning models and methods that are capable of acquiring knowledge in a flexible and environmentdriven process that is closer to learning of biological systems. [22] Hassabis D., Kumaran D., Summerfield C. and Botvinick M., “Neuroscience inspired Artificial Intelligence”, Neuron vol. 95, 245-258, 2017.