Modeling and analysis of small data raises essential problems and challenges stemming from insufficient sampling of unknown observable distributions, complicating and exacerbating confident analysis and often reducing statistical confidence of the conclusions. In this work, an original approach to analysis of small data is proposed that is based on an ensemble of generative neural network models with the intent of identifying stable clusters of data in informative generative representations. We demonstrate how characteristic structure of stable clusters in generative representations of a dataset of images of basic geometric shapes can be determined from representations produced by a generative ensemble. The method can be used to identify characteristics structure, perform correlation analysis and augment data of different types and under some conditions that were discussed, improve the performance of supervised classification in cases with a deficit of training data.
Modeling and analysis of small data raises essential problems and challenges stemming from
insufficient sampling of unknown observable distributions, complicating and exacerbating confident
analysis and often producing lower statistical confidence of the conclusions. Nevertheless, early
analysis of structure and trends in emerging data can be essential in situations and events of novel or
rare nature / condition where large volumes of confident decisions may not be available for any
reason [
Among the admitted challenges in practical applications of methods of machine intelligence in the
analysis of small data are those of stability of learning and produced results. It can be observed for
example, as a strong dependency of the learning success on the choice of training parameters,
selection, temporal ordering of batches and other training factors. Issues that have been noted [
Numerous efforts attempted to examine the problem of stability of small data learning and a
number of promising approaches and directions described, including: cross-validation; ensemble
methods [
In parallel to these developments, methods of unsupervised generative learning [
To address these challenges as outlined above, we propose the ensemble approach [
An ensemble of generative neural network models with an architecture of a deep convolutional autoencoder with a strongly compressed representation layer was used to produce two or threedimensional representations of small datasets of images of basic geometrical shapes.
The advantage of the proposed architecture stems from previous applications in producing
informative low-dimensional representations of complex data as well as universal approximation
capacity of neural network models [
Following successful generative training of the models in the ensemble, embedded distributions of the evaluated datasets were produced in the spaces of latent coordinates. We then attempted to identify stable populations of recognizable latent structures, such as visually identifiable clusters. The resulting structure of identified stable clusters had to be invariant with respect to individual training model and therefore represent innate characteristics of the input data that was possible to verify due to known composition of the dataset. 2.1.
A deep convolutional autoencoder neural [
Overall, generative autoencoder models in this work had 48,000 – 96,000 trainable parameters
depending on configuration of layers. The decoding or generating stage was fully symmetrical to the
encoder. The models were implemented in Tensorflow / Keras [
An architecture diagram of the model is shown in Figure 1. 2.3.
We decided to approach the question of stability of learning with small data with a set (an ensemble) of generative neural network models that do not require prior knowledge, including in the form of labeled data, for successful training. An ensemble of trained generative models of a size n thus produced an array of two-dimensional representations of the input data as shown in Figure 3.
As a result of this phase of unsupervised generative ensemble learning was produced a set of pairs: R = { (trained model, map(input data point, 2D latent position) × n }. Association of a unique id to a latent position of an input data point (Figure 2, e1) relative to that of the other points in the set allowed to identify stable clusters K in the input data by entirely unsupervised process as follows (pseudocode, where D is input dataset, E: encoding phase of the generative model, where e(x) = E(x) encoded (latent) image of observable data point, m: the number of identified clusters): for x(k) in D: if ek = E(x(k)), ek-l in Kl : K(x) = Kl (known cluster) else if conf(ek, L) > γ : K(x) = L; m: = m+1 (a new cluster) else: K(x) = A (not in cluster) where A: an arbitrary id for elements with uncertain cluster; γ: confidence of identification of the latent position ek belonging to cluster L. The process can be described as follows: if the next element is in the same cluster as an earlier one (i.e., with a lower sequential id in the dataset), the cluster of that element is assigned. If cluster of the element is uncertain, an arbitrary constant number is assigned; finally, if neither of the conditions is satisfied, a new cluster is assigned and the process is repeated until the dataset is exhausted. This process is deterministic, and as can be seen does not depend on selection of ordering sequence, as long as the same sequence is maintained throughout the process.
There result is a matrix K(D) of (data point, cluster id) pairs of a dimension (M × n) where M is the size of the dataset and n, the of the ensemble of generative models, where points in the same cluster (including uncertain cluster association) have the same cluster id.
In the final step one can obtained the set of stable clusters Ks(D) identified by the ensemble as a subset of K(D) satisfying certain confidence criteria cs: (1)
For example, if correlation of K(x) in the matrix K(D) was found to be 0.9 (i.e., 9 out of 10 models in the ensemble produced the same cluster id for a given element) and the size of the ensemble, 20 then the 95% confidence interval for the correlation coefficient of the element and the cluster would be [0.76, 0.96] indicating a strong and confident association of an element to a cluster.
The resulting subset of stable clusters Ks(D) identified in the described process can be used in an analysis of composition of the input data and a number of other applications as discussed in the subsequent sections. It may be worth reiterating that at no point in the analysis any true known samples of classes in the input dataset were used.
The results of evaluation of the cluster structure with the datasets of images in a process described
in the preceding section are presented in Table 2 and Table 3. Identification of clusters was performed
by visual analysis (examples in Figure 3) that demonstrate both stability and accuracy of the
identified cluster structure with the data in the study. In the future, unsupervised clustering methods
such as DbScan [
Table 2 Cluster composition, G-150 and G-300 datasets, n(1) = 20
at conf = 0.8 1.0 1.0
separation
high high, very high (1) Ensemble size (2) Fraction of the dataset in identified stable clusters, at confidence level Table 3
With a large unsupervised dataset, from G-150 to G-300 significantly improved accuracy of the cluster to type association was observed, rising to the level of 95 – 100%. Stability of the latent structure of clusters is a key observation and a necessary requirement for successful generation of new data, confirming that clusters identified by the unsupervised generative ensemble method indeed described stable characteristic patterns in the observable data. 3.2.
The architecture of generative models provides a direct way to propagate positions in the latent space of generative models to the space of input (observable) parameters. The mapping can be obtained by taking a latent position with coordinates p = (l1, l2) as input to the generator component of the model (Figure 1) to produce an observable position Xobs: (2) where G: R O, is the generator component of the model, operating from the latent space R into the space of inputs, O.
Based on (2) generative ability of successfully trained generative models can be used to create new data points with “similarity” to identified characteristic patterns by selecting positions in the latent regions of identified stable clusters as illustrated in Figure 4.
The effectiveness of the proposed method of ensemble cluster-based data augmentation can be supported by the arguments:
Consider a small data set S of size N with p input parameters. With a conventional method of approximation, e.g. Gaussian, the error of the mean in each of the parameters of the data can be estimated as: pmean / √N [21]. Where N is small the dispersion of observable parameters can be sufficiently large. Next, if a successful generative representation of a lower dimensionality d with a good cluster structure existed, the data can be approximated with a good accuracy by a quasi-multimodal distribution with the number of modes Nc, where Nc: the number of stable latent clusters, and the dispersion, dmean / √nclus (the size of a latent cluster). Where d is small: (i.e., a strong reduction of dimension of observable data) and the number of samples in the principal clusters, sufficiently large one obtains a statistical problem of significantly lower complexity and dispersion. As an example, in this study, the reduction of dimensionality from 4,096 input parameters (grayscale images with resolution 64 × 64) to 2, i.e. by a factor of ~2,000 was achieved.
Decomposition of unsupervised datasets into a structure of stable latent clusters, where successful, can be helpful in the analysis of the distribution of data and associated factors of interest. As discussed earlier, the proposed approach offers a general, independent of specific types of data capability to decompose observable data into a structure of more homogenous regions, or clusters. Even without known samples of classes of interest, distributions of clusters, both latent and observable can be analyzed in detail, including observation of characteristic representatives of clusters, prototypes.
Generative ability of models in the study combined with cluster decomposition of informative representations allows non-trivial analysis of composition of input data by generating typical observable instances of stable clusters, or prototypes of characteristic natural classes of data. With stable clusters identified with proposed methods, one can generate specific latent positions associated with clusters for example, as a mean of cluster member positions and propagate them to the observable space with generative transformation (2). (3) where K: stable latent cluster, P(K): observable prototype. Examples of observable prototypes of clusters in the dataset G-150 are shown in Figure 5.
Based on the discussion in Section 3.2 cluster decomposition can be used to generate new data points and augment small datasets, again without any limitations on the prior knowledge of the distribution of the input data. Once a structure of stable latent clusters has been identified, it is straightforward to determine their distribution regions and produce latent candidate positions for augmentation of the original data. Generative transformation (2) can then be applied to obtain the related data points in the space of observable parameters.
To summarize the results and discussion on data generation, ensemble-based generative augmentation of small data can be successful under these conditions: The latent dimensionality is sufficiently small: . The models demonstrate good learning success and consistent, stable, cluster structure. The number of stable clusters is small compared to the size the dataset (the number of samples), and the population of main clusters is sufficiently large: N / nclus 1.
If the conditions are satisfied, the original data can be described by a multi-modal distribution of stable latent clusters that can be identified with density clustering or another method as demonstrated earlier; further, augmentation of data can be performed in an unsupervised process based on the identified distribution in the latent space and will provide stable results invariant with respect to the selection of a specific instance of generative model. 4.3.
The method of augmentation based on unsupervised cluster structure produced in generative learning can be employed to improve the success of classification with models of supervised learning trained with small datasets, as in the conventional practice of supervised learning, the size and representative quality of training data can have strong influence on the accuracy of classification [22].
The success of the method essentially depends on the presence of a correlation between stable latent structure of clusters and the factor of interest for classification that can be used as a label in supervised learning. If such a correlation can be established between the data points in identified stable clusters and the factor of interest as discussed in Section 4.1, an augmentation process outlined in Sections 3.2, 4.2 can be applied, with class labels assigned to generated data points based on the established association between latent clusters and known classes. Such a process of augmentation can produce an improvement in classification accuracy due to larger and more representative dataset in supervised learning.
For example, a clustering analysis performed with the ECE dataset [
The method of identification of unsupervised cluster structure in generative representations of small datasets with an ensemble of unsupervised generative models has been described and verified with small datasets of visual data of basic geometrical shapes. It was shown that a stable structure of clusters can be identified in the latent representations of successful generative models with high confidence in the interval 90–100% with the dataset used in the study. The structure of stable clusters representing characteristic types in the input data can be used to augment small datasets by generating new data points with several potential applications, including enhancement of labeled datasets with an objective to improve the success of classification in supervised learning.
It needs to be noted that the method described in this work may not have universal applicability to all datasets and its effectiveness is defined not only by the observable parameters, but also by the composition and characteristics of the dataset such as: the size, the number and population of principal clusters in the latent representation of the data and their correlation with the factors of interest. Where the conditions described in Section 3.2 are met, augmentation can produce additional data points associated with principal clusters and improve the performance of conventional classification methods trained with augmented datasets.
In novel, rare, non-standard cases, events and environments, large volumes of known data may be needed for confident analysis with conventional methods of machine learning. Availability of training data may present strong challenges in such cases. Methods of unsupervised generative learning, including the ensemble approach presented in this work, can be successful in identification of characteristic structure and patterns even with smaller sets of observable data without requirements of massive prior knowledge offering a practical direction toward improving the confidence of the analysis.
[21] Wendland H.: Scattered data approximation. Cambridge University Press 9 (2005). [22] Richards J.A.: Supervised classification techniques. In: Remote Sensing Digital Image Analysis.
Springer, Berlin, Heidelberg 247–318 (2013).