Informative representations play an important role in learning and intelligent functions of artificial and biological systems. Sparsity constraint in neural network models has been known to be effective in producing informative representations of sensory data. In this work, the structure in low-dimensional representations created by a class of generative neural network models of unsupervised learning was analyzed to establish the relation between the sparsity constraint imposed in the representation layer and the effectiveness of unsupervised concept learning. It was demonstrated that sparsity constraint allows to achieve two essential objectives in successful realistic learning: increasing the effective conceptual capacity of latent representations, while simultaneously limiting the range of latent activations. Sparsity emerges as a simple and effective mechanism of producing functional conceptual representations of complex sensory data in both artificial and biological systems. The results provide empirical support for the connection between unsupervised generative learning and conceptual latent representations correlated with characteristic patterns in the sensory inputs and clarify the role of the sparsity constraint in improving the quality and conceptual capacity in generative representation learning.
Unsupervised representations obtained with models of
generative self-learning were studied for considerable time
with the intent to identify and separate informative
components and patterns in the data to improve performance
of learning models. Hierarchical representations obtained
with Restricted Botzmann Machines (RBM), Deep Belief
Networks (DBN) [
These and a number of other results [
In a number of results, sparsity constraint [
The paper is organized as follows: in Section 2 the model and data used in the study are described. Section 3 contains the results of experiments with generative models with sparsity constraint (sparse representation layer) and those without it (flat representation layer) are presented. Section 4 contains a brief discussion of results and the connections to the current state of the research in the field. 2
The models used to produce unsupervised latent representations of images modeling visual sensory inputs had the architecture of a convolutional autoencoder [15] with strong dimensionality reduction in the encoding layer producing latent representation with the coordinates of the activations of encoding neurons. The data was represented by a dataset of images of basic geometric shapes, greyscale and color, as described in this section. 2.1
The models were of a common type of a convolutional autoencoder, with the addition of a deep dimensionality reduction stage producing latent representation as shown in Figure 1. The dimension of the latent layer was variable and different in two studied classes of models. The first class (denoted ”F”) had a flat latent layer with the dimension of 3 to 10 chosen based on discussed results in sensory processing in biological systems. These models did not have sparsity constraint on activations in the latent layer. The second class of models, ”S” had the latent dimension of up to 10, with sparsity constraint imposed in the latent layer as L1 regularization [16] limiting the norm of activations of the encoding neurons producing sparse latent representation. The models were implemented with Keras/Tensorflow [17], for measurement and visualization of distributions common machine learning libraries and packages were used.
The compression of information achieved in the representation layer of the model was approximately 1,300 for greyscale and 4,000 for color images of size (64,64). An advantage of the chosen architecture is that it allowed to measure and visualize the distributions of data in the latent representation directly by visualizing activations of neurons in the encoding layer of the model. Two datasets of images of basic geometric shapes of the size 64 64 pixels, greyscale and color were used. The first dataset, Shapes-G contained the total of 600 – 1,000 greyscale images of circles, triangles and backgrounds with variation in size in the range 0.3 – 1.0 of the image size (that is, 0.3 64 pixels), with variation of contrast of fore- vs. background for each size.
The second dataset (Shapes-C) was a color version of Shapes-G with the total of 1,200 images of the following combinations: red and blue circles of varying size with grey background of varying contrast; red and blue triangles with varying size and background; red and blue horizontal bands (wide red bands, narrow blue bands), with varying background; and empty grey backgrounds with varying contrast.
The composition of the dataset allowed to experiment with
data of different complexity: while the conceptual content
of the first dataset (Shapes-G, including grey backgrounds)
was three concepts, for the Shapes-C data it was 7 (with
wide and narrow bands of different color considered as
different concepts).
The models were trained in an unsupervised autoencoder
mode to achieve good reproduction of inputs measured by
the cost function, such as Mean Squared Error (MSE) and
binary cross-entropy (BCE). Several criteria of
effectiveness of unsupervised training were used, such as
monitoring the cost function and cross-categorical accuracy
that both shown significant improvement in unsupervised
training with minimization of the generative error.
Additionally, generative performance of trained models was
measured by comparing a subset of input images to their
reproduction by trained models as illustrated in Figure 2.
A trained model performs the encoding transformation
from the observable data space X 2 O to the latent
representation y 2 R obtained with the activations of the latent
layer of the encoder model E : O ! R, and the
generative transformation from the latent representation to the
observable space G : R ! O as:
y = E(X ); X 0(y) = G(y)
(1)
where X 0 is the generated image or interpretation of input
X by the model. The distance between the original input
X and its interpretation X 0 then indicates the ability to
interpret inputs in the observable data and training of a
generative model is equivalent to minimization of the norm
jjX ; X 0 = G(E(X ))jj defined by the cost function on the
training set of samples in the observable space.
The structure in the latent representation that emerges as a
result of unsupervised training can be observed and
evaluated with a number of methods:
1. By applying unsupervised clustering such as
DBScan, MeanShift and variations [18] in the
representation to identify density distribution in general
unlabeled data sample as well as concept samples.
2. By evaluating distributions of general and concept
data samples in the representation space via direct
observation, multi-dimensional histograms and other
methods.
3. By producing observable images of latent positions
with the methods of generative probing and scanning
[
Sparsity constraint was imposed in the encoding layer of
the model as L1 regularization with the value of 10
510 6. Level one regularization means a constraint on the
sum of absolute values of activations of the neurons in
the encoding layer and in most cases resulted in no more
than two to three neurons being active at any interpretation
task, consistent with the results in visual sensory system in
humans that indicated sparse, low-dimensional activation
pattern in interpretation of visual data [
To examine the effect of sparsity constraint on unsupervised latent representations, methods of comparative analysis were applied to representations created by models of similar neural network architecture with sparsity constraint imposed in the representation layer and without it. This approach allowed to clarify the role and demonstrate the significance of sparsity constraint for the ability of the learning model to create effective conceptual representations of the training data.
As described in the previous section, the success of generative learning was judged by the improvement in the value of the cost function and by the quality of reproduction of images, similar to those the models were trained with. The outcomes of two evaluations were mostly consistent, that is, models with significant improvement in the cost function had better interpretation quality and vice versa, those with better generative quality also had better training metrics.
In this experiment, training results of models of
different architectures, flat and sparse, were compared with the
dataset Shapes-C of higher conceptual complexity. Ten
models of each type were trained independently with the
dataset Shapes-C with evaluation of learning quality
factors, such as:
• Learning success, the fraction of models that were
able to interpret a representative set of images
successfully
• Generative quality indicated by a factor in the range
[
These results indicated that sparsity constraint imposed in the representation layer allowed models to maintain the effectiveness of learning with more complex data of higher conceptual content, whereas flat models were either significantly less successful in training, or required more resources such as higher activations and / or the number of active synapses. The causes of this difference will be discussed in the following sections.
Another important conclusion that can be drawn based on results in this section and the earlier ones obtained with the greyscale dataset is that for a given complexity of data and model architecture there exists a certain limit or threshold Tf of conceptual content that can be learned successfully in a given low-dimensional subspace of the representation space defined by the coordinates of activations of active neurons. If and when the threshold is exceeded, in a flat model with realistic activations constraints (F3-AL, Table 1), the learning performance begins to deteriorate. In the next section, the role of activation constraint will be discussed in more detail. 3.2
For a realistic learning system, biological or artificial operating autonomously in the real world, the range of activations up to the maximum values can be essential as it directly translates into the operational cost of the system in both energy and physical resources. Particularly, very strong activations expressed in a biological system as an electric voltage may require significant resources to maintain currents under control. Thus keeping activations within a limited range and preventing high activation values can be an essential objective and a constraint for a realistic learning system.
In the experiments with the models in this work it was observed that whereas flat and sparse models were able to attain similar levels of generative quality with the data of limited conceptual content (dataset Shapes-G), an essential difference was found with data of higher conceptual complexity in the dataset of color images, Shapes-C. Successful flat models with good generative quality, without sparsity constraint in the representation layer were significantly more rare (as indicated by the learning success factor, Table 1) and produced significantly higher activations, up to an order of magnitude higher than recorded in the sparse models.
A straightforward explanation of this finding can be given
based on the earlier results [
From these results and the results of section 2.2 it follows that increasing conceptual complexity of the data from 3 to 7 resulted in a strong deterioration of the learning ability of the flat models pointing at the value of the conceptual threshold for low-dimensional flat models, 3 Tf < 7. It also showed that sparsity offered a simple and effective solution to the high activations problem, allowing to ”pack” higher conceptual content within the allowed range of activations. 3.3
Methods of topological analysis of latent representations
such as latent probing and scanning developed in [
Producing latent scans of representations of sparse generative models trained with dataset Shapes-C allowed to confirm this hypothesis and explicitly describe the structure of the resulting sparse representation space. The method involves producing an array (a hypercube) of images generated by a trained model from latent positions in a grid associated with a latent region of interest; by adjusting parameters of the grid it allows to describe the generative structure that emerges in the latent space as a result of unsupervised learning to any level of detail.
To identify the structure, sparse models with dimensionality d of the encoding layer that were trained with Shapes-C were presented with a subset of images representing characteristic patterns in the dataset, T . By recording latent positions l(T ) = E(T ) as defined by equation (1), it was possible to detect the allocation of concepts in the dataset to slices w(d; 3) in the d-dimensional latent representation space. The examples of distributions of concepts in d-slices for d = 5; 8; 10 are shown in Table 2, with slices indexed by the tuples of indices of participating neurons, i = 1 : : : d: As can be seen from the results in Table 2, sparse architecture of the representation layer indeed distributed the patterns in the training data quite efficiently between available slices in the stacked representation space. The stacked structure of the latent representations was exhibited by all sparse models though different arrangement of slices could be used by individual models, for example, another S10 model used slices (3,4,10) and (2,9,10) for red and blue circles respectively, indicating that slice allocation is determined by the training process as is not invariant between individual models trained with the same data.
With the slices of interest in the d-dimensional
representation space thus identified, in the next step latent
scanning was applied in the slices associated with
representative samples of visual concepts to examine the generative
structure in the slice subspaces identified by neuron
indices w = (i; j; k). By producing generative scans of the
slices as an array of observable images one can
understand how activations of specific neurons are interpreted
by a sparse model. The examples of generative scans
of a sparse representation, model S10 and a flat
threedimensional representation (model F3, no activation
constraint) are shown in Figures 4 and 5 respectively.
Clearly, the sparse model distributed the concepts in the
training data between different low-dimensional slices in
its 10-dimensional latent representation space as shown
in Fig.4, in direct confirmation of the stacked structure of
sparse representations hypothesis discussed earlier. On the
other hand, the flat model with a three-dimensional latent
representation and no activation limit attempted to pack all
concepts into single low-dimensional latent space
available to it, resulting in the maximum activation value of 86,
compared to under 20.0 for stacked models (Figure 5).
From generative scans in Fig.4, 5 it can be observed that
the topology of latent regions associated with distinct
concepts is continuous and well-defined, shaped as broad
slanted ”columns” in the flat latent space or slice
subspaces, with characteristic parameters of images such as
color, shape, size and contrast encoded in the latent
position. These observations are consistent with the earlier
results on greyscale data [
The results discussed in the previous sections allow to describe the concept learning ability of sparse models in terms of topological structure of effective latent representations, namely, stacked low-dimensional representations spaces (slices) defined by collections of active neurons participating in producing interpretations of sensory data. In a way, each slice is thus equivalent to a flat lowdimensional representation of constant dimensionality. It was commented earlier based on comparison of interpretation ability of flat models with datasets of different complexity that there appears to be a threshold of maximum conceptual content Tf (M; D) that can be successfully interpreted by a flat model of given architecture M with a given type of observable data D. The results in the earlier sections indicated that for the generative architecture in this work, 3 Tmx < 7.
For a sparse system of dimension d and slice dimension l defined, as commented earlier by the sparsity constraint in the encoding layer, the total conceptual capacity Km, i.e., the maximum number of independent concepts the model can learn successfully can be described by the formula: Km = fe f
N(d; l)
Tf (2) Here, N(d; l) is the number of slices and fe f 1, efficiency factor related to the ability of the model to maximize the use of available slices. In the case of simple single encoding layer models similar to those used in this work, N(d; l) is the binomial coefficient, Bd;l .
From (2) the bound on the conceptual capacity of a sparse model S5 with the latent dimension of 5 and 3-dimensional slices (d = 5; l = 3) can be readily estimated as 10 Tmx, whereas for a general model with an N-dimensional sparse latent layer it would be:
Km(SN )
N (N 1) (N 6 2)
Tmx with sufficiently large dimensionality of the sparse layer, N. Accordingly, sparsity allows to strongly increase conceptual capacity of generative models with only a small addition of neurons in the latent layer and without significant increase in the cost of processing of sensory inputs as the number of active neurons remains constant. 4
The analysis of distributions of image data in the latent representations of generative self-learning models in this work resulted in several essential findings: 1. For a given type, content and characteristic parameters of data and model architecture D; M there exists, under the constraints of realistic learning, a natural threshold Tf (D; M) for the effective number of characteristic patterns or concepts that can be learned successfully. 2. Models with flat architecture of the encoding layer can maintain learning performance if either: conceptual content of the data does not exceed the threshold Tf ; or by expanding the volume of the latent representation space necessary to encode the conceptual content, resulting in higher activations of encoding neurons. 3. Models with sparsity constraint in the encoding layer can bypass the conceptual threshold by creating stacked representation spaces and associating lowdimensional slices with subsets of concepts. 4. An empirical formula for evaluation of maximum conceptual content of sparse models.
These results demonstrated that sparsity allows to achieve two essential objectives in successful realistic learning: increasing the effective conceptual capacity of the latent representations, while limiting the range of latent activations. Thus, sparsity emerges as a simple and effective adaptation that allows to produce functional conceptual representations of complex sensory data in both artificial and biological systems.
Low-dimensional representations, especially those
obtained in unsupervised learning can be of interest due to
growing evidence that similar systems of sensory neurons
can play an important role in processing sensory data by
biological systems. Recent results demonstrated that
effective representations of sensory data such as images,
audio signals and odors can be produced by small sets
of active neurons in biological neural networks [
ICANN (2011). [16] Goodfellow, I., Bengio, Y., Courville, A.: Deep learning.
MIT Press (2016) [17] Keras: Python deep learning library. https://keras.io/ [18] Fukunaga, K., Hostetler, L.D.: The estimation of the gradient of a density function, with applications in pattern recognition. IEEE Trans. Inf. Theory bf 21(1) (1975) 32–40 [19] Dolgikh, S.: Low dimensional representations in generative self-learning models. Proc. 20th Int. Conf. Info. Tech. – App.
Theo. (ITAT-2020) Slovakia 2718 (2020) 239–245 [20] Zhou, X., Belkin M.: Semi-supervised learning. In: Acad.
Press Lib. in Signal Proc. Elsevier (2014) 1239–1269