Encoder-decoder architectures have widespread use, both in research and the industry. Recently, similarity analysis applied to the representations of neural networks has contributed to a better understanding of the architectures. Previous work has found that for two instances (under diferent initialization) of the same layer of a given model, the learned representation becomes more and more dissimilar the farther away that layer is from the input layer. Since the encoder-decoder is often mirrored (causing a representational bottleneck), we investigate how the representation changes and if the objective of reconstructing the input influences this. Using representation similarity analysis, we find out that corresponding layers from the encoder and decoder are not very similar to each other and are more similar to their neighbor layers. In addition, our experiments show that except for classification tasks, the representations of the same decoder with diferent initialization become more and more similar the closer a layer is to the output layer. Some of our analysis includes comparing the average distances between the layers to the average distance of the current layer clusters, the impact of varying the latent dimension as well as having multiple bottleneck layers on the representational consistency.
One of the first successful cases of training deep architectures involved successively training
(and consequently stacking) denoising autoencoders [
Popal et al. [
© 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
order to characterize a system’s inner stimulus representations in terms of pairwise response
diferences [
In this paper, we apply RSA to the representations of (convolutional) autoencoders and
observe their consistency. We observe that in contrast to the supervised classifiers examined in
Mehrer et al. [
We also investigate the influence of the size of the latent layer. The larger it is, the more consistent it stays. Analyzing a single model, we see that the layers of a network are more similar to the neighbor layers than to their corresponding mirror layers.
In this section, we describe the encoder-decoder architectures and introduce representational similarity techniques.
Encoder-decoder architectures are networks in which an encoder takes a variable-length sequence as the input and transforms it into a state with a fixed shape. The decoder then maps the encoded state of a fixed shape to a variable-length sequence. The final encoder layer from which the decoder maps the encoded state is known as the latent dimension layer, which has the least number of neurons among all the layers of the network.
Encoder-Decoder architectures have widespread use in Natural Language Processing (NLP) and Image Denoising. In NLP, the common approach is to use sequential models such as Recurrent Neural Networks (RNNs) as encoder/decoder.
Baldi [
Representational Similarity Analysis (RSA) was first proposed in the field of neuroscience
[
, = (ℎ , ℎ )
where (ℎ , ℎ ) is a dissimilarity function. Kriegeskorte et al [
Given an RDM, it is possible to analyze how well the distribution of representational distances generalizes across network instances. This can be done using Representational Consistency (RC), which is defined as the shared variance between the upper triangle of a given RDM matrix. More specifically, given a set of instances ∈ { 1, ..., }, and a layer (), the RC for layer ( ) is defined as: = = ∑< ( ‖ ‖(‖ ‖ − 1) 2 ( ), ( )) (1) (2) (3) where ‖ ‖ is the number of instances in , and for a given model is defined as the average is the Pearson correlation coeficient. The of all the layers in that model.
The intuition behind RC is that the closer the value is to 1, the higher the consistency for that particular layer. For example, the input layer is always 1 because the inputs are the same for all the diferent instances of the network. As we progress deeper into the network layers, we see a continuous drop in consistency until we reach the bottleneck layer.
Multidimensional scaling (MDS) [
The goal of our experiments is to get a better understanding of the representational behavior in autoencoder architectures. To that end we analyze representation similarity and consistency under slight variations of the experimental setup described below:
Index
Except for the experiment where we were attempting to analyse the change in consistency with the epochs increasing logarithmically till 256, all the other models were trained using early stopping, monitoring the validation loss with a patience of 10.
Additionally, for the experiments conducted on representational consistency - for a Kerasbased architecture for an autoencoder, the default kernel initialization is generally a GlorotUniform for a dense/convolutional layer. So to change this default initialization, the experiments in the thesis use a Glorot-Normal instead to see the diference obtained.
In our experiments, we use two diferent architectures, one with only fully connected layers and one with also convolutional ones. In Table 1a, we see the base architecture of the fully connected networks, whereas, in Table 1b, the convolutional architecture is depicted. In several experiments, we extend the networks as indicated.
Below is a short description of the datasets used in the experiments conducted in the paper.
The MNIST database [
(a) Sample image from the MNIST dataset [
Similar to the MNIST dataset, CIFAR10 [
In this section, we depict our results.
In our first experiment (Figure 2a), we show all layers of ten diferently initialized instances embedded into two dimensions according to their respective similarities. The input and the output layers are comparatively closer to each other, and all the other layers are quite far apart. We can conclude that the points of the same layer get further apart from each other as we move deeper into the layers of the network, and they start getting closer to each other again as we reach the end. The input representations are obviously all the same. The next layer (Encoder1 E1 in the plot in Figure 2a) has all the points very close to each other, except maybe one point, (a) Sample MNIST dataset MDS plots for 10 models (b) Sample comparison between cluster average versus next layer distance. which is comparatively far away from all the other points with the same color. This becomes more frequent as we keep going deeper into these layers.
In Figure 2b, we observe that except for the first and last layers, the distance to the successive (next) layer is smaller the the overall cluster average.
(a) Representational consistency: logarithmically increasing epochs comparison for a Fully-connected (b) Comparison between multiple latent dimenautoencoder with 8 neurons in the bottleneck lay- sions for an autoencoder architecture on an ers. MNIST dataset
Similar to Mehrer et al. [
In this experiment, we observe the representation consistency when changing the dimension of the latent vector and letting everything be the same.
Based on the results of the experiments shown in Figure 3b, we can see a clear correlation between the representational consistency and the number of neurons we have in the latent dimension layers. We notice that as the number of neurons increases, the representational consistency of the bottleneck layer also increases.
We can interpret that with the increase in the number of neurons of the bottleneck layer from 10 to 40 though these values are comparatively less, the information held in the previous layers has to be less compressed in the case of the 40 neurons than in the case of the ten neurons. We would always attribute some amount of loss in the case of information transfer from the previous encoder layer to the bottleneck layer, but it would be comparatively less in the case of the 40 neurons than the ten neurons, which automatically correlates to a more consistent reproducibility of the input image in the output and hence a better consistency comparison in the final output layer as well.
When training architectures with multiple bottleneck layers (3, 5 and 7), the below experiments were conducted with ten diferently initialized instances for all the experiments: (i) Fully-connected autoencoder: MNIST dataset with 8 and 16 neurons in the bottleneck layers respectively, (ii) Convolutional autoencoder: MNIST dataset with 8 and 16 neurons in the bottleneck layers respectively, (iii) Convolutional autoencoder: CIFAR10 dataset with 12 and 25 neurons in the bottleneck layers respectively. The results of these experiments can be seen in the plots in Figure 4:
The above plots show the comparisons while working with a varying number of bottleneck layers for an MNIST dataset for a Fully-connected autoencoder as well as a convolutional autoencoder, with 16 neurons in the bottleneck layer. A common latent dimension and dataset is chosen so that the networks are more comparable. We see, in both cases, that there is not much separating the consistencies of the networks with five and seven bottleneck layers. However, the network with three bottleneck layers has a much higher consistency at the bottleneck level (a) Fully-connected autoencoder with 16 (b) Convolutional autoencoder with 16 neurons in the neurons in the bottleneck layers. bottleneck layers. as well as overall in both cases. In the case of the networks with five and seven bottleneck layers, we see the consistencies flattening out in the region shaded blue in the above plots. And though this cannot be confirmed with a guarantee as yet, this trend has more or less been consistent regardless of the dataset and the network. A possible, interesting future work for this thesis would be to replicate these results on much larger and more complicated datasets.
To check the dependence of the consistency on the number of trained model instances with diferent random initial weights, two experiments were performed. For the fully-connected autoencoder trained on MNIST as well as the convolutional autoencoder trained on CIFAR-10 we compared the consistency values in the diferent layers for diferent number of model instances. The results are shown in Figure 5a for the fully connected autoencoder and Figure 5b for the convolutional autoencoder. The fully connected autoencoder, tested for 10, 20, 30 and 40 model instances, shows convergence at 20 instances. Due to technical limitations the convolutional autoencoder was only tested for 5, 10, 15, and 20 model instances. While the consistency is not convergent at these instance numbers, we can see the same pattern as for the fully connected autoencoder and assume that using more than 20 model instances would not result in relevant changes.
Figure 6 shows all the representational consistencies in a single plot for the MNIST dataset for a convolutional autoencoder. We can see here that, regardless of which layer has how many multiple layers in other networks, the overall highest consistency was shown by the network, which has no multiple numbers for any of the layers.
The basic motivation behind this above experiment was to see how multiple adding layers of a certain type of layer, before and after the bottleneck layer, impacted the representational consistencies across multiple instances. A small note here is that when working with a convo(a) Fully connected Autoencoder on MNIST (b) Convolutional Autoencoder on CIFAR-10 lutional autoencoder, the bottleneck layer was a dense layer, but the other multiple layers (C2 and C5) were all convolutional layers.
So far, we have only looked at the representation consistency regarding ten diferent models with diferent random initialization. In this section, we want to look at individual models to understand single models better and observe if mirrored layers share a structure. In Figure 7 are the MDS plots for two samples that were obtained using a CIFAR10 dataset on a convolutional autoencoder with 12 neurons in its bottleneck layer.
A slight distance is seen between the input and output. However, it is important to understand here that the motive behind performing these experiments was not to find the best possible solution as to which parameters or architectures give the most accurate results. Hence, it was chosen to work with neurons (8 and 16 neurons in the case of an MNIST dataset and 12 and 25 neurons in the case of a CIFAR10 dataset) that were not giving us pixelated outputs while at the same time, they were not overcompensating what is being attempted to study when conducting these experiments. Although the individual instance plots look similar in nature, we see some diferences in the directions of the layers, which would be attributed to the diferent random initializations that take place inside these networks.
It is important to note that when the input and output layers are included in an instance plot, these two layers would always be close or far away from each other depending on the number of neurons chosen to work with for the latent dimension. This, however, can be misleading at times because of what the autoencoder attempts to do. For this very reason, the above plots of Figure 8 were included where the input and output layers were left out, and a simpler comparison was made between all the other layers of the network. As shown in the above Figure 7, we do not see much diference between diferent instances of the same latent dimension. For the current analysis in Figure 8, we see four diferent instance plots (all four of them are taken from the 3 instance of each experiment for a consistent comparison), where the instances have been plotted without the input and output layers with linearly increasing latent dimensions (i.e., 10, 20, 30 and 40 neurons). Even in such a scenario, we see consistency among the models. As previously discussed, we would normally expect a U-shape between the mirror layers of the model, but we see a sort of a -shape forming between the mirror layers with the shortest distance between the final encoder layer and the first decoder layer as the distance keeps increasing all the way till the first encoder layer and the final decoder layer. This distance appears to be consistent across all the instances, even with an increasing latent dimension.
(a) Latent dimension with 10 neurons (b) Latent dimension with 20 neurons (c) Latent dimension with 30 neurons (d) Latent dimension with 40 neurons
In this paper, we demonstrated that for an encoder-decoder neural network, as information is propagated through encoder layers, there is a decrease in the representational consistency of the layer. And the opposite happens for the decoder layers. Such type of representational bottleneck forces the encoder to “prioritize” features, discarding features not relevant for reconstruction. We also demonstrated that for any given instance, the representations tend to diverge (from the input) as information is propagated through the encoder layers and then converge (towards the input) as information is propagated through the decoder.
It is also observed that even when we have multiple bottlenecks, a consistent line is not seen for that particular period. There was a dip even among the bottleneck layers. Furthermore, the dip was observed in the middle bottleneck layer in these instances with multiple bottleneck layers.
In practical terms, these findings validate the common intuition behind encoder-decoder neural networks (that such bottleneck provides some sort of “lossy compression”). For practitioners, such type of analysis can be used a helpful tool when debugging novel architectures.