We investigate the ability of Difusion Variational Autoencoder ( ΔVAE) with unit sphere 2 as latent space to capture topological and geometrical structure and disentangle latent factors in datasets. For this, we introduce a new diagnostic of disentanglement: namely the topological degree of the encoder, which is a map from the data manifold to the latent space. We derive and implement an algorithm that computes this degree, and we use it to compute the degree of the encoder of models that result from the training procedure. Our experimental results show that the ΔVAE achieves relatively small LSBD scores, and that regardless of the degree after initialization, the degree of the encoder after training becomes −1 or +1, which implies that the resulting encoder is at least homotopic to a homeomorphism.
CEUR
ceur-ws.org
1. Introduction
homeomorphism.
mismatch problem [
Besides formal definitions of disentanglement, there are desired characteristics for disentangled latent
factors, for instance that nearby points in the dataspace should correspond to nearby points in the latent
space representation. This could lead to a requirement that the encoder should be a homeomorphism
[
Given that an agreed definition of disentanglement does not yet exist, we consider it desirable to develop a wide range of diagnostics that are somehow related to the intuitive concept of disentanglement. Moreover, in practice it can be dificult to test whether a given encoder is a homeomorphism. Therefore, we introduce the topological degree as a discrete diagnostic for disentanglement. A homeomorphic encoder always has degree +1 or −1, whereas an encoder with degree ±1 is at least homotopic to a
To achieve a homeomorphic encoder, or to get an encoder with degree ±1, one needs to choose a latent space that matches the topology of the dataset, otherwise one will encounter the manifold
In order to have a wider range of latent spaces and solve the manifold mismatch problem [
We immediately apply the degree as a diagnostic for disentanglement in an evaluation of the ΔVAE, in which we test the ΔVAE with a spherical latent space on data which naturally has a spherical latent ⋆This work was supported by NWO GROOT project UNRAVEL, OCENW.GROOT.2019.044.
It can be challenging to train a ΔVAE, and we wondered whether this was due to the initialization and
topological obstructions, see also [
Our experiments show that regardless of the initial model weight, the topological degree of the
encoder can change to become eventually constant equal to +1 or −1, after some epochs of the training
process. We perform the same experiments for the -VAE [
The VAE [
Intuitions and some aspects of disentangled representation are presented in [
The encoder of a VAE can be considered as a continuous map from a dataspace ⊆ ℝ to the latent space . Intuitively speaking, its topological degree is the number of times that the encoder wraps the data manifold around the latent space, counted in such a way that positive cancels negative orientation (cf. [28, Page 134] and [29, Page 27]). Encoder degree unequal to 1 or −1 indicates that the encoder cannot be a homeomorphism [29, Page 51].
Computing the topological degree of the encoder Although general methods exist [30], we developed and implemented a basic algorithm targeted to the case at hand of computing the degree of a map between spheres. We triangulate the two spheres, and create a “rounding” of the original map that maps vertices to vertices, edges to collections of edges and faces to collections of faces. We finally count how many times the faces in the target sphere are covered, taking into account orientation. Using tools from homology theory we can prove that the algorithm gives the correct result cf. [31].
We train the ΔVAE with 2 as latent space, using a second-order expansion of the heat kernel [32]. We
use a dataset of spherical harmonics as a proxy for a more natural dataset of axisymmetric pictures on the
unit sphere, which naturally has the topology of 2 [33, Page 88] [34, 35]. We include a semisupervised
LSBD-loss as in [
Evolution of the degree during the training We conducted more experiments for spherical
harmonics of degree = 7, 5, 3 with ΔVAE in order to get insight into the evolution of the degree during
the training. We performed 5 experiments where the absolute value of the degree before training was
not 1, whereas the absolute value of the degree after all training was 1. In particular, even though
we share the opinion that topological obstructions might hamper training [
We derive a second order expansion of the heat kernel on the unit sphere 2 by using the theoretical result of [32], and use it as approximation in the ΔVAE loss function. Though the efect of such higher order approximation in the performance of ΔVAE is not studied yet.
Our algorithm for degree computation could be generalized to higher dimensional sphere with > 2 , but due to the curse of dimensionality, practical computation is most likely only feasible in very low dimensions: for a -dimensional manifold and a discretization length , the number of faces needed in the triangulation scales as − .
The amount of semisupervision is relatively high in our experiments. For lower degree spherical harmonics ( = 1, 3, 5 ), the amount of semisupervision can be reduced drastically, although we have not yet performed a systematic study.
We evaluate to what extent the ΔVAE can capture topological properties or disentangle generating factors, as measured by the LSBD score, and as expressed by a new discrete diagnostic for disentanglement: the degree of the encoder. We use the encoder degree as a means to gain more insight in the training behavior.
First, we obtain relatively small LSBD scores, which expresses that the ΔVAE indeed can capture or disentangle the latent rotational factor relatively well. In comparison with the -VAE, we find that the -VAE typically obtains better log-likelihood scores, while the reconstruction error and LSBD score are a bit better for the ΔVAE.
Secondly, we implemented an algorithm for computing the topological degree of the encoder and ifnd that even though the encoder is typically initialized with degree 0, this degree can change and after training the encoder indeed has degree of ±1, which means that the encoder is at least homotopic to a homeomorphism and that the learned spherical representation preserves the topological structure of the dataset at least up to a homotopy. In particular, we find that the sphere in latent space is completely covered by the image of the data manifold. disentanglement learning, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 7920–7929. [20] I. Huh, J. M. Choe, Y. KIM, D. Kim, et al., Isometric quotient variational auto-encoders for structurepreserving representation learning, Advances in Neural Information Processing Systems 36 (2024). [21] K. Do, T. Tran, Theory and evaluation metrics for learning disentangled representations, International Conference on Learning Representations, ICLR 2020 (2020). [22] S. Van Steenkiste, F. Locatello, J. Schmidhuber, O. Bachem, Are disentangled representations helpful for abstract visual reasoning?, Advances in neural information processing systems 32 (2019). [23] M.-A. Carbonneau, J. Zaidi, J. Boilard, G. Gagnon, Measuring disentanglement: A review of metrics, IEEE transactions on neural networks and learning systems (2022). [24] A. Sepliarskaia, J. Kiseleva, M. de Rijke, How not to measure disentanglement, in: ICML Workshop on Theoretic Foundation, Criticism, and Application Trend of Explainable AI, 2021. [25] I. Higgins, L. Matthey, A. Pal, C. P. Burgess, X. Glorot, M. M. Botvinick, S. Mohamed, A. Lerchner, beta-VAE: Learning basic visual concepts with a constrained variational framework., ICLR (Poster) 3 (2017). [26] F. Locatello, S. Bauer, M. Lucic, G. Raetsch, S. Gelly, B. Schölkopf, O. Bachem, Challenging common assumptions in the unsupervised learning of disentangled representations, in: international conference on machine learning, PMLR, 2019, pp. 4114–4124. [27] R. Suter, D. Miladinovic, B. Schölkopf, S. Bauer, Robustly disentangled causal mechanisms: Validating deep representations for interventional robustness, in: International Conference on Machine Learning, PMLR, 2019, pp. 6056–6065. [28] A. Hatcher, Algebraic topology, Cambridge University Press, 2005. [29] J. Milnor, Topology from the diferentiable viewpoint, univ, Press of Virginia, Charlottesville 1990 (1965). [30] T. Kaczynski, K. M. Mischaikow, M. Mrozek, Computational homology, volume 157, Springer, 2004. [31] M. R. Ravelonanosy, V. Menkovski, J. W. Portegies, Topological degree as a discrete diagnostic for disentanglement, with applications to the ΔVAE, https://arxiv.org/abs/2409.01303 (2024). [32] V. Menkovski, J. W. Portegies, M. R. Ravelonanosy, Small time asymptotics of the entropy of the heat kernel on a riemannian manifold, Applied and Computational Harmonic Analysis 71 (2024).
URL: https://www.sciencedirect.com/science/article/pii/S1063520324000198. [33] T. Bröcker, T. tom Dieck, Representations of compact lie groups, Graduate Texts in Mathematics (1985). [34] M. A. Blanco, M. Flórez, M. Bermejo, Evaluation of the rotation matrices in the basis of real spherical harmonics, Journal of Molecular structure: THEOCHEM 419 (1997) 19–27. [35] S. Harmonics, Claus mulier, Lecture Notes in Mathematics (LNM)) 17 (1966). [36] Y. Burda, R. B. Grosse, R. Salakhutdinov, Importance weighted autoencoders, in: Y. Bengio, Y. LeCun (Eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL: http://arxiv.org/abs/1509.00519.