Persistent homology is a tool from topological data analysis that can be used to de ne functions that extract topological features from images. These derivatives of persistent homology are invariant under continuous symmetries in the image space (for example translation or rotation). We provide theoretical argument for using such functions in conjunction with convolutional lters to improve perception algorithms. We discuss using these functions as a method for adding topological inductive priors on the convolutional network. We propose a coupled UNet architecture for learning sematic segmentation in the presence of topological priors involving restrictions on the persistent betti values of the segments. We run experiments on perception data for autonomous driving and are able to report qualitative improvements.
Autonomous driving presents a variety of perception problems. Traditionally deep learning methods involving convolutional neural networks have been employed with great success to address these problems. In spite of its tremendous success deep learning methods are known to be extremely brittle. The source of the brittleness is the dependence of the resulting algorithm on the training data. As a result the trained models are sensitive to distributional shifts in the test data. We apply methods from topology to address this problem in the case of semantic segmentation. The key insight is to use topological loss functions during training. Loss functions derived from persistent homology can be used to put topological constraints on the output of the network. Classical machine learning leverages local geometry, as can be seen from the Euclidean loss functions. Topological Data Analysis (TDA) produces invariants of the global geometry of the input. The constraints introduced through topological loss can be utilized as topological prior assumptions on the model or to reduce distance in topological feature space to the training images. Persistent homology has two important properties of invariance under continuous symmetries and Lipschitz stability. The topological features we consider are therefore very well suited to the robust vision problem. 1.1
In this paper we address the lack of robustness of current
convolutional neural networks when applied to test data
obtained from a distribution di erent from the training
data distribution. As a concrete example we consider
the task of semantic segmentation of street scenes in
adverse weather conditions with a neural network trained
on images in good weather. We start with a baseline
convolutional network, the UNet
Topological signals have been previously used in deep
learning, in particular applied to auto-encoders
The novelty of our method is twofold. Firstly, we introduce the coupled architecture which is o ers exibility in terms of how the new topological features can a ect the learning process of the convolutional network. Secondly, our method can make use of the topological losses to introduce both topological priors and posteriors during learning. To our knowledge this is the rst time topological features have been applied to perception related learning tasks in driving scenarios.
0Copyright '2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0)
Brief Overview of Topological Data
Analysis
In this section we give a brief review of the fundamental
ideas from Topological data analysis, and in particular
the persistent homology algorithm. For references see
Persistent homology (PH) is an algorithm to encode the evolution of homology groups of nested topological space across the scales. The nested topological space is modelled as a ltered simplicial complex. The PH algorithm extends the algorithm for computing ordinary simplicial homology of a simplicial complex to that of a ltered simplicial complex. The homology in dimension k of a simplicial complex X is an abelian group Hk(X). The rank of Hk(X) is also known as the k-th Betti number k and represents the number of k-dimensional holes in X. The PH in dimension k of a ltered simplicial complex X = f = X0 X1 ::: Xi g is a sequential diagram of abelian groups P Hk(X) = Hk(X0) ! Hk(X1) ! : : : Hk(Xn) ! : : :. The persistence diagram P Dk is a way to encode the birth and death of every homology cycle that appears in the diagram. The longer a homology cycle survives the more persistent it is. The full information can be represented as a multi-set of pairs (b; d) where b and b are the birth and death ltration indices of a homology class. We denote P Dk(X) = f(bi; di)gi2Ik . We order the indexing of points by decreasing lifetimes i.e. i < j if di bi dj bj . Let's denote the space of persistence diagrams by PD. Let PDk be the subspace of k-dimensional persistence diagrams. The space PDk has a a metric de ned by the bottleneck distance. For D1; D2 2 PDk,
B(D1; D2) = inf max kp m (p;q)2m qk1 (1) where m is the set of all possible pairwise matchings of the points in D1 and D2
The space of persistence diagrams is however not
amenable to statistical analysis or even calculus, as they
don't even form a vector space. There has been some
alternative formulations of persistence diagrams, like
persistence landscapes
E (p; q; i0; dim)(D) =
X (bi;di)2D jdi bijp di + bi 2 q (2) The parameter p and q de ne a polynomial function on the points (bi; di) of a diagram D in PDdim. The output is the sum of lifetimes of all points in the diagram D after skipping the rst i0 points. The map E (p; q; i0; dim) is continuous given the topology induced by the bottleneck distance on PDdim.
In this paper we also consider the following special cases of the functional E ,
S(dim) := E (1; 0; 0; dim) PS(dim; skip) := E (1; 0; skip; dim)
T (dim; k) = [jdi
bij : 0
i
k]
(3)
(4)
(5)
and,
They stand for the functions SumBarcodeLengths,
PartialSumBarcodeLengths and TopKBarcodeLengths
de ned in
The function S(dim) sums up the lifetimes of all the homology cycles in dimension dim. The function PS(dim; skip) sums up the lifetimes of all homology cycles in dimension dim skipping the rst skip ones. The function T (dim; k) returns a vector of length k whose components are the lifetimes of the rst k homology cycles in dimension dim.
Given a ltered simplicial complex, the authors of
Topological features from images
Given a real valued continuous function on a topological space f : M ! R, we de ne the sublevel set ltration on M by increasing the parameter , with M = f 1( 1; ]. Let us denote the k-th persistent homology of the level-set ltration associated where (M; f ) by P Dk(f ).
Consider a simplicial complex K with vertex set V and a function f : V ! R. The function f can be extended to all simplices of K by f ([v0; : : : ; vk]) = maxff (vi) : i = 1; : : : ; kg for any simplex = [v1; : : : ; vk] 2 K and the family of sub-complexes Kr = f 2 K : f ( ) de nes a ltration called the sublevel ltration of f . rg 3.2
An image can be seen as a real valued function f : I ! [ 1; 1] on the space I ' [0; 1]2. The graph of this function is a surface embedded in I [ 1; 1]. The level sets of this graph at various heights Fr = f 1( 1; r] produces a ltration Fr on I. The persistent homology of this ltration is a way of summarizing of the evolving topologies of the level sets. We need to formulate this in the discrete language of tensors and simplicial complexes.
A gray-scale image can be represented as 2-d tensor. For an image X of dimension n m the tensor is a function X : [1::n] [1::m] ! R Let V = [1::n] [1::m] be the vertex set of a 2-dimensional simplicial complex homeomorphic to I. Call this complex V and the lifted function X~ : V ! R. The sublevel ltration of X is the ltered simplicial complex Xr = f 2 V : X( ) rg. Let I denote the space image tensors. In our case I is a subspace of Rn m.
I ! ltered simplicial complexes
(6)
The dim-dimensional persistent homology applied to the ltered simplicial complex fXrgr2R produces a persistent diagram P Ddim(X) 2 PDdim.
I ! PDdim
X 7! P Ddim(fXig)
Given any functional tensor valued functional F on PD, we can compose to form topological feature maps as follows.
Fe : I ! tensors
The result is a topological feature map called Fe. In this paper our working examples of topological feature maps are Se(dim), PgS(dim; skip) and Te (dim; k). For example the map Ee(1; 0; 1; 0) is the sum of lifetimes of all homology 0-cycles skipping the rst one.
Simplicial homology is a homotopy invariant. Given a homotopy equivalence f : X ! Y of simplicial complexes, the induces map of homology f : Hn(X) ! Hn(Y ) is an isomorphism. The notion of homotopy equivalence of ltered simplicial complexes is given by a commutative diagram of simplicial homotopy equivalences.
X0 ' f0 Y0 / X1 ' f1 / Y1 / X2 ' f2 / Y2 / /
Let us de ne two images to be topologically equivalent if there is a homotopy equivalence between the corresponding sublevel set ltrations. This is the case for example if X(a; b) = Y ( (a; b)) for some Euclidean transformation on I2. This includes translational and rotational symmetries of the image. In general a topological symmetry between images is not necessarily of this form however.
Persistent homology satis es a Lipschitz stability property in the following sense. Let f; g : X ! R be two real-valued functions de ned on a topological space M that are q-tame. Then for any dimension k,
B(P Dk(f ); P Dk(g)) kf gk1 = sup jf (x) g(x)j x2M (10) Filtration of nite simplicial complexes are always tame. As a consequence, given any two images tensors X; Y : I ! [ 1; 1], the associated persistence diagrams in any dimension k satis es the Lipschitz stability property (11)
B(P Dk(X); P Dk(Y )) kX Y k1 where jjX Y jj1 can be interpreted as the L1 distance between the unrolled tensors obtained from the tensors X and Y .
This Lipschitz stability property means persistent homology is robust to perturbations in the image space. (7) (8) (9)
The space of persistence diagrams forms a continuous space, unlike the classical betti numbers. The map P Dk : I ! PD is a continuous map. Therefore continuous morphisms in the image space, i.e. continuously transforming one image into another image will result in a continuous morphism of one persistence diagram into another. Composition with an continuous function into some additive and continuous space F : PD ! A could result in a di erentiable topology feature map. The gradient of this map could used to continuously transform an image to optimise the topological feature map.
The maps Fe are di erentiable and the derivative has
be computed by
With this di erentiable mechanism in place we can optimize in the the space of images I to maximize of minimize any desired topological feature map. In the examples in Figure 3 the input image X has been continuously morphed to nd the closest image X which minimizes Ee(1; 0; 10; 0) and Ee(1; 0; 10; 1) (in top gure) and maximizes P(Te (dim = (0; 1); k = 20)) and P(Te (dim = (0; 1); k = 20)) (in bottom gure).
In notation,
X = argminX2I Ee(1; 0; skip; dim)(X)
X = argmaxX2I Te (dim; k)(X) 1 where dim = f0; 1g, skip = 10 and k = 20.
The e ect of the rst operation is a topological smoothing by moving the points in persistence diagram P D0, except for the rst 10, closer to the diagonal. This results in killing the homology cycles that are not too signi cant in terms of lifetimes. The e ect of the second operation is a topological sharpening by pushing the top k points in the persistence diagrams further away from the diagonal. This results in increasing the lifetimes of the the top k homology cycles.
In the following section we apply this to a neural network optimizer. Given an neural network N applied to an image tensor, the aim is to optimize the pair N to maximize or minimize the topological feature maps. For example, = argmin EX Ee(p; q; i0; dim)(N (X)) (14) 4
Combining topology feature maps with Convolutional networks The optimization task in Figure 3 can be alternative expressed as a optimizing a neural network. Consider (12) (13) The topological feature map can be augmented serially at the end of the decoder network. The training process aims to solve the following optimization problem. ; = argmin ; EX Ee(1; 0; skip; dim)(D E (X)) (15) ; = argmax ; EX Te (dim; k)(D E (X)) (16) 4.1
In this section we de ne the coupled UNet architecture. We apply this to semantic segmentation. Semantic segments have very distinct topological properties. We wish to combine the UNet in the previous section trained to optimize certain topological features characteristic of semantic segments with a standard UNet used for semantic segments. The two UNets are combined in parallel with shared weights in the deeper dimensions.
Let us continue with the same notation for a UNet represented by fully-convolutional ReLU encoder-decoder pair (E ; D ). Let U1 = (E 1 ; D 1 ) and U2 = (E 2 ; D 2 ) be two structurally identical UNets. The weights are shared in the middle. See Figure 4. k1 = k1 = k2 (1 k2 (1 k k
K); K < n K); K < n
1; 1; 2; 2 = argmin 1; 1; 2; 2 (L1 + L2 + L3 + L4) (19) where,
L1 = EX CrossEntropy(D 1 E 1 (X); X) (20) L3 = EX Ee(1; 0; skip; dim)(D 2 E 2 (X)) L4 =
EX Te (dim; k)(D 2
E 2 (X)) 2
The variables K (number of layers of the coupled
UNet with shared weights), skip, t and dim are
hyperparameters.
an UNet architecture
Te (dim; k)(X)) (17) (18)
2 (21) (22) (23)
Our baseline is the UNet model de ned in [Ronneberger et al., 2015]. Our model has the same architectural pattern used in this paper, with changes made to the input/output image dimensions and number of output channels (depending of number of segmentation classes). We shall denote this model as UNET
The coupled UNet consists of two copies of the UNET with shared weights in the middle layers. The second copy of the UNET has a single output channel. The number of layers in the idle with shared weights is a hyperparameter. De ne CUNET to the coupled UNet where all but the nal layers have shared weights. De ne CUNET-wc (weakly coupled) to be the coupled UNet in which 7 layers (about 70%) in the middle of the UNETs have shared weights.
The network architectures are implemented in
PyTorch. The persistent homology functional di erentiable
computational block is implemented using the Topology
Layer PyTorch library
The BDD100k (Berkeley Deep Drive) dataset
The Cityscapes dataset
The ACDC (Adverse Conditions Dataset with
Correspondences) dataset
We use the metrics Pixel-level Accuracy (Accuracy) and Intersection-over-Union (IOU) to get quantitative comparisons of our models. The IOUs are computed by taking the mean of the IOUs over all classes. 5.3
The gure 5 shows output semantic segmentation of UNET and CUNET trained on BDD100k training dataset and evaluated on BDD100k validation dataset. The gure 6 shows output semantic segmentation of UNET, CUNET and CUNET-wc trained on Cityscapes training data and evaluated on Cityscapes validation data.
Both examples show marked improvement in segmentation quality in the CUNets compared to the baseline UNETs. The precise nature of this improvement is the preservation of the topological properties of the class segments. Examples of topological properties of segments are connected components and number of 1-dimensional loops. The topological loss during the training process of the CUNET helps to learn these topological features on top of the usual convolutional features. It is also worth noting that segmentation quality in the weakly coupled CUNET is marginally better than the strongly coupled CUNET 5.4
In this section we present the quantitative results of our experiments. The scores are based on pixel-wise accuracy and Intersection-over-Union. Table 1 show comparison of scores between UNET and CUNET on the BDD100K and Cityscapes datasets. Our model outperforms the baseline in both cases.
In table 2 we produce results comparing performance of the CUNET with the UNET on the ACDC dataset. Here both models were trained on the Cityscapes (clean weather) training data and evaluated on the ACDC (adverse weather) validation data. Our model CUNET is consistently better than the baseline UNET except for the nighttime scenes. These results demonstrate the capacity of the topological coupled network to perform better than the baseline when distribution shift via adverse weather is introduced during inference. This is due to the fact the global topological features learned during training are more robust to distribution shifts than convolutional features alone.
In table 3 we produce the results comparing the performance of the CUNET with a weakly-coupled
UNET CUNET UNET CUNET UNET CUNET UNET CUNET CUNET-wc. Both models are trained on the Cityscapes. The results show improvement in the weakly-coupled architecture when evaluated on the Cityscapes validation dataset. The results are however mixed and inconclusive when adverse weather distribution shift is introduced. 6
Summary and Conclusions Persistent homology captures global topological information. Their de nitions are based on algebraic-topological invariants called homology adapted to point-cloud data. RGB image tensors can be represented as scalar functions on the domain and the natural ltration can be used to de ne persistent homology. This is captured by persistent homology functionals, and not detected by standard convolutional lters. These topological features are furthermore invariant under continuous transformations. Lipschitz stability property satis ed by persistent homology also means it is robust under perturbations in the image space.
Semantic segments of images are have very well dened topological features and serve as a good test case for experimenting with these new features. Our experiments show that learning topological features on top of a convolutional features improves performance of current semantic segmentation models on urban road scenes. We also empirically show that the robustness of the topological features help the new model perform better when distribution shift is introduced during inference via adverse weather conditions.