Integrated gradients is prevalent within machine learning to address the black-box problem of neural networks. The explanations given by integrated gradients depend on a choice of base-point. The choice of base-point is not a priori obvious and can lead to drastically diferent explanations. There is a longstanding hypothesis that data lies on a low dimensional Riemannian manifold. The quality of explanations on a manifold can be measured by the extent to which an explanation for a point lies in its tangent space. In this work, we propose that the base-point should be chosen such that it maximises the tangential alignment of the explanation. We formalise the notion of tangential alignment and provide theoretical conditions under which a base-point choice will provide explanations lying in the tangent space. We demonstrate how to approximate the optimal base-point on several well-known image classification datasets. Furthermore, we compare the optimal base-point choice with common base-points and three gradient explainability models.
Deep learning provides state-of-the-art solutions to a wide array of computer vision tasks [
Several metrics have been proposed to measure the quality of explainability models. In [
The tangent space captures the features of an image that can be changed whilst remaining in the
distribution of images. The intuition is if an explanation lies in the tangent space of the image, the
explanation will contain meaningful components of the image [
Integrated gradients (IG) [
The contributions of this work are twofold: 1. We provide suficient conditions for when integrated gradient explanations are tangentially aligned. We extend these results to any base-point attribution method. 2. We provide a framework to choose a base-point point which provides meaningful explanations to the user. We compare our method with three gradient explainability models and IG with common base-points. We demonstrate that our base-point choice provides better tangential alignment and consequently more meaningful explanations. We validate our approach on four well-known computer vision datasets.
The remainder of this work is structured as follows: Section 2 provides related work and background. Section 3 investigates theoretical conditions for tangential alignment of base-point attribution methods. Section 4 calculates base-points for tangential alignment of IG on four well known datasets. We compare tangential IG with four common base-point choices and three gradient explainability models. We conclude in Section 5 with a discussion for future works.
2.1. Tangentially Aligned Integrated Gradients Explanations
Post hoc explainability models are methods for providing an attribution for the features that influence
the output of a neural network. Post hoc explainability is a step towards addressing the black-box
problem [
Base-point attribution methods (BAM) [20] are a specific class of post hoc explainability models. A BAM is a function
Z b
a Aγv (x, x′, F ) =
⟨∇F (γ (t)), v⟩⟨γ ′(t), v⟩dt.
Aγ (x, x′, F ) =
d X Aγvi (x, x′, F )vi.
i=1 Aγei (x, x′, F ) =
Z b ∂F a ∂xi (γ (t)) ∂γ i (t)dt.
∂t A : M × M × F (M ) → Rd
(x, x′, F ) 7→ A(x, x′, F ) where, M ⊂ Rd is a manifold, F (M ) denotes the set of neural networks on M and x, x′ ∈ M are an input and a base-point, respectively.
We will further restrict the space of BAM functions to path methods, and we will generalise the
definition of path methods to be independent of coordinates. Given a closed interval I := [a, b] ⊂ R, a
path γ : I → M and a unit vector v ∈ Rd, the component of a path method Aγ : M × M ×F (M ) → Rd
in the direction of v is defined as
In this way, for a given orthonormal basis {v1, . . . , vd} of Rd, Aγ is expressed as
Particularly, for the standard orthonormal basis {e1, . . . , ed} of Rd, we obtain the usual definition
(1)
(2)
(3)
(4)
(5)
The prominent path method, integrated gradients [
IGv(x, x′, F ) := ⟨x − x′, v⟩ ⟨∇F (x′ + t(x − x′)), v⟩dt.
Letting I : M ×
M × F (M ) → Rn be the map defined by
Z 1 0
Z 1
0 I(x, x′, F ) :=
(∇F )(x′ + t(x − x′))dt, we can express µ x as v = π xv + π x⊥v, ∥v∥22 = ∥π xv∥22 + ∥π x⊥v∥22, µ x(v) :=
∥π xv∥22 ∥π xv∥22 + ∥π x⊥v∥22
Minimising the norm of the projection into TxM ⊥ provides a framework to ensure tangential alignment. integrated gradients can be expressed succinctly in the standard orthonormal basis of Rd as
IG(x, x′, F ) = (x − x′) ⊙ I(x, x′, F ), where ⊙ denotes the Hadamard product.
Several metrics have been proposed to measure the quality of explainability models. In [
For the reminder of this work we will consider Rd, equipped with its standard inner product ⟨· , ·⟩ ,
and we will let M ⊂ Rd be a manifold of dimension n < d. We will also write ⟨· , ·⟩ for the restriction
of the inner product of Rd to M , such that (M, ⟨· , ·⟩ ) is an embedded Riemannian submanifold of
(Rd, ⟨· , ·⟩ ). We will denote the tangent space of M at a point x by TxM which, in the context of data
manifolds, will consist of all v ∈ Rd such that x + v is “close" to M , with ∥v∥2 small [
TxM ⊥ := {u ∈ TxRd : ⟨u, v⟩ = 0, ∀v ∈ TxM }.
We will let π x : TxRd → TxM denote the natural projection from TxRd to TxM defined by (6) (7) (8) (9) (10) (11) (12) (13) (14) n π x(v) = X⟨v, τ ℓ⟩τ ℓ,
ℓ=1
µ x(v) := ∥π∥vx∥v22∥22 ,
v ∈ TxRd.
where, {τ 1, . . . τ n} is an orthonormal basis for TxM . We define the map µ x : TxRd → [
Usually p = 1 or 2. 3. Uniform. We sample uniformly over a valid range of M 4. Gaussian. A Gaussian filter is applied to the input x.
α zero = 0. α xmax = argmax ∥x − y∥p.
y∈M α iuniform ∼ U (min, max).
i i α Gaussian = σ · v + x, where, vi ∼ N (0, 1) and σ ∈ R. We require the α Gaussian is still within the data distribution so α Gaussian → α Uniform as σ → ∞ [21].
The zero base-point (Equation 15) will not highlight the aspects of the image which may be important if the object of interest contains black pixels [21, 22]. To address the issue of a constant base-point missing important features maximum distance (Equation 16) was proposed in [21]. Maximum distance takes the furthest point (in ℓp distance) from the input image such that the base-point does not contain important information of the input. Another alternative is to sample a base-point from a distribution such as uniform (Equation 17) or Gaussian (Equation 18) [21, 23]. Despite the various choices of base-point we demonstrate none of the aforementioned base-points provide perceptually aligned explanations.
Zaher et al. [24] propose Manifold Integrated Gradients (MIG). MIG replaces the straight line in IG with a geodesic such that the attribution lies in the Riemannian manifold. Whilst MIG addresses the problem of IG not conforming to the geometry of the data. MIG does not resolve the issue of base-point choice nor does MIG ensure that the attribution lies in the tangent space of the manifold. 2.2. Base-point Selection for Integrated Gradients The attribution of IG depends on the base-point chosen. Base-point selection is domain dependent and chosen heuristically. Here we review common base-point choices as provided by [21]. 1. Zero. Here the base-point for all points is a constant zero vector
in general the zero base-point can be any constant vector. 2. Maximum Distance. For a given input x ∈ M , α is defined as the point in distance from x i.e.
3. Optimising the Base-point for Tangentially Aligned Explanations Throughout this section, we will study the map defined in Equation 11, to identify possible choices of base-points for the attribution given by a BAM to be tangent to M at a point. To be precise, for a given BAM, we want to find α ∈ M such that the map
x′ 7→ µ x(A(x, x′, F )) attains its maximum and, particularly, when this maximum value is equal to 1. We note that α = x is always a solution, however, we will always require α ̸= x for non-trivial solutions. Definition 1. Let A : M × M × F (M ) → Rd be a BAM and x, α ∈ M , F ∈ F (M ). A is tangentially aligned at x, with base-point α , if µ x(A(x, α, F )) = 1. (15) (16) (17) (18) (19)
In the remainder of this section x ∈ M and F ∈ F (M ) will be fixed, unless otherwise stated. Letting π x⊥ : TxRd → TxM ⊥ denote the natural projection and defining the maps
Hx : M → TxM ⊥,
Hx(x′) := π x⊥A(x, x′, F ) and 1
Ex : M → R, Ex(x′) := 2 ∥Hx(x′)∥22, we can characterise tangentially aligned BAM explanations with the following theorem. Theorem 1. Let A : M × M × F (M ) → Rd be a BAM and x, α ∈ M , F ∈ F (M ). Then A is tangentially aligned at x, with base-point α , if and only if Hx(α ) = 0 or, equivalently, if Ex(α ) = 0. Proof. It is immediate from the definitions of Hx and Ex, since they are the projection to TxM ⊥ of A and a multiple of its norm, respectively.
Choosing an orthonormal basis
⟨Hx(α ), (∂i′Hx)(α )⟩ = 21 ∂i′⟨Hx, Hx⟩|α = (∂i′Ex)(α ) = 0. for all i = 1, . . . , d, as claimed.
{τ 1, . . . , τ n, ν n+1, . . . , ν d} of TxRd such that {τ i}in=1 and {ν i}i=n+1 are orthonormal basis of TxM and TxM ⊥, respectively, we d observe that and
Hx(x′) =
A(x, x′, F ) − n P ⟨A(x, x′, F ), τ i⟩τ i = i=1 d
P ⟨A(x, x′, F ), ν i⟩ν i i=n+1 Ex(x′) = 2 i=n+1 1 Xd ⟨A(x, x′, F ), ν i⟩2.
Therefore, any choice of a basis for TxRd, adapted to the splitting of TxRd into tangent and normal spaces of M at x, will provide us with with a system of equations to test for tangentially aligned explanations.
Theorem 1 provides us with a necessary condition that a base-point must satisfy to obtain a tangentially aligned explanation. To observe this, suppose that there exists α ∈ M such that A(x, α, F ) is tangentially aligned. Then, by Theorem 1, Ex(α ) = 0 and since Ex(x′) ≥ 0 for all x′ ∈ M , it is in fact a global minimum of Ex and, consequently, (∇Ex)(α ) = 0. Moreover, its Hessian matrix HessEx is positive definite at α .
To simplify notation, in what follows we will denote the partial derivatives with respect to xi and x′i by ∂i and ∂i′, respectively.
Corollary 2. It is a necessary condition for A(x, α, F ) to be tangentially aligned, that ⟨Hx(α ), (∂i′Hx)(α )⟩ = 0, for all i = 1, . . . , d.
Proof. If A(x, α, F ) is tangentially aligned, then (∇Ex)(α ) = 0, which is equivalent to (∂i′Ex)(α ) = 0 for all i = 1, . . . , d. It follows from the definition of Ex that: (20) (21) (22) (23) (24) (25) (26) Lemma 3. Let A be a real symmetric matrix such that Aii > Ri for all i, then A is positive definite.
Lemma 3 follows immediately from [25]. The following theorem is an immediate consequence of Corollary 2 and of Lemma 3 applied to HessEx.
⟨Hx(α ), (∂i′Hx)(α )⟩ = 0 (Hess Ex)(α )ii > Ri(α ), and that
where Ri(α ) denotes the radius of the i-th Geršgorin disk of (HessEx)(α ).
In order to find conditions for the Hessian matrix of Ex to be positive definite, we will make use of Geršgorin circle theorem [25] to find bounds for the eigenvalues of Hess Ex. For a given complex n × n matrix A, its i-th Geršgorin disk is the closed disk Gi(A) := D(Aii, Ri) ⊂ C, where the radius is given by the formula
Ri = X j∈Ji |Aij |,
Ji = {1, . . . , i − 1, i + 1, . . . , n}.
(27)
(28)
(29)
(30)
(31)
In this section we approximate tangential base-point choices on four well-known datasets in computer
vision: MNIST [26], Fashion-MNIST [27], CIFAR10 and FER2013 [28]. We demonstrate that the four
common base-point choices defined in Section 2.2 consistently provide explanations that are not
well aligned with the tangent space. We further demonstrate tangentially aligned IG provides higher
tangentially aligned explanations than three gradient explainability models: Gradient [29], Smooth
Grad (SG) [
Jdec(x) : TxL → Tdec(x)M.
Null (τ 1, . . . , τ n) ,
The Jacobian of the decoder can be computed via back-propagation [31]. The tangent space of M is
spanned by the gradient of dec [
Fashion MNIST 0.0 ity 0.0 s n e 2D.0 4.3. Complexity Analysis The problem of finding a base-point that gives tangentially aligned explanations can be phrased as α x∗ = argmin Ex(α ).
x̸=α (32) If we suppose M ⊆ Rd is compact, then by Weierstrass’s theorem such an α ∗ exists [33]. The continuity of Ex follows from the continuity of IG and norms. The condition x ̸= α is required to ensure non-trivial CIFAR10 0.4 0.6 Fashion MNIST
Tangential Uniform Max dist L2 Guassian
Zero 0.8
1.0 Gradient Smooth Grad Input*Gradient Tangential IG solutions. A solution to the optimisation problem in Equation 32 can be approximated via gradientdescent. We note that zero, Gaussian and Uniform base-points have constant time O(1) complexity. Maximum ℓ2 distance is O(|D|) where |D| is the number of points in the dataset. Calculating a tangential base-point has complexity O(ε), where ε is the number of iterations in gradient descent to solve Equation 32. IG with base-point α x∗ from Equation 32 will be referred to as tangentially aligned IG.
Ground Truth
Tangential
Zero
Max dist
Gaussian
Uniform 4.4. Comparison of Diferent Base-points with Tangential Integrated Gradients For each dataset IG is applied to the CNN with the base-points defined in Section 2.2 and the fraction of each explanation is calculated via Equation 11. To calculate each base-point in Section 2.2 we use the implementation provided by [21].
For each point we approximate the solution to Equation 32 to provide tangential explanations on each dataset. To approximate the solution to Equation 32 over all points we use the same learning rate and number of iterations. Some points may require a diferent learning rate and number of iterations to achieve higher tangential alignment. We leave this to future work. In Figure 1a we have the distributions of the fraction in the tangent space on FER2013, CIFAR10, MNIST32 and Fashion MNIST. We see in Figure 1a that approximating solutions to Equation 32 consistently provides explanations with high tangential alignment.
We see in Figure 1a that the uniform base-point provides explanations consistently close to the
normal space; followed by maximum ℓ2 distance and Gaussian. We note that on FER2013 and CIFAR10
the Gaussian base-point performs better than zero, uniform, and maximum ℓ2 distance. The better
performance of a Gaussian base-point is likely due to the smoothing parameter σ defined in Section 2.2.
It is the goal of future work to determine the impact of σ on tangential alignment. The vertical lines
in Figure 1a indicate the expectation a random vector will lie in the tangent space. The expectation
is approximately pn/d, where n and d are the dimensions of the tangent space approximation and
manifold, respectively [
We provide in Figure 2, example integrated gradient explanations for a point on MNIST32, FER2013, and Fashion-MNIST with difering base-point choice. We see that our method provides tangentially aligned explanations with µ x > 0.91 for all datasets. The tangentially aligned integrated gradient attributions are clear and perceptually aligned with the object to classify in the image. We see in Figure 2 that uniform, maximum ℓ2 distance, and Gaussian are consistently random noise. 4.5. Comparison of Gradient Explainability Models with Tangential Integrated
Gradients In this section we compare tangentially aligned integrated gradients with three common gradient explainability models: Gradient, Smooth Grad and Input *Gradient. The aforementioned gradient explainability models do not require a base-point choice. We demonstrate that tangentially aligned integrated gradients significantly improves upon integrated gradients. The gradient explainability models for a given model are defined as follows: 1. Gradient The gradient of a model f at x ∈ Rd for class i is defined as: 2. Smooth Grad We define Smooth Grad with n samples and standard deviation σ as: grad(x)i := ∂f(x)i ∂x
.
n SmoothGrad(x) = 1 X n i=1 ∇f (x + a), where, a ∼ N
(0, σ 2). Following [
Input ∗ Gradient := x ⊙ ∂f (x)i ∂x .
In Figure 1b we provide density plots of the fraction an attribution is in the tangent space for: Gradient, Smooth Grad, Input*Gradient and tangentially aligned integrated gradients. We see in Figure 1b that tangentially aligned integrated gradients provides attributions consistently in the tangent space, out-performing the aforementioned gradient explainability models. In Figures 1a and 1b Gradient, Smooth Grad and Input*Gradient provide better tangential alignment than the common base-point choices provided in Section 2.2 on MNIST and CIFAR10. On Fashion-MNIST we see that the zero base-point choice provides comparable performance with Gradient, Smooth Grad and Input*Gradient. In Figures 1a and 1b we see that on FER2013, Gradient, Smooth Grad and Input*Gradient perform similarly to Gaussian, maximum ℓ2 distance and zero base-point choices for integrated gradients. All gradient models on FER2013 outperform the uniform base-point choice for integrated gradients. We see in Figures 1a and 1b, Gradient, Smooth Grad, and Input*Gradient tend to out-perform Integrated gradients with standard the standard base-point choices. Tangential integrated gradients out-performs the aforementioned gradient explainability models and standard base-point choices. (33) (34) (35)
In this work we investigated how to choose base-points for IG that provide tangentially aligned explanations. We provided theoretical conditions for a base-point to provide tangentially aligned explanations for any BAM. We demonstrated how to numerically approximate the base-point which provides tangentially aligned explanations and validated this approach on several well-known image classification datasets. In future work we seek to further investigate the theoretical conditions a base-point must have to provide tangential explanations.
The Commonwealth of Australia (represented by the Defence Science and Technology Group) supports this research through a Defence Science Partnerships agreement. Lachlan Simpson is supported by a scholarship from the University of Adelaide. [17] P. Xenopoulos, G. Chan, H. Doraiswamy, L. G. Nonato, B. Barr, C. Silva, GALE: Globally Assessing Local Explanations, in: Proceedings of Topological, Algebraic, and Geometric Learning Workshops 2022, volume 196 of Proceedings of Machine Learning Research (PMLR), 2022, pp. 322–331. [18] B. Sanchez-Lengeling, J. Wei, B. Lee, E. Reif, P. Wang, W. Qian, K. McCloskey, L. Colwell, A. Wiltschko, Evaluating Attribution for Graph Neural Networks, in: Advances in Neural Information Processing Systems, volume 33, 2020, pp. 5898–5910. [19] D. Drakard, R. Liu, J. Yosinski, Exploring unfairness in Integrated Gradients based attribution methods, OpenReview (2022). [20] D. Lundstrom, T. Huang, M. Razaviyayn, A Rigorous Study of Integrated Gradients Method and Extensions to Internal Neuron Attributions, Proceedings of the 39th International Conference on Machine Learning 162 (2022) 14485–14508. [21] P. Sturmfels, S. Lundberg, S.-I. Lee, Visualizing the Impact of Feature Attribution Baselines, Distill (2020). Https://distill.pub/2020/attribution-baselines. [22] M. Sundararajan, A. Taly, A Note About: Local Explanation Methods for Deep Neural Networks Lack Sensitivity to Parameter Values, arXiv preprint arXiv:1806.04205 (2018). arXiv:1806.04205. [23] R. C. Fong, A. Vedaldi, Interpretable Explanations of Black Boxes by Meaningful Perturbation, in: 2017 IEEE International Conference on Computer Vision (ICCV), IEEE, 2017, pp. 3449–3457. [24] E. Zaher, M. Trzaskowski, Q. Nguyen, F. Roosta, Manifold Integrated Gradients: Riemannian
Geometry for Feature Attribution, arXiv preprint arXiv:2405.09800 (2024). arXiv:2405.09800. [25] S. Geršgorin, Über die Abgrenzung der Eigenwerte einer Matrix, Bulletin de l’Académie des
Sciences de l’URSS (1931) 749–754. [26] L. Deng, The MNIST Database of Handwritten Digit Images for Machine Learning Research [Best of the Web], IEEE Signal Processing Magazine 29 (2012) 141–142. [27] H. Xiao, K. Rasul, R. Vollgraf, Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine
Learning Algorithms, arXiv preprint:1708.07747 (2017). [28] I. J. Goodfellow, D. Erhan, P. L. Carrier, A. Courville, M. Mirza, B. Hamner, W. Cukierski, Y. Tang, D. Thaler, D.-H. Lee, Y. Zhou, C. Ramaiah, F. Feng, R. Li, X. Wang, D. Athanasakis, J. Shawe-Taylor, M. Milakov, J. Park, R. Ionescu, M. Popescu, C. Grozea, J. Bergstra, J. Xie, L. Romaszko, B. Xu, Z. Chuang, Y. Bengio, Challenges in Representation Learning: A report on three machine learning contests, arXiv preprint arXiv:1307.0414 (2013). [29] K. Simonyan, A. Vedaldi, A. Zisserman, Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps, arXiv preprint arXiv:1312.6034 (2014). arXiv:1312.6034. [30] A. Shrikumar, P. Greenside, A. Shcherbina, A. Kundaje, Not Just a Black Box: Learning Important Features Through Propagating Activation Diferences, arXiv preprint arXiv:1605.01713 (2017). arXiv:1605.01713. [31] H. Shao, A. Kumar, P. T. Fletcher, The Riemannian Geometry of Deep Generative Models, in: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2018, pp. 428–4288. [32] N. Kokhlikyan, V. Miglani, M. Martin, E. Wang, B. Alsallakh, J. Reynolds, A. Melnikov, N. Kliushkina, C. Araya, S. Yan, O. Reblitz-Richardson, Captum: A Unified and Generic Model Interpretability Library for PyTorch, arXiv preprint arXiv:2009.07896 (2020). arXiv:2009.07896. [33] W. Rudin, Principles of Mathematical Analysis, McGraw Hill, 1976.