Estimating the strength of causal e ects from observational data is a common problem in scienti c research. A popular approach is based on exploiting observed conditional independences between variables. It is wellknown that this approach relies on the assumption of faithfulness. In our opinion, a more important practical limitation of this approach is that it relies on the ability to distinguish independences from (arbitrarily weak) dependences. We present a simple analysis, based on purely algebraic and geometrical arguments, of how the estimation of the causal e ect strength, based on conditional independence tests and background knowledge, can have an arbitrarily large error due to the uncontrollable type II error of a single conditional independence test. The scenario we are studying here is related to the LCD algorithm by Cooper [1] and to the instrumental variable setting that is popular in epidemiology and econometry. It is one of the simplest settings in which causal discovery and prediction methods based on conditional independences arrive at non-trivial conclusions, yet for which the lack of uniform consistency can result in arbitrarily large prediction errors.
Inferring causation from observational data is a common problem in several elds, such as biology and economics. To deal with the presence of unmeasured confounders of observed random variables the so-called instrumental variable technique [2] has found applications in genetics [3], epidemiology [4, 5] and economics [6]. Given two observable random variables possibly in uenced by a hidden confounder, an instrumental variable is a third observed variable which is assumed to be independent of the confounder. In practice it is di cult to decide whether the instrumental variable de nition is satis ed, and the method has aroused some skepticism [7]. In this paper, we study a setting that is similar in spirit to the instrumental variable model, but where all conditional independence assumptions are directly testable on the observed data.
ods based on conditional independences is faithfulness, which means that the observed joint distribution does not contain any additional (conditional) independences beyond those induced by the causal structure. Usually, faithfulness is justi ed by the assumption that unfaithful distributions are a set of Lebesgue measure zero in the set of the model parameters. By showing that one can create a sequence of faithful distributions which converges to an unfaithful one, Robins et al. proved the lack of uniform consistency of causal discovery algorithms [9]. Zhang and Spirtes [10] then introduced the \Strong Faithfulness" assumption to recover the uniform consistency of causal discovery. Using geometric and combinatorial arguments, Uhler et al. [11] addressed the question of how restrictive the Strong Faithfulness assumption is in terms of the volume of distributions that do not satisfy this assumption. Even for a modest number of nodes and for sparse graphs, the \not strongly faithful" regions can be surprisingly large, and Uhler et al. argue that this result should discourage the use of large scale causal algorithms based on conditional independence tests, such as the PC and
In this work, we analyse in the context of the LCD setting how an error in a single conditional independence test may already lead to arbitrarily large errors in predicted causal e ect strengths, even when the faithfulness assumption is not violated. Our results may not be surprising for those familiar with the work of [9], but we believe that the analysis we present here may be easier to understand to those without a background in statistics, as we separate statistical issues (the possibility of type II errors in the conditional independence test from a nite sample) from a rather straightforward analysis of the problem in the population setting. We use an algebraic approach, showing how causal prediction may lead to wrong predictions already in the simple context of linear structural equation models with a multivariate Gaussian distribution. In Section 1, we begin with a brief description of the problem setting in a formal way, giving the de nitions of the causal e ect, instrumental variable, LCD algorithm and the toy model we present. We consider three observed random variables (X1; X2; X3), which is the minimal number such that a non-trivial conditional independence test can be obtained. In Section 2, we show how an (arbitrarily weak) conditional dependence that goes undetected can in uence our estimation of the causal e ect of X2 on X3 from the observed covariance matrix, when a confounder between X2 and
arbitrarily large error in the estimated causal e ect as the noise variance of X2 approaches zero. We nish with conclusions in Section 3. 1 1.1
The model we are interested in arises from the work of Cooper [1], who proposed the \LCD" algorithm for causal discovery in observational databases and the more recent paper of Chen et al.[8], who proposed the \Trigger" algorithm to infer transcriptional regulatory networks among genes. Throughout this section we will assume:
De nition 1.1. (LCD setting) Given three random variables X1; X2; X3 such that the following statistical properties and prior assumptions are satis ed: Statistical dependences:
X1 6?? X2 X2 6?? X3
X1 ?? X3jX2
An(X1) \ fX2; X3g = ;
where An(X) is the set of the causal ancestors of X (which includes X itself), so this condition means that we assume that X1 is not caused by the other observed variables X2; X3.
Theorem 1.1. Under the assumptions in De nition 1.1, the causal structure must be a subgraph of: X1
X2
X3 Here, the directed arrows indicate a direct causal relationship and the bidirected edge denotes an unobserved confounder.
Our primary interest is to predict p(X3jdo(X2)), the distribution of X3 after an intervention on X2. In general, this quantity may di er from p(X3jX2), the conditional distribution of X3 given X2 [13]. In the linear-Gaussian case, the quantity @E(X3jdo(X2)) measures the causal e ect of X2 on X3.
Corollary 1.1. Under the LCD assumptions in Definition 1.1,
p(X3jdo(X2)) = p(X3jX2): Therefore, in the linear-Gaussian case, the quantity @E(X3jdo(X2)) 1.2
Relationship with instrumental variables
De nition 1.2. (Instrumental Variable setting)
an instrumental variable if the following conditions are satis ed: Statistical dependences:
X1 6?? X2
X1 ?? X3jdo(X2)
pendent after an intervention on the variable X2. In terms of the causal graph, this means that all the unblocked paths between X1 and X3 contain an arrow that points to X2.
X1 X2 X3
lowed, in contrast with the LCD setting. Note that the LCD setting is a special case of the IV model. Lemma 1.1. Under the IV assumptions in De nition 1.2 and for the linear-Gaussian case, the quantity Cov(X1; X3)
Cov(X1; X2) is a valid estimator for the causal e ect of X2 on X3. 1.3
ditional independence test X1 ?? X3jX2 in the LCD setting depends on the sample size. Indeed, it could be hard to distinguish a weak conditional dependence
X1 6?? X3jX2 from a conditional independence using a sample of nite size. Here we study the question of what happens to our prediction of the causal e ect of X2 on X3 if the conditional independence test encounters a type II error (i.e., erroneously accepts the null hypothesis of independence).
Note that a type I error (i.e., erroneously rejecting the null hypothesis of independence) in the tests X1 6?? X2 and X2 6?? X3 will not be as dangerous as a type II error in the conditional independence test. Indeed, the probability of a type I error can be made arbitrarily small by tuning the signi cance level appropriately. In addition, a type I error would let the LCD algorithm reject a valid triple, i.e., lower the recall instead of leading to wrong predictions.
For these reasons we study the model described in the following de nition, which allows the presence of a hidden confounder X4, and a direct e ect from X1 on X3 (not mediated via X2). We assume that these additional features result in a possible weak conditional dependence between X1 and X3 given X2. For simplicity we consider only the linear-Gaussian case. We also assume no confounders between X1 and X2, or between X1 and X3, or between X1; X2; X3. This simpli cation will not in uence the nal result of the paper, because we will prove how unboundedness of the causal e ect estimation error is already achieved for this special case.
X4
X2 X1
X3 which is one of the possible causal structures that is compatible with the following conditions: Statistical dependences:
X1 6?? X2 X2 6?? X3 A weak conditional dependence
X1 6?? X3jX2
An(X1) \ fX2; X3g = ;
The joint distribution of the observed variables is assumed to be a multivariate Gaussian distribution with covariance matrix and zero mean vector. We also assume that the structural equations of the model are linear. Then where is the vector of the extended system,
X = AX + E; X =
X1; : : : ; X4 E = E1; : : : ; E4
T
T
(
N 0; = diag i2 ; and A = ( ij ) 2 M4 R is (up to a permutation of indices) a real upper triangular matrix in the space M4(R) of real 4 4 matrices that de nes the causal strengths between the random variables of the system.
the confounder X4 is used, by using non-zero covariance between the noise variables E2; E3. It can be shown that for our purposes, the two representations are equivalent and yield the same conclusions.
Lemma 1.2. Given a set of three random variables (X1; X2; X3) with a multivariate Gaussian distribution the conditional independence
X1 ?? X3 j X2
is equivalent to a vanishing partial correlation
13 2 = q 1
13
23:
In contrast with the LCD model in De nition 1.1, the
equality (
E X3jdo(X2) j = jg A;
jjX2j;
where the \causal e ect estimation error" is given by:
(
Estimation of the causal e ect error from the observed covariance matrix
Proposition 2.1. The mapping : (A; ) 7! that maps model parameters (A; ) to the observed covariance matrix according to the model in De nition 1.3 is given by: 11 = 12 12 = 13 = 11 23 = 11 22 = 11 33 =
2 12 1
Remark 2.1. (Instrumental variable estimator)
From equation (
are always free to choose the scale for the unobserved confounder X4: Lemma 2.1. The equations of proposition 2.1 are invariant under the following transformation 4j =
Remark 2.2. (Geometrical Interpretation) From a geometrical point of view the joint system of equations for the observed covariance matrix de nes a manifold M in the space of the model parameters M4(R) D 2 , where M4(R) is the space of the possible causal strengths ij and
D 2 =
3
Y[0; ii]
i=1
is the compact hypercube of the noise variances. Note
that we have used the symmetry 44 = 42 = 1 and
that
2
i
ii
from equations (
results of this paper. We will prove that there still remain degrees of freedom in the noise variances 22; 32 and the signs s1; s2, given the observed covariance matrix , that will lead to an unbouded causal e ect estimation error g(A; ).
Theorem 2.1. Given the causal model in De nition 1.3, there exists a map : M3(R)
D( ) f 1; +1g2 !
M4(R) (
Corollary 2.1. It is possible to express the error in the estimated causal e ect as g
( ; 22; 32; s1; s2); s1s2 pdet m 32pm mp 2 2 = # 12 + m 22 11 22 : By optimizing over 32 we get: with b ( 22) =
m The length of the interval [b ; b+] is a function of ( ; 22) and satis es 11 22 : (21) 23 2 [b ; b+]
R; pdet pm mp 2
2 23 = 22 m + # 12 : m 22
From equation (
[b ; b+]. Its length is a decreasing function of 22.
tion (18) is unbounded. This means that for all the choices of the observed covariance matrix that are in accordance with the model assumptions in De nition 1.3, the set of model parameters (A; ) 2 M that would explain leads to an unbounded error g.
2 = 0, which corresponds to making the random 2 variable X2 deterministic with respect to its parents X1; X4. Figure 1 shows the singularity of the function jg( ; 22; 32)j in the limit 22 ! 0. The rate of growth is proportional to the inverse of the standard deviation of the noise variable E2:
1 jgj / 2 as 2 ! 0.
(22)
Remark 2.3. (Lower bound for 22) Corollary 2.1 is the main result of our analysis. The right hand term in (20) consists of two terms: the rst one, through #, represents the contribution of the partial correlation, and is small if 13 2 is small. The second term is a fundamental, intrinsic quantity not controllable from the conditional independence test and the sample size. However, in situations where one is willing to assume a lower bound on 22: it is possible to give a con dence interval [b+; b ] for the function g, depending on the choice of the lower bound ^22.
Remark 2.4. (IV estimation error) In the instrumental variable literature the IV estimator is used, presented in Lemma 1.1. Unfortunately, this estimator and its error function 13 (23) is proportional to 13 and from (17) one can deduce a similar growing rate of the function h in terms of the variance of the noise term E2: jhj / 1 2 as 2 ! 0.
(24) Remark 2.5. (Singularity analysis) Figure 2 shows a contour plot of jgj on the rectangle D( ) 3 ( 22; 32). The singularity in the causal e ect function g is reached in the degenerate case, when the conditional distribution of X2 given X1 and X4 approaches a Dirac delta function. This cannot be detected empirically, as we can still have well-de ned covariance matrices of the observed system even if the covariance matrix of the extended one is degenerate. Let us investigate in detail the limit for 22 ! 0 from the point of view of the causal model. This proposition will show a simple example of how the causal strengths can be arbitrarily large, keeping the entries of the observed covariance matrix ij nite. Proposition 2.2. Assume that the observed covariance matrix is positive-de nite. Then, for the limit 2 2 ! 0 we have the following scenario for the causal strength parameters: 8 > 23 <
43 > : 13
1
2
sgn( 42) 2 1
sgn(
X1 12 X2 1
1 1
X3 Corollary 2.1 shows how the causal e ect estimation error can be extremely sensitive to small perturbations of our model assumptions. Equation (20) holds for any value of # (which is proportional to the partial correlation 13 2) and the second term vanishes when the confounder is not present. This shows that with a nite sample, a type II error in the conditional independence test may lead to an arbitrarily large error in the estimated causal e ect. Even in the in nite sample limit, this error could be arbitrarily large if faithfulness is violated. The result is in agreement with the results in [9], and it shows in a clear algebraic way how type
wrong conclusions.
We believe that this conclusion holds more generally: even when we increase the complexity and the number of observed variables, the in uence of confounders will still remain hidden, mixing their contribution with the visible parameters, thereby potentially leading to arbitrarily large errors. This means that for individual cases, we cannot give any guarantees on the error in the estimation without making further assumptions.
this negative worst-case analysis can be supplemented with more positive average-case analysis of the estimation error. Indeed, this is what one would hope if
Study more complex models, in terms of the number of nodes, edges and cycles. Bayesian model selection: We hope that the
Bayesian Information Criterion: We could directly assign a score based on the likelihood function of the data given the model parameters (A; ) and the model complexity, without assuming any prior distribution for the model parameters.
Nonlinear structural causal equations: To deal with nonlinearity it is possible to consider Spearman's correlation instead of the usual one, using the following relationships: m = = = ! = # = \Environment" variable: In many applications in biology, for example where X1 is genotype, X2 gene expression and X3 phenotype, the observed random variables X2 and X3 are strongly dependent on the environmental conditions of the experiment. It might be reasonable to assume that most of the external variability is carried by the covariance between the environment variable W and the other measured ones, including possible confounders. This leads to the following graphical model, which could be useful in deriving some type of guarantees for this scenario:
W X4
X2 X1
X3