Post on 16-Jan-2016
transcript
1
Searching for Causal Models with Latent
Variables
Peter Spirtes, Richard Scheines, Joe Ramsey, Erich
Kummerfeld, Renjie Yang
2
What is the Causal Relation Between Economic Stability and Political Stability?
Economicalstability
Politicalstability
Economicalstability
Politicalstability
Economicalstability
Politicalstability
Economicalstability
Politicalstability
L
?
?
?
?
3
Measure Latents with Indicators
Country XYZ
1. GNP per capita: _____2. Energy consumption per capita: _____3. Labor force in industry: _____4. Ratings on freedom of press: _____5. Freedom of political opposition: _____6. Fairness of elections: _____7. Effectiveness of legislature _____
Task: learn causal model
4
To draw causal conclusions about the unmeasured Economical stability and Political stability variables we are interested in, usehypothesized causal relations between X’s , Es
and Psstatistics gathered on X’s (correlation matrix)
Multiple Indicator ModelsEconomical
stabilityPoliticalstability
?
Pure Measurement Model
X1 X2 X3 X5 X6 X7X4
L1 L3
X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 X11 X12 X13
L2 L4
Structural Model – Two Factor Model
L1 L3
X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 X11 X12 X13
L2 L4
Measurement Model
L1 L3
X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 X11 X12 X13
L2 L4
Impurities
8
A pure n-factor measurement model for an observed set of variables O is such that:Each observed variable has exactly n latent
parents.No observed variable is an ancestor of other
observed variable or any latent variable. A set of observed variables O in a pure n-
factor measurement model is a pure cluster if each member of the cluster has the same set of n parents.
Pure Measurement Models
Alternative Models
L1
X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 X11 X12 X13 Bifactor
L2 L3
X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 X11 X12 X13 Higher-Order
L1 L3 L2
Higher-Order ⊂ Bifactor ⊂ Connected Bifactor ⊂ Connected Two-Factor
L1 L3
X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 X11 X12 X13
L2 L4
1. Estimate and test pure Higher-order model. 2. Estimate and test pure Two-Factor model. 3. Choose whichever one fits best.
Common Strategy
If a measurement model is impure, and you assume it is pure, this will hinder the inference of the correct structural model.
If a higher-order model has impurities, it will fit a more inclusive pure model such as a pure two-factor model better than a pure higher-order model.
Two Problems
L1 L3
X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 X11 X12 X13
L2 L4
Generating Model
Finding the Structural Model
L1 L3
X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 X11 X12 X13
L2 L4
Data fits model with black edges + pure measurement model better than model without black edges + pure measurement model.
Finding the Structural Model
L1 L3
X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 X11 X12 X13
Generating Model
Finding the Right Kind of Measurement Model
L1 L3
X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 X11 X12 X13
Worse Fit
Finding the Right Kind of Measurement Model
L1 L3
X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 X11 X12 X13
L2 L4
Better Fit
Finding the Right Kind of Measurement Model
L1 L3
X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 X11 X12 X13
Generating Model
1. Identify pure submodel {1,2,3,4,5,8,9,10,11,12,13}. 2. See if it fits Higher-order.3. If it does select Higher –order, otherwise see if it fits Two-Factor model.
Finding the Right Kind of Measurement Model
L1 L3
X1 X2 X3 X4 X5 X8 X9 X10 X11 X12 X13
Pure submodel fits Higher-order model, so select Higher-order.
Alternative Strategy?
L1 L3
X1 X2 X3 X4 X5 X8 X9 X10 X11 X12 X13
L2 L4
Data will also fit Two-Factor model (slightly lower chi-squared), but when adjusted for degrees of freedom, p-value will be lower.
Alternative Strategy?
?
Rank Constraints
21
An algebraic constraint is linearly entailed by a DAG if it is true of the implied covariance for every value of the free parameters (the linear coefficients and the variances of the noise terms)
Entailed Algebraic Constraints
L1 L3
X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 X11 X12 X13
L2 L4
Trek and Sides of Treks
L1 L3
X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 X11 X12 X13
L2 L4
(CA:CB) trek-separates A from B iff every trek between A and B intersects CA on the A side or CB on the B side.
Trek-Separation
L1 L3
X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 X11 X12 X13
L2 L4
< {L1,L2}, ∅> Trek-Separate {1,2,3}:{8,9,10}
L1 L3
X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 X11 X12 X13
L2 L4
<∅,{L3,L4}> Trek-Separate {1,2,3}:{8,9,10}
L1 L3
X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 X11 X12 X13
L2 L4
If (CA:CB) trek-separates A from B, and the model is an acyclic linear Gaussian model, then rank(cov(A,B)) ≤ #CA + #CB.
Theorem (Sullivant, Talaska, Draisma)
L1 L3
X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 X11 X12 X13
L2 L4
< {L1,L2}, ∅> Trek-Separate {1,2,3}:{8,9,10}
L1 L3
X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 X11 X12 X13
L2 L4
If #CA + #CB ≤ #C’A + #C’B for all (C’A:C’B) that trek-separate A from B, then for generic linear acyclic Gaussian models, rank(cov(A,B)) = #CA + #CB.
Theorem (Sullivant, Talaska, Draisma)
L1 L3
X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 X11 X12 X13
L2 L4
If #CA + #CB > r for all (CA:CB) that trek-separate A from B in DAG G, then for some linear Gaussian parameterization, rank(cov(A,B)) > r.
Theorem (Sullivant, Talaska, Draisma)
L1 L3
X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 X11 X12 X13
L2 L4
{1,2,3}:{10,11,12} linear acyclic below <{L1,L2}, ∅>
Linear Acyclic Below the Choke Sets
f(L1,εL3)
g(L2,εL4)
L1 L3
X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 X11 X12 X13
L2 L4
{1,2,3}:{10,11,12} not linear acyclic below < ∅, {L1,L2}>
Linear Acyclic Below the Choke Sets
f(L1,εL3)
g(L2,εL4)
L1 L3
X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 X11 X12 X13
L2 L4
If (CA:CB) trek-separates A from B, and model is linear acyclic below (CA:CB) for A, B, then rank(cov(A,B)) ≤ #CA + #CB.
Theorem (Spirtes)
ProofCA
… …
full rank
A B
CB
L1 L3
X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 X11 X12 X13
L2 L4
If #CA + #CB > r for all (CA:CB) that trek-separate A from B in DAG G, then for some linear acyclic below (CA:CB) for A, B parameterization, rank(cov(A,B)) > r.
Theorem (Spirtes)
If a rank constraint is not entailed by the graphical structure, then the rank constraint does not hold.
If the constraints do not hold for the whole space of parameters (i.e. they are not entailed), but are the roots of rational equations in the parameters, they are of Lebesgue measure 0.
Faithfulness Assumption
This says nothing about the measure of constraints that are not entailed but “almost” hold (i.e. cannot be distinguished from 0 reliably given the power of the statistical tests.)
However, the performance of the algorithm will not depend upon the extent to which individual non-entailed constraints “almost” hold, but the extent to which sets of non-entailed constraints “almost” hold.
This depends upon which sets of constraints affect the performance of the algorithm, and the joint distribution of the constraints which we do not know.
Faithfulness Assumption
37
AdvantagesNo need for estimation of model.
No iterative algorithmNo local maxima.No problems with identifiability.Fast to compute.
DisadvantagesDoes not contain information about
inequalities.Power and accuracy of tests?Difficulty in determining implications among
constraints
Advantages and Disadvantages of Algebraic Constraints
Find a list of pure pentads of variable.
Merge pentads on list that overlap.Select which merged subsets to
output.
Find Two Factor Clusters (FTFC) Algorithm
L1 L3
X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 X11 X12 X13
L2 L4
For each subset of size 5, if it is Pure, add to PureList.
{1,2,3,4,5}; {9,10,11,12,13}; {8,10,11,12,13}; {8,9,11,12,13};{8,9,10,12,13}; {8,9,10,11,12}
1. Construct a List of Pure Fivesomes
L1 L3
X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 X11 X12 X13
L2 L4
<{L1,L2},∅> Trek-Separate All Partitions of {1,2,3,4,5,x}
Test for Purity of {1,2,3,4,5}
L1 L3
X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 X11 X12 X13
L2 L4
No Pair Trek-Separate All Partitions of {1,2,3,4,8,x}, e.g. {1,2,8}:{3,4,9}
Test for Purify of {1,2,3,4,8}
L1 L3
X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 X11 X12 X13
L2 L4
No Pair Trek-Separates All Partitions of {1,2,3,4,6,x}, e.g. {1,2,6}:{3,4,7}
Test for Purify of {1,2,3,4,6}
L1 L3
X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 X11 X12 X13
L2 L4
No Pair Trek-Separate {1,2,3}:{7,8,9}
L1 L3
X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 X11 X12 X13
L2 L4
{1,2,3,4,5}; {9,10,11,12,13}; {8,10,11,12,13}; {8,9,11,12,13};{8,9,10,12,13}; {8,9,10,11,12} → {1,2,3,4,5}; {8,9,10,11,12,13}
2. Merge Overlapping Items - Theory
L1 L3
X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 X11 X12 X13
L2 L4
{1,2,3,4,5}; {9,10,11,12,13}; {8,10,11,12,13}; {8,9,11,12,13};{8,9,10,12,13}; {8,9,10,11,12}; {1,2,3,8,9} (false positive)
2. Merge Overlapping Items - Practice
L1 L3
X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 X11 X12 X13
L2 L4
{9,10,11,12,13}; {8,10,11,12,13} → {8,9,10,11,12,13};
All subsets of size 5 of {8,9,10,11,12,13} are in PureList so accept merger, and remove both from PureList.
2. Merge Overlapping Items - Practice
L1 L3
X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 X11 X12 X13
L2 L4
{1,2,3,4,5}; {1,2,3,8,9} → {1,2,3,4,5,8,9}
All subsets of size 5 except {1,2,3,8,9} and {1,2,3,4,5}not on PureList – so reject merger
2. Merge Overlapping Items - Practice
L1 L3
X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 X11 X12 X13
L2 L4
{1,2,3,4,5}; {8,9,10,11,12,13}; {1,2,3,8,9}
2. Final List
L1 L3
X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 X11 X12 X13
L2 L4
{1,2,3,4,5}; {8,9,10,11,12,13}; {1,2,3,8,9}Output {8,9,10,11,12,13} because it is largest. Output {1,2,3,4,5} because it is next largest that is disjoint.
3. Select Which Ones to Output
IfThe causal graph contains as a subgraph a pure 2-
factor measurement model with at least six indicators and at least 5 variables in ech cluster;
The model is linear acyclic below the latent variables;
Whenever there is no trek between two variables they are independent;
There are no correlations equal to zero or one;The distribution is LA faithful to the causal graph;
then the population FTFC algorithm outputs a clustering in which any two variables in the same output cluster have the same pair of latent parents.
Theorem
L1 L3
L5
X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 X11 X12 X13
L6
L2 L4
Undetectible Impurities
X1 X2 X3 X4 X5 X6Spider Model (Sullivant, Talaska, Draisma)
Alternative Models with Same Constraints
L
L1 L2
L3 L4
L5 L6
53
However, the spider model (and the collider model) do not receive the same chi-squared score when estimated, so in principle they can be distinguished from a 2-factor model. ExpensiveRequires multiple restartsNeed to test only pure clustersIf non-Gaussian, may be able to detect
additional impurities.
Checking with Estimated Model
In case of linear pure single factor models (with at least 3 indicators per cluster), all of the latent-latent edges are guaranteed to be identifiable.
Can apply causal search model using the estimated covariance matrix among the latents as input.
Inferring Structural Model
L1 L3
X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 X11 X12 X13
L2 L4
Non-identified edges in Two-Factor Model
56
For sextads, the first step is to check 10 * n choose 6 sextads.
However, a large proportion of social science contexts, there are at most 100 observed variables, and 15 or 16 latents. If based on questionairres, generally can’t get
people to answer more questions than that. Simulation studies by Kummerfeld indicate that
given the vanishing sextads, the rest of the algorithm is subexponential in the number of clusters, but exponential in the size of the clusters.
Complexity
ΣIJ is the I×J submatrix of the inverse of ΣIJ,
and ΣIJ×IJ is the (I ∪ J) × (I ∪ J) submatrix of Σ. This can be turned into a statistical test by substituting the maximum likelihood estimate of Σ in for the population values of Σ.
Drton Test – Assuming Normality
τ is a column vector of independent population sextad differences implied by a model to vanish
t is a column vector of corresponding sample sextad differencesσ is a column vector of covariances that appear in one of more
vanishing sextad differences in tΣss is the covariance matrix of the limiting distribution of
sample covariances appearing in t, σefgh is the fourth order moment matrix.
Delta Test – Asymptotically Normal
59
Problems in Testing ConstraintsTests require (algebraic) independence among
constraints.
Additional complication – when some correlations or partial correlations are non-zero, additional dependencies among constraints arise
Some models entail that neither of a pair of sextad constraints vanish, but that they are equal to each other
3 hypothesized latent variables: Stress, Depression, and (religious) Coping.
21 indicators for Stress, 20 each for Depression and Coping
n = 127
Application to Depression Data
Lee modelp(χ2) = 0
Application to Depression Data
Silva et al. modelp(χ2) = .28
Application to Depression Data
Silva et al. modelp(χ2) = .28
Application to Depression Data
The current version of the FTFC algorithm cannot be applied to all 61 measured indicators in the Lee data set as input in a feasible amount of time.
We applied it at several different signicance levels to look for 2-pure sub-models of the 3 original given subsets of measured indicators.
We ran the FTFC algorithm at a number of dierent significance levels. Using the output of FTFC as a starting point, we searched for a model that had the highest p-value using a chi-squared test.
The best model that we found contained a cluster of 9 coping variables, 8 stress variables, and 8 depression variables (all latent variables directly connected).
p(χ2) = 0.27.
Application to Depression Data
L1 L3 L5
X1 X2 X3 … X10 X11 … X20 X21 … X30
L2 L4 L6
Generated from model, and pure submodel. 3 sample sizes: n = 100 (alpha = .1), 500 (alpha = .1), 1000 (alpha = .4).
Non-linear function are convex combination of linear and cubic.
Simulation Studies
Purity
P/I – Generated from pure/impure submodelL/N – Generated from linear/non-linear latent-latent functionsL/N – Generated from linear/non-linear latent-measured connectionsPurity – percentage of output cluster from same pure subcluster.
The average number of clusters output ranged between 2.7 and 3.1 for each kind of model and sample size, except for PNN (pure submodel, non-linear latent-latent and latent-measured functions.)
For PNN at sample sizes 100, 500, and 1000 average number of clusters were 1.05, 1.38, and 1.54 respectively.This is expected, because non-linear latent-
measurd connections violates the assumptions under which the algorithm is correct.
Number of Clusters
The percentage of each pure subcluster that was in the output cluster.
Fraction of Possible Output
Larger clusters are more stably produced and more likely to be (almost) correct.
Informal Observation
70
Described algorithm that relies on weakened assumptionsWeakened linearity assumption to linearity
below the latentsWeakened assumption of existence of pure
submodels to existence of n-pure submodelsConjecture correct if add assumptions of no
star or collider models, and faithfulness of constraintsIs there reason to believe in faithfulness of
constraints when non-linear relationships among the latents?
Summary
71
Give complete list of assumptions for output of algorithm to be pure.
Speed up the algorithm.Modify algorithm to deal with almost
unfaithful constraints as much as possible.Add structure learning component to output
of algorithm. Silva – Gaussian process model among latents,
linearity below latentsIdentifiability questions for stuctural models
with pure measurement models.
Open Problems
72
Silva, R. (2010). Gaussian Process Structure Models with Latent Variables. Proceedings from Twenty-Sixth Conference Annual Conference on Uncertainty in Artificial Intelligence (UAI-10).
Silva, R., Scheines, R., Glymour, C., & Spirtes, P. (2006a). Learning the structure of linear latent variable models. J Mach Learn Res, 7, 191-246.
Sullivant, S., Talaska, K., & Draisma, J. (2010). Trek Separation for Gaussian Graphical Models. Ann Stat, 38(3), 1665-1685.
References
73
Drton, M., Massam, H., and Olkin, I. (2008) Moments of minors of Wishart matrices, Annals of Statistics 36, 5, pp. 2261-2283.
Drton, M., Sturmfels, B., Sullivant, S. (2007) Algebraic factor analysis: tetrads, pentads and beyond, Probability Theory and Related Fields, 138, 3-4, 463-493
Harman, H. (1976) Modern Factor Analysis, University of Chicago Press Books
References