Counterfactual Normalization: Proactively Addressing Dataset Shift andImproving Reliability Using Causal Mechanisms

Adarsh SubbaswamyDepartment of Computer Science

Johns Hopkins UniversityBaltimore, MD 21218

Suchi SariaDepartment of Computer Science

Johns Hopkins UniversityBaltimore, MD 21218

Abstract

Predictive models can fail to generalize fromtraining to deployment environments becauseof dataset shift, posing a threat to model relia-bility and the safety of downstream decisionsmade in practice. Instead of using samplesfrom the target distribution to reactively cor-rect dataset shift, we use graphical knowledgeof the causal mechanisms relating variables ina prediction problem to proactively remove re-lationships that do not generalize across envi-ronments, even when these relationships maydepend on unobserved variables (violations ofthe “no unobserved confounders” assumption).To accomplish this, we identify variables withunstable paths of statistical influence and re-move them from the model. We also augmentthe causal graph with latent counterfactual vari-ables that isolate unstable paths of statisticalinfluence, allowing us to retain stable paths thatwould otherwise be removed. Our experimentsdemonstrate that models that remove vulner-able variables and use estimates of the latentvariables transfer better, often outperforming inthe target domain despite some accuracy loss inthe training domain.

1 INTRODUCTION

Classical supervised machine learning methods for pre-diction problems assume that training and test data areindependently and identically distributed from a fixeddistribution over the input features X and target outputlabel T , p(X, T ). When this assumption does not hold,training with classical frameworks can yield unreliablemodels and, in the case of safety-critical applicationslike medicine, dangerous predictions (Dyagilev and Saria,

2015; Caruana et al., 2015; Schulam and Saria, 2017). Un-reliable models may have performance that is not stable—model performance varies greatly when the test distribu-tion is different from the training distribution in scenarioswhere invariance to the underlying changes are desirableand expected. Unreliability arises because models are of-ten deployed in dynamic environments that systematicallydiffer from the one in which the historical training datawas collected—a problem known as dataset shift whichresults in poor generalization. Most existing methods foraddressing dataset shift are reactive: they use unlabeleddata from the target distribution during the learning pro-cess (see Quionero-Candela et al. (2009) for an overview).However, when the differences in environments are un-known prior to model deployment (e.g., no available datafrom the target environment), it is important to under-stand what aspects of the prediction problem can changeand how we can train models that will be robust to thesechanges. In this work we consider this problem of proac-tively addressing dataset shift for discriminative models.

To illustrate, we will consider diagnosis, a problem com-mon to medical decision making. The goal is to detect thepresence of a target condition T . The features used canbe split into three categories: risk factors for the targetcondition (causal antecedents), outcomes or symptoms ofthe condition (causal descendents), and co-parents thatserve as alternative explanations for the observations (e.g.,comorbidities and treatments). The causal mechanisms(directional knowledge of causes and effects, e.g., betablockers lower blood pressure) relating variables in a pre-diction problem can be represented using directed acyclicgraphs (DAGs), such as the one in Figure 1a. As anexample (Figure 1b), a hospital may wish to screen formeningitis T , which can cause blood pressure (BP) Yto drop dangerously low. Smoking D is a risk factor formeningitis, and also causes heart disease for which pa-tients are prescribed beta blockersC (a type of medicationthat lowers blood pressure). However, domain-dependentconfounding (Figure 1b) and selection bias (Figure 1c)

arX

iv:1

808.

0325

3v1

[st

at.M

L]

9 A

ug 2

018

can cause certain distributions in the graph to changeacross domains, resulting in dataset shift.

Consider domain-dependent confounding in which rel-evant variables may be unobserved and distributions in-volving these variables may change across domains. Indiagnosis, unobserved variables are likely to be risk fac-tors (e.g., behavioral factors, genetics, and geography)that confound the relationship between the target condi-tion and comorbidities/treatments. For example (Figure1b), smoking (D) may not be recorded in the data, and thepolicy used to prescribe beta blockers to smokers p(C|D)will vary between doctors and hospitals. When D isobserved, the changes in the prescription policy can beadjusted for. More generally, others have described solu-tions to ensuring model stability across environments withdifferences in policies (Schulam and Saria, 2017). Specif-ically, they optimize the counterfactual risk to explicitlyaccount for variations in policy between train and testenvironments (e.g., Swaminathan and Joachims (2015);Schulam and Saria (2017)). However, this requires ig-norability assumptions (also known as the no unobservedconfounders assumption in causal inference), that maynot hold in practice (such as when D is not observed).Violations of this assumption have implications on modelreliability. For example, in Figure 1b by d-separation(Koller and Friedman, 2009) C has two active paths to Twhen conditioned on Y : C ← D → T and C → Y ← T .The first path is unstable because it contains an edgeD → C encoding the distribution that changes betweenenvironments p(C|D). The second path, however, en-codes medical effects that are stable—p(Y |T,C) doesnot change. Naively including C and Y in the modelwill capture both paths, leaving the model vulnerable tolearning the relationship along the unstable path.

Similarly, selection bias (Figure 1c) adds auxiliary vari-ables to the graph (i.e., S) which can create unstable pathsthat contribute to model unreliability. Certain subpopu-lations with respect to the target and comorbidities maybe underrepresented in the training data (S = 1). Forexample, patients without meningitis who take beta block-ers (T = 0, C = 1) may be underrepresented becausethey rarely visit the hospital due to a local chronic carefacility which helps them manage their chronic condition.This introduces a new unstable active path from C to T :C → S ← T . As before, the path through Y remainsstable. In the case of selection bias or domain-dependentconfounding, can we remove the influence of unstablepaths while retaining the influence of stable paths?

We propose removing vulnerable variables—variableswith unstable active paths to the target— from the con-ditioning set of a discriminative model in order to learnmodels that are stable to changes in environment. In Fig-

Figure 1: (a) General diagnosis DAG. (b) The DAG cap-turing causal mechanisms for the medical screening exam-ple. The features are blood pressure Y and beta blockersC. The target label T is meningitis. Smoking D is unob-served. (c) Selection bias S is introduced.

ure 1, this means we must remove C from the model. Indoing so, Y becomes vulnerable as well because of thepaths Y ← C ← D → T in 1b and Y ← C → S ← Tin 1c, so we must remove Y . While this removes allunstable paths, it also removes stable paths (in fact, itremoves all stable paths in this example). However, incertain situations we describe, we can retain some of thestable paths between the target vulnerable variables byconsidering counterfactual variables. In our example, ifwe somehow knew an adjusted counterfactual value of Y ,denoted Y (C = ∅)—the value of Y for which the effectsof C were removed (e.g., the blood pressure had the pa-tient not been treated)—then this adjusted Y would onlycontain the information along the stable path T → Y .This concept is inspired by potential outcomes in causalinference and allows us to retain stable paths that wouldotherwise be removed along with the unstable paths.

Contributions: First, we identify variables which makea statistical model vulnerable to learning unstable rela-tionships that do not generalize across datasets (due toselection bias or unobserved domain-dependent confound-ing) which must be removed from a discriminative modelfor its performance to be stable. Second, we define a node-splitting operation which modifies the DAG to containinterpretable latent counterfactual variables which isolateunstable paths allowing us to retain some stable paths in-volving vulnerable variables. By allowing unstable pathsto depend on unobserved variables, we generalize previ-ous works that learn stable models by assuming there areno unobserved confounders, intervening on the unstablepolicy, and predicting potential outcomes (see e.g., Schu-lam and Saria (2017)). Third, we provide algorithms fordetermining stable conditioning sets and which counter-factuals to estimate. Fourth, we explain how including thelatent features can make a classification problem measur-ably simpler due to their reduced variance. In simulatedand real data experiments we demonstrate that our methodimproves stability of model performance.

2 RELATED WORK

Proactive and Reactive Approaches: Reactive predic-tive modeling methods for countering dataset shift typi-cally require representative unlabeled samples from thetest distribution (Storkey, 2009). These methods work byre-weighting the training data or extracting transferablefeatures (e.g., Shimodaira (2000); Gretton et al. (2009);Gong et al. (2016); Zhang et al. (2013)). To proactivelyaddress perturbations of test distributions, recent workconsiders formal verification methods for bounding theperformance of trained models on perturbed inputs (e.g.,Raghunathan et al. (2018); Dvijotham et al. (2018)). Com-plementary to this, others have developed methods basedon distributional robustness for training models to be min-imax optimal to perturbations of bounded magnitude inorder to guard against adversarial attacks (Sinha et al.,2018) and improve generalization (Rothenhausler et al.,2018). We consider the related problem of training mod-els that are stable to arbitrary shifts in distribution.

Beyond predictive modeling, previous work has consid-ered estimation of causal models in the presence of se-lection bias and confounding. For example, Spirtes et al.(1995) learn the structure of the causal DAG from data af-fected by selection bias. Others have studied methods andconditions for identification of causal effects under simul-taneous selection and confounding bias (e.g., Bareinboimand Pearl (2012); Bareinboim and Tian (2015); Correaet al. (2018)). Correa and Bareinboim (2017) determineconditions under which interventional distributions areidentified without using external data.

Transportability: The goal of an experiment is for thefindings to generalize beyond a single study, a conceptknown as external validity (Campbell and Stanley, 1963).Similarly, in causal inference transportability, formalizedin Pearl and Bareinboim (2011), transfers causal effect es-timates from one environment to another. Bareinboim andPearl (2013) generalize this to transfer causal knowledgefrom multiple source domains to a single target domain.Rather than transfer causal estimates from source to tar-get, the proposed method learns a single statistical modelwhose predictions should perform well on the source do-main while also generalizing well to new domains.

Graphical Representations of Counterfactuals: Thenode-splitting operation we introduce in Section 3 is sim-ilar to the node-splitting operation in Single World Inter-vention Graphs (SWIGs) (Richardson and Robins, 2013).However, intervening in a SWIG results in a generativegraph for a potential outcome with the factual outcomeremoved from the graph. By contrast, our node-splittingoperation yields a modified generative graph of the factualoutcomes with new intermediate counterfactual variables.

Other graphical representations such as twin networks(Pearl, 2009) and counterfactual graphs (Shpitser andPearl, 2007) simultaneously represent factual and coun-terfactual outcomes, rather than the intermediate counter-factuals exploited in this work.

3 METHODS

3.1 BACKGROUND

Potential Outcomes

The proposed method involves the estimation of coun-terfactuals, which can be formalized using the Neyman-Rubin potential outcomes framework (Neyman, 1923;Rubin, 1974). For outcome variable Y and interventionA, we denote the potential outcome by Y (a): the valueY would have if A were observed to be a.

In general, the distributions p(Y (a)) and p(Y |A = a) arenot equal. For this reason, estimation of the distributionof the potential outcomes relies on two assumptions:

Consistency: The distribution of the potential outcomeunder the observed intervention is the same as the distribu-tion of the observed outcome. This implies p(Y (a)|A =a) = p(Y |A = a).

Conditional Ignorability: Y (a) ⊥⊥ A|X , ∀a ∈ A.There are no unobserved confounders. This impliesp(Y (a)|X,A = a′) = p(Y (a)|X,A = a).

Counterfactuals and SEMS

Shpitser and Pearl (2008) develop a causal hierarchy con-sisting of three layers of increasing complexity: asso-ciation, intervention, and counterfactual. Many worksin causal inference are concerned with estimating aver-age treatment effects—a task at the intervention layerbecause it uses information about the interventional dis-tribution p(Y (a)|X). In contrast, the proposed methodrequires counterfactual queries which use the distributionp(Y (a)|Y, a′, X) s.t. a 6= a′ 1. That is, given that weobserved an individual’s outcome to be Y under interven-tion a′, what would the distribution of their outcome havebeen under a different intervention a?

In addition to the assumptions for estimating potential out-comes, computing counterfactual queries requires func-tional or structural knowledge (Pearl, 2009). We can repre-sent this knowledge using causal structural equation mod-els (SEMs). These models assume variables Xi are func-tions of their immediate parents in the generative causalDAG and exogenous noise ui: Xi = fi(pa(Xi), ui). Rea-

1The distinction is that p(Y (a)|X) reasons about the effectsof causes while p(Y (a)|Y, a′, X) reasons about the causes ofeffects (see, e.g., Pearl (2015)).

Algorithm 1: Constructing a Stable Conditioning SetInput: Graph G, number of variables N , observed

variables O, target TOutput: Stable conditioning set Z, Vulnerable set VZ = O \ T ;V = ∅;for k = 1 to N − 1 do

Conditioned on Z, find the set A of active pathsstarting with T and ending at v ∈ Z of length k;

for active path a ∈ A dov = last variable in a;if a is unstable then

Z = Z \ v;V = V

⋃v;

soning counterfactually at the level of an individual unitrequires assumptions on the form of the functions fi andindependence of the ui, because typically we are inter-ested in reasoning about interventions in which the ex-ogenous noise variables remain fixed. We build on thisto estimate the latent counterfactual variables introducedwithin the proposed procedure.

3.2 COUNTERFACTUAL NORMALIZATION

3.2.1 Assumptions About Structure of the Graph

Counterfactual Normalization uses a DAG, G, that repre-sents the causal mechanisms relating variables in a predic-tion problem. Let O denote the observed variables, andT ∈ O be the target variable to predict (T is unobservedin test distributions). We make no further assumptionsabout the edges relating observed variables. Let ch(·) andpa(·) represent children and parents in G, respectively.

G can contain unobserved variables U, which we will useto represent domain-dependent confounding. An unob-served variable must have at least two children so that itconfounds the relationship between its children. Domain-dependent confounding occurs when p(U|pa(U)) orp(ch(U)|U) changes across domains. G can also containan additional variable S which represents the selectionmechanism that induces selection bias in the training data.The mechanism is given by p(S = 1|pa(S)) where pa(S)is assumed to be nonempty and S is always assumed tobe conditioned upon in the training domain.

3.2.2 Constructing a Stable Set

The goal of Counterfactual Normalization is to find a setof observed variables and adjusted versions of observedvariables that contains no active unstable paths while max-imizing the number of active stable paths it contains. First,we define an unstable path to be a path to the target T that

Algorithm 2: Node-splitting OperationInput: Graph G, node Y , observed parents of Y to

intervene upon POutput: Modified graph G∗1. Insert counterfactual node Y (P = ∅)2. Delete edges {x→ Y : x ∈ pa(Y ) \P}3. Insert edges {x→ Y (P = ∅) : x ∈ pa(Y ) \P}4. Insert edge Y (P = ∅)→ Y

contains variables or edges which encode a distributionthat can change across environments. These are edges in-volving unobserved variables U (domain-dependent con-founding) or the selection mechanism variable S. Thus,an unstable path is a path to T which contains S or avariable in U.

We can find a set, Z, of observed variables with no activeunstable paths using Algorithm 1, which considers activepaths of increasing length that begin with T , and removesvulnerable variables V reachable by unstable active paths.

Theorem 1. Algorithm 1 will result in a set Z that con-tains no unstable active paths to T .

Proof Sketch. We show that on iteration k, removing avariable from Z does not create an active unstable path toa member of Z of length ≤ k (see supplement).

We now consider expanding the stable conditioning setZ by including some variables in V or adjusted versionsof these variables. The adjusted versions are counterfac-tuals which we place on a modified DAG G∗ through aprocedure called node-splitting.

3.2.3 Node-Splitting

Assume each variable v ∈ G has a corresponding struc-tural equation in which it is a function of its parents andan exogenous, unobserved, and independent noise term:v = fv(pa(v), εv). We want to compute a counterfactualversion of v in which we remove the effects of (i.e., inter-vene upon) some of its parents. Denote the set of parentswe intervene upon as P. Given v’s factual value and thefactual values of P, we calculate the counterfactual valuev(P = ∅) (remove the effects of parents in P by interven-ing and setting these parents to “null”). In the diagnosisexample of Figure 1b, an example counterfactual variablewould be Y (C = ∅): the patient’s blood pressure if weremoved the effects of the treatments they were given.Note that we must observe the factual value of parents weintervene on—they must be observed variables.

Removing the effects of only a subset of the parents re-quires being able to consider the effects of a parent whileholding fixed the effects of the other parents of the vari-

able. For this reason, we assume that the effects of parentson children are independent—they have no interactions.We specifically consider additive structural equationswhich satisfy this requirement. Estimation of the counter-factuals requires fitting the relevant structural equationsusing the factual outcome data by maximum likelihoodestimation. We can now define the node-splitting oper-ation, which is given in Algorithm 2. Given a variableand the subset of its parents to intervene upon, we set theintervened parents to “null” and place a latent counterfac-tual version of the variable onto the graph as a parent ofits factual version. Unlike traditional SEM interventions,we retain the factual version of the parents we interveneon in the graph. The counterfactual version subsumes theparents (in the original graph G) of its factual version thatwere not intervened upon. The modified graph G∗ is anequivalent model of the factual data generating process.

The consequence of node-splitting is that while the factualversion of a variable may be vulnerable, after interveningon some of its parents its counterfactual version may nolonger be vulnerable. Consider a vulnerable variable vwhich, if added to a stable conditioning set Z, would yieldat least one unstable active path to T . If the unstable pathis of the form v ← X . . . T , then since X is a parent ofv we can intervene on X . After node-splitting the newpath would be v(X = ∅) → v ← X . . . T . Since v isnot conditioned on, this collider path is blocked. Thus,this unstable path is not active for v(X = ∅) thoughit was active for v. However, if the unstable path wereof the form v → X . . . T , then we cannot intervene onX (not a parent of v) and any counterfactual version ofv will inherit the unstable active path: v(∅) → v →X . . . T . The first case shows that for unstable paths froma vulnerable variable v that begin through a parent ofv, intervening on the parent yields a counterfactual inwhich these unstable paths are not active. The secondcase shows that unstable active paths from v that beginthrough a child of v cannot be removed by node-splitting.We can also intervene on a variable’s observed parents thatare not along unstable paths. As we discuss in Section 4,the potential benefit is that counterfactual variables havereduced variance than their factual versions.

A question remains: does conditioning upon a stable coun-terfactual version of a vulnerable variable cause any un-stable paths to become active? Conditioning on a variablecan only open collider paths, so the only cases we mustconsider are when the counterfactual is a collider or de-scendant of a collider. In these cases, the active pathsthat meet at the collider are reachable by the counter-factual through at least one of its parents. However, weknow that these paths are stable since the counterfactualis stable: we would have intervened on any parents whichwere along unstable paths. Thus, conditioning on a stable

Algorithm 3: Retaining Vulnerable VariablesInput: Graph G, Vulnerable set in reverse topological

order V, Stable set Z, Target TOutput: Final conditioning set Z′

Z′ = Z;for v ∈ V do

if v has no active stable paths w.r.t. Z′ thenpass;

else if v has no active unstable paths w.r.t. Z′ thenZ′ = Z′

⋃v

else if all unstable paths w.r.t. Z′ from v to T arethrough observed parents P ⊆ pa(v) of v then

Node-split and modify G;Z′ = Z′

⋃v(P = ∅);

elsepass;

Prune Z′ of variables with no active stable paths.

counterfactual does not activate any new unstable paths.

3.2.4 Adding to the Conditioning Set

After finding a stable set Z of observed variables to con-dition upon, we must consider adding back each of thevulnerable variables that were removed. First, there maybe variables with no unstable active paths because col-lider paths became inactive after these variables wereremoved from Z. Second, we know that if a variable’sactive unstable paths go through observed parents, we canintervene on those parents, node-split in G∗, and add thecounterfactual version to the conditioning set. Becauseconditioning on the counterfactual may open stable pathsinvolving its non-vulnerable parents, we want to makesure that non-vulnerable parents that may be in V (firstcase) are considered after the counterfactual. For this rea-son, we consider adding the variables in V to Z in reversetopological order. Algorithm 3 shows the procedure foradding variables to Z. We condition on the resulting set,Z′ and use it to predict T by modeling p(T |Z′).Theorem 2. Algorithm 3 does not activate any unstablepaths and results in a stable set Z′.

Proof Sketch. We show all branches in the algorithm donot activate unstable paths (see supplement).

3.2.5 An Example

To illustrate node-splitting and Counterfactual Normal-ization, consider the expanded domain-dependent con-founding diagnosis example in Figure 2a. C representsa chronic condition (e.g., heart disease), A representstreatments (e.g., beta blockers), and X represents age

Figure 2: (a) The DAG of causal mechanisms for theexpanded medical screening example. (b) The modifiedDAG after node-splitting yielding the latent signal valueunder no treatment or chronic condition Y (a, c). (c) Themodified DAG after node-splitting and additionally ad-justing for other covariates Y (a, c, x).

(a demographic risk factor). D ∈ U is an unobservedvariable, and we allow p(C|D) to vary across domains.

In finding a stable set Z, we remove C (unstable path oflength 2), and then A and Y (unstable paths of length 3)which yields Z = {X}. Now we consider the variablesin V = {C, Y,A} in reverse topological order. Y hasunstable active paths through A and C. Since they areobserved variables, we intervene on them to generatethe counterfactual Y (C = ∅, A = ∅) in Figure 1b andadd it to Z′ after node-splitting. Now consider A, whichhas no stable active paths to T so we do not add it toZ′. Similarly, C has no stable active paths. Thus, Z′ ={Y (C = ∅, A = ∅), X} is the conditioning set we woulduse to predict T by modeling p(T |Z′).

4 COMPLEXITY METRICS

Beyond removing unstable paths, what are other benefitsof the proposed method? For binary prediction problems,the geometric complexity (on the basis of euclidean dis-tance) of the class boundary of a dataset can decreasewhen using the latent counterfactual variables instead ofthe factual and vulnerable variables. This is similar tothe work of Alaa and van der Schaar (2017) who use thesmoothness of the treated and untreated response surfacesto quantify the difficulty of a causal inference problem.To measure classifier-independent geometric complexitywe will use two types of metrics developed by Ho andBasu (2000, 2002): measures of overlap of individualfeatures and measures of separability of classes.

For measuring feature overlap, we use the maximumFisher’s discriminant ratio of the features. For a singlefeature, this measures the spread of the means for eachclass (µ1 and µ2) relative to their variances (σ2

1 and σ22):

(µ1−µ2)2

σ21+σ

22

. Since the proposed method uses counterfac-tual variables in which we have removed the effects ofsome parents, this removes sources of variance in thevariable. Thus, we expect the variances of each class to

reduce resulting in increased feature separability and acorresponding increased Fisher’s discriminant ratio.

One measure of separability of classes is based off of atest (Friedman and Rafsky, 1979) for determining if twosamples are from the same distribution. First, computea minimum spanning tree (MST) that connects all thedata points regardless of class. Then, the proportion ofnodes which are connected to nodes of a different classis an approximate measure of the proportion of exampleson the class boundary. Higher values of this proportiongenerally indicate a more complex boundary, and thus amore difficult classification problem.

However, this metric is only sensitive to which class neigh-bors are closer, and not the relative magnitudes of intra-class and interclass distances. Another measure of classseparability is the ratio between the average intraclassnearest neighbor distance and the average interclass near-est neighbor distance. This measures the relative magni-tudes of the dispersion within classes and the gap betweenclasses. We expect intraclass distances to decrease be-cause the data units are transformed to have the samevalue of the intervened parents, reducing sources of vari-ance (e.g., less variance in counterfactual untreated BPthan in factual BP).

The non-T parents of a variable add variance to the pre-diction problem through their effects on children of T . Byremoving their effects from children of T , the proposedmethod can directly increase the signal-to-noise ratio ofthe classification problem. With respect to the geometriccomplexity of the class boundary, this manifests itselfthrough reductions in the variance within a class, as wedemonstrate in a simulated experiment.

5 EXPERIMENTS

We demonstrate that without requiring samples from thetarget distribution during training, Counterfactual Normal-ization results in discriminative models with more stableperformance across datasets. In all experiments we trainmodels using only source data and evaluate on test datafrom both the the source and target domains.

5.1 SIMULATED EXPERIMENTS

5.1.1 Linear Gaussian Example

We consider a regression version of the simple domain-dependent confounding example in Figure 1b in whichD is unobserved. We simulate data from linear GaussianSEMs in which every variable is a linear combination ofits parents plus Gaussian noise. Thus, every edge in thegraph has a corresponding weight which is the coefficient

�2 0 2 4 6Test w2

0.005

0.010

0.015

0.020

0.025

0.030M

SECFN: E[T |Z]

Ideal: E[T |C,Y,D]

Naive: E[T |C,Y ]

Train w2

Figure 3: Test MSE as w2 varies in the linear experiment.Performance of CFN and Ideal remains stable while thenaive model’s performance can arbitrarily worsen.

Table 1: Simulated Experiment ResultsMethod Source AUROC Target AUROCBaseline 0.95 0.80CFN 0.96 0.97CFN (vuln) 0.97 0.92

of the parent in the SEM (for full specification consult thesupplement). In particular, we let C = w2D + εC , εC ∼N (0, σ2

C). We manifest domain-dependent confoundingby varying w2 in different test domains (from w2 = 2 inthe train domain), thus changing p(C|D).

First, note that in the ideal case if we could observe Dthen the unstable edge D → C is not in any active pathsto T when C, Y and D are conditioned upon. This meansa least squares regression modeling E[T |C, Y,D] willhave stable predictive performance regardless of changesto w2 and p(C|D). This is visible in Figure 3 in whichthe mean squared error (MSE) of the ideal model (redpoints) stays constant despite changes in w2.

We could naively ignore changes to p(C|D) and model byconditioning on vulnerable variables E[T |C, Y ], but thenaive model’s performance will vary in test domains as aresult of using the unstable path. In fact, the MSE of thenaive model (blue points in Figure 3) appears to increasequadratically as w2 changes from its training value.

Alternatively, we can use Counterfactual Normalization(CFN). C and Y are vulnerable, but we can conditionon Y (C = ∅). First, we fit the structural equation for Yfrom training data: Y = w3T + w4C.2 We then estimateY (∅) = Y − w4C. Finally, we model E[T |Y (∅)] whichconditions on a stable set. The MSE of CFN is stable(green points in Figure 3), but it can be outperformed bythe naive and ideal models because CFN isolates pathsthat include D.

5.1.2 Cross Hospital Transfer

Ensuring Stable Performance

2∧ denotes an estimated value.

Table 2: Simulated Classification Complexity MetricsMethod Fisher’s Distance MSTBaseline 0.66 0.10 0.56CFN 3.13 0.02 0.22

We consider a simulated version of the diagnosis problemin Figure 2(a), but remove X from the graph. We letA represent the time since treatment and simulate theexponentially decaying effects of the treatment as f(A) =2 exp(−0.08A) where the treatment policy depends onC. C and its descendants (A and Y ) are vulnerable.

We simulate patients from two hospitals (full specifica-tion in the supplement). In the source hospital there isa positive correlation between C and T , while in thetarget hospital p(C|D) changes yielding a negative cor-relation. At the source hospital smaller A are associatedwith T = 1 while at the target hospital A is uncorrelatedwith T . The structural equation for Y remains stable:Y = −0.5T − 0.3C + f(A) + εY , εY ∼ N (0, 0.22).We train using data from the source hospital and evaluateperformance at both the source and target hospitals.

Counterfactual Normalization requires us to estimatethe latent variable Y (A = ∅, C = ∅). We first fitthe structural equation for Y using maximum likeli-hood estimation, optimized using BFGS (Chong andZak, 2013). Then, we compute the counterfactual:Yi(A = ∅, C = ∅) = Yi − βCi − f(si) for every in-dividual i at both hospitals, which can be done with-out observing T . We compare a counterfactual model(CFN) p(T |Y (∅, ∅)) with a baseline vulnerable modelp(T |Y,A,C) and counterfactual model that uses vulnera-ble variables p(T |Y (∅, ∅), Y, A,C) using logistic regres-sion and measure predictive accuracy with the area underthe Receiver Operating Characteristic curve (AUROC).

The results of evaluation on the patients from the sourceand target are shown in Table 1. The accuracy of mod-els that use vulnerable variables does not transfer acrosshospitals, with the baseline suffering large changes in per-formance. Instead, CFN transfers well while performingcompetitively at the source hospital, despite not using un-stable paths which are informative in the training domain.

Normalizing BP (Y ) for treatment (A) and chronic con-dition (C) greatly increases the separability by class inthe training data as measured through the classificationcomplexity metrics in Table 2. The feature with the maxi-mum Fisher’s Discriminant Ratio in the baseline modelis C, but this is much smaller than the ratio for the latentfeature in CFN. The large decrease in the MST metricindicates fewer examples lies on the class boundary inthe normalized problem, and the decrease in intraclass-interclass distance is due to a combination of increasedseparability and reduced intraclass variance of the latent

Figure 4: Performance as the accuracy of counterfactualestimates decreases.The error bars denote the standarderror of 50 runs.

variables. This is visible in the class conditional densitiesof factual and counterfactual Y (see supplement).

Accuracy of Counterfactual Estimates

In this experiment, we examine how the accuracy ofcounterfactual estimates affects model stability and per-formance. We expect models that do not use vulnera-ble variables to have more stable performance, but theymay be less accurate in the source domain than mod-els which use vulnerable variables. We bias the truecounterfactual values by adding normally distributednoise of increasing scale. Then, we train the counterfac-tual logistic regressions (with p(T |Y (∅, ∅)) and withoutp(T |Y (∅, ∅), C,A, Y ) vulnerable variables) to predict Tand evaluate the AUROC at the source and target hospitals.We vary the standard deviation of the perturbations from0.05 to 1 in increments of 0.05, repeating the process 50times for each perturbation.

The results, shown in Figure 4, confirm what we expect:removing vulnerable variables leads to more stable perfor-mance, but performance in the source domain is alwayslower than when including vulnerable variables. Fur-ther, when the counterfactual estimates are accurate (lowMSE), removing vulnerable variables yields better perfor-mance in the target domain. However, when the MSE ishigh, the noise removes both the information captured bythe adjustment and the information contained in Y itself,causing the model to perform worse in the target domainthan a model using vulnerable variables.

5.2 REAL DATA: SEPSIS CLASSIFICATION

5.2.1 Problem and Data Description

We apply the proposed method to the task of detectingsepsis, a deadly response to infection that leads to organfailure. Early detection and intervention has been shownto result in improved mortality outcomes (Kumar et al.,

Figure 5: Real data experiment DAG of causal mecha-nisms. The outcome Y is INR and the target T is sepsis.S represents selection bias.

2006) which has resulted in recent applications of ma-chine learning to build predictive models for sepsis (e.g.,Henry et al. (2015); Soleimani et al. (2017); Futoma et al.(2017a,b)).

To illustrate, we consider a simplified3 cross-sectionalversion of the sepsis detection task using electronic healthrecord (EHR) data from our institution’s hospital. Work-ing with a domain expert, we determined the primaryfactors in the causal mechanism DAG (Figure 5) for theeffects of sepsis on a single physiologic signal Y : the in-ternational normalized ratio (INR), a measure of the clot-ting tendency of blood. The target variable T is whetheror not the patient has sepsis due to hematologic dysfunc-tion. We include seven conditions (such as chronic liverdisease and sickle cell disease) C affecting INR that arerisk factors for sepsis (Goyette et al., 2004; Booth et al.,2010). We consider five types of relevant treatments A:anticoagulants, aspirin, nonsteroidal anti-inflammatorydrugs (NSAIDs), plasma transfusions, and platelet trans-fusions, where Aij = 1 means patient i has receivedtreatment j in the last 24 hours. Finally, we include ademographic risk factor, age X . For each patient, we takethe last recorded measurements while only consideringdata up until the time sepsis is recorded in the EHR forpatients with T = 1.

27,633 patients had at least one INR measurement, 388of whom had sepsis due to hematologic dysfunction. Weintroduced selection bias S as follows. First, we tookone third of the data as a sample from the original targetpopulation for evaluation. Second, we subsample theremaining data by rejecting patients with any treatmentand without sepsis with probability 0.9. Third, we splitthe subsampled data into a random two thirds/one thirdtrain/test splits for training on biased data and evaluatingon both the biased and unbiased data to measure stabilityof prediction performance. We repeated the three steps100 times. We normalize INR in all experiments.

3Sepsis involves many physiologic markers and correspond-ing treatments and chronic conditions. We select a small numberof variables to demonstrate the key technical concepts.

5.2.2 Experimental Setup

We apply the proposed method by fitting an additive struc-tural equation for Y using the Bayesian calibration formof Kennedy and O’Hagan (2001):

Yi = β0 + β1Ti + βT2 Ai + β

T3 Ci + β4Xi

+ δ(Ti,Ai,Ci, Xi) + ε

δ(·) ∼ GP(0, γ2Krbf )

ε ∼ N (0, σ2)

where δ(·) is a Gaussian process (GP) prior (with RBFkernel) on the discrepancy function since our linear re-gression model is likely misspecified.

Due to selection bias and few sepsis examples, for bettercalibration we place informative priors on β1, β2, and β3usingN (1, 0.1) for features that increase INR (e.g., T andanticoagulants) andN (−1, 0.1) for features that decreaseINR (e.g., sickle cell disease and plasma transfusions).For full specification of the other priors consult the sup-plement. We compute point estimates for the parametersusing MAP estimation and the FITC sparse GP (Snel-son and Ghahramani, 2006) implementation in PyMC3(Salvatier et al., 2016).

While the only vulnerable variables are A and Y , weadditionally remove the effects of C and X:

Yi(∅,∅, ∅) = Yi − βT2 Ai − βT3 Ci − β4Xi (1)

We consider three logistic regression models trained onthe biased data for predicting T : a baseline using vul-nerable variables p(T |A,C, Y,X), a counterfactuallynormalized model p(T |C, Y (∅,∅, ∅), X), and a coun-terfactually normalized model with vulnerable variablesp(T |C, Y (∅,∅, ∅), Y,X). We evaluate prediction accu-racy on biased and unbiased data using AUROC and thearea under the precision-recall curve (AUPRC).

5.2.3 Results

The selection bias causes a small shift in the marginaldistribution of T between populations, such that 2% ofthe selection biased population has sepsis while 1.4% ofthe unbiased population has sepsis. Since most of theexamples are negative, the AUPRC is a more interestingmeasurement because it is sensitive to false positives.

The resulting AUCs when predicting on selection biaseddata are shown in Figure 6. As expected, the counter-factually normalized model (CFN) performs worse thanmodels using vulnerable variables because it does nottake advantage of the unstable path created by selectionbias. On unbiased data (Figure 7), however, CFN not onlyoutperforms both vulnerable models, but its performance

CFN (vuln) CFN Baseline (vuln) CFN(vuln) CFN Baseline (vuln)0.0

0.2

0.4

0.6

0.8

1.0

AU

RO

C

Prediction Performance on Selection Bias Data

0.0

0.1

0.2

0.3

0.4

AU

PRC

Figure 6: Results for models trained and tested on theselection biased data. In order the average AUROCs are0.98, 0.96, and 0.98 and the average AUPRCs are 0.45,0.38, and 0.45. Error bars denote 100 run 95% intervals.

CFN (vuln) CFN Baseline (vuln) CFN (vuln) CFN Baseline (vuln)0.0

0.2

0.4

0.6

0.8

1.0

AU

RO

C

Prediction Performance on Unbiased Data

0.00

0.05

0.10

0.15

0.20

0.25

0.30

AU

PRC

Figure 7: Results for models trained on biased data andtested on unbiased data. In order the average AUROCsare 0.95, 0.96, and 0.95 and the average AUPRCs are 0.24,0.30, and 0.24. Error bars denote 100 run 95% intervals.

is also more stable to the selection bias: the decrease inAUPRC from source to target is much smaller for CFN.

Interestingly, the the performance of the two vulnerablemodels is nearly identical. This implies that the CFNmodel with vulnerable variables does not learn to usecounterfactual features, perhaps because the unstable paththrough selection bias encodes a much stronger relation-ship. The AUPRC of the non-vulnerable CFN modelin selection biased and unbiased data is in between theAUPRC of the vulnerable CFN model in selection biaseddata (upper bound) and unbiased data (lower bound). Thisis encouraging, because Figure 4 suggests that if CFNperformance were worse in the unbiased data than thevulnerable model’s performance, then the counterfactualestimates may be inaccurate. Ultimately, we were ableto leverage Counterfactual Normalization to remove vul-nerable variables resulting in stabler performance, whileoutperforming the vulnerable models in unbiased data.

6 CONCLUSION

When environment-specific artifacts cause training andtest distributions to differ, naively training models underi.i.d. assumptions can result in unreliable models whichpredict using unstable relationships that do not general-ize. While some previous solutions use prior knowledgeof causal mechanisms to predict potential outcomes thatare invariant to differences in policy across environments,they require strong assumptions about no unobserved con-founders that may not hold in practice (e.g., Schulam andSaria (2017)). Our proposed solution, Counterfactual Nor-malization, generalizes these approaches to cases in whichthe unstable relationships (such as ones due to domain-specific policy) may depend on unobserved variables orselection bias. Specifically, we train discriminative mod-els using conditioning sets that only contain variables withstable relationships with the target prediction variable.Then, for vulnerable variables with unstable relationshipsto the target, we consider adding to the conditioning setcounterfactual versions of these variables which sever theunstable paths of statistical influence. Further, becauseof their causal interpretations, we believe these counter-factual variables are more intelligible for human expertsthan existing adjustment-based methods. For example,we think it is easier to reason about “the blood pressure ifthe patient had not been treated” than interaction featuresor kernel embeddings—we would like to test this in afuture user study. As demonstrated by our experiments,models trained using Counterfactual Normalization haveperformance that is more stable to changes across envi-ronments and is not coupled to artifacts in the trainingdomain.

Acknowledgements

The authors would like to thank Katie Henry for her helpin developing the sepsis classification DAG and PeterSchulam for suggesting experiment 5.1.1 and help clari-fying presentation of the method.

ReferencesAlaa, A. M. and van der Schaar, M. (2017). Bayesian

nonparametric causal inference: Information rates andlearning algorithms. arXiv preprint arXiv:1712.08914.

Bareinboim, E. and Pearl, J. (2012). Controlling selectionbias in causal inference. In AISTATS, pages 100–108.

Bareinboim, E. and Pearl, J. (2013). Meta-transportabilityof causal effects: A formal approach. In AISTATS,pages 134–143.

Bareinboim, E. and Tian, J. (2015). Recovering causaleffects from selection bias. In AAAI, pages 3475–3481.

Booth, C., Inusa, B., and Obaro, S. K. (2010). Infectionin sickle cell disease: a review. International Journalof Infectious Diseases, 14(1):e2–e12.

Campbell, D. T. and Stanley, J. C. (1963). Experimentaland quasi-experimental designs for research. Handbookof research on teaching.

Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M.,and Elhadad, N. (2015). Intelligible models for health-care: Predicting pneumonia risk and hospital 30-dayreadmission. In KDD, pages 1721–1730. ACM.

Chong, E. K. and Zak, S. H. (2013). An introduction tooptimization, volume 76. John Wiley & Sons.

Correa, J. D. and Bareinboim, E. (2017). Causal effectidentification by adjustment under confounding andselection biases. In AAAI, pages 3740–3746.

Correa, J. D., Tian, J., and Bareinboim, E. (2018). Gen-eralized adjustment under confounding and selectionbiases. In AAAI.

Dvijotham, K., Stanforth, R., Gowal, S., Mann, T., andKohli, P. (2018). A dual approach to scalable verifica-tion of deep networks. In UAI.

Dyagilev, K. and Saria, S. (2015). Learning (predictive)risk scores in the presence of censoring due to inter-ventions. Machine Learning, 102(3):323–348. FirstOnline 2015. Printed Version 2016.

Friedman, J. H. and Rafsky, L. C. (1979). Multivariategeneralizations of the wald-wolfowitz and smirnov two-sample tests. The Annals of Statistics, pages 697–717.

Futoma, J., Hariharan, S., and Heller, K. (2017a). Learn-ing to detect sepsis with a multitask gaussian processrnn classifier. In ICML.

Futoma, J., Hariharan, S., Sendak, M., Brajer, N.,Clement, M., Bedoya, A., O’Brien, C., and Heller, K.(2017b). An improved multi-output gaussian processrnn with real-time validation for early sepsis detection.arXiv preprint arXiv:1708.05894.

Gong, M., Zhang, K., Liu, T., Tao, D., Glymour, C.,and Scholkopf, B. (2016). Domain adaptation withconditional transferable components. In ICML, pages2839–2848.

Goyette, R. E., Key, N. S., and Ely, E. W. (2004). Hema-tologic changes in sepsis and their therapeutic impli-cations. In Seminars in respiratory and critical caremedicine, volume 25, pages 645–659. Thieme MedicalPublishers, Inc., NY, USA.

Gretton, A., Smola, A. J., Huang, J., Schmittfull, M.,Borgwardt, K. M., and Scholkopf, B. (2009). Covariateshift by kernel mean matching.

Henry, K. E., Hager, D. N., Pronovost, P. J., and Saria,S. (2015). A targeted real-time early warning score

(trewscore) for septic shock. Science translationalmedicine, 7(299):299ra122–299ra122.

Ho, T. K. and Basu, M. (2000). Measuring the complex-ity of classification problems. In Pattern Recognition,volume 2, pages 43–47. IEEE.

Ho, T. K. and Basu, M. (2002). Complexity measuresof supervised classification problems. IEEE transac-tions on pattern analysis and machine intelligence,24(3):289–300.

Kennedy, M. C. and O’Hagan, A. (2001). Bayesian cal-ibration of computer models. Journal of the RoyalStatistical Society: Series B, 63(3):425–464.

Koller, D. and Friedman, N. (2009). Probabilistic graphi-cal models: principles and techniques. MIT press.

Kumar, A., Roberts, D., Wood, K. E., Light, B., Parrillo,J. E., Sharma, S., Suppes, R., Feinstein, D., Zanotti,S., Taiberg, L., et al. (2006). Duration of hypotensionbefore initiation of effective antimicrobial therapy isthe critical determinant of survival in human septicshock. Critical care medicine, 34(6):1589–1596.

Neyman, J. (1923). On the application of probabilitytheory to agricultural experiments. essay on principles.Annals of Agricultural Sciences, 10:1–51.

Pearl, J. (2009). Causality. Cambridge university press.

Pearl, J. (2015). Causes of effects and effects of causes.Sociological Methods & Research, 44(1):149–164.

Pearl, J. and Bareinboim, E. (2011). Transportability ofcausal and statistical relations: a formal approach. InAAAI, pages 247–254. AAAI Press.

Quionero-Candela, J., Sugiyama, M., Schwaighofer, A.,and Lawrence, N. D. (2009). Dataset shift in machinelearning. MIT Press.

Raghunathan, A., Steinhardt, J., and Liang, P. (2018).Certified defenses against adversarial examples. InICLR.

Richardson, T. S. and Robins, J. M. (2013). Single worldintervention graphs (swigs): A unification of the coun-terfactual and graphical approaches to causality. Centerfor the Statistics and the Social Sciences, University ofWashington Series. Working Paper, 128(30):2013.

Rothenhausler, D., Buhlmann, P., Meinshausen, N., andPeters, J. (2018). Anchor regression: heterogeneousdata meets causality. arXiv preprint arXiv:1801.06229.

Rubin, D. B. (1974). Estimating causal effects of treat-ments in randomized and nonrandomized studies. Jour-nal of Educational Psychology, 66(5):688.

Salvatier, J., Wiecki, T. V., and Fonnesbeck, C. (2016).Probabilistic programming in python using pymc3.PeerJ Computer Science, 2:e55.

Schulam, P. and Saria, S. (2017). Reliable decision sup-port using counterfactual models. In NIPS, pages 1696–1706.

Shimodaira, H. (2000). Improving predictive inferenceunder covariate shift by weighting the log-likelihoodfunction. Journal of statistical planning and inference,90(2):227–244.

Shpitser, I. and Pearl, J. (2007). What counterfactuals canbe tested. In UAI, pages 352–359. AUAI Press.

Shpitser, I. and Pearl, J. (2008). Complete identificationmethods for the causal hierarchy. Journal of MachineLearning Research, 9(Sep):1941–1979.

Sinha, A., Namkoong, H., and Duchi, J. (2018). Certi-fying some distributional robustness with principledadversarial training. In ICLR.

Snelson, E. and Ghahramani, Z. (2006). Sparse gaussianprocesses using pseudo-inputs. In NIPS, pages 1257–1264.

Soleimani, H., Hensman, J., and Saria, S. (2017). Scal-able joint models for reliable uncertainty-aware eventprediction. IEEE transactions on pattern analysis andmachine intelligence.

Spirtes, P., Meek, C., and Richardson, T. (1995). Causalinference in the presence of latent variables and selec-tion bias. In UAI, pages 499–506.

Storkey, A. (2009). When training and test sets are differ-ent: characterizing learning transfer. Dataset shift inmachine learning, pages 3–28.

Swaminathan, A. and Joachims, T. (2015). Counterfac-tual risk minimization: Learning from logged banditfeedback. In ICML, pages 814–823.

Zhang, K., Scholkopf, B., Muandet, K., and Wang, Z.(2013). Domain adaptation under target and condi-tional shift. In International Conference on MachineLearning, pages 819–827.

A Counterfactual Normalization Proofs

A.1 Proof of Theorem 1

We must show that on iteration k, removing a variablefrom Z does not create an active unstable path to a mem-ber of Z of length ≤ k.

Proof. Suppose, by contradiction, that on iteration k re-moving a variable v ∈ Z with an active unstable path oflength k to T results in an active unstable path of length≤ k with respect to another variable x ∈ Z. Note: remov-ing a variable from a conditioning set cannot create newcollider paths. Let . . . denote that direction of edge doesnot matter. We will consider all cases of how v can relateto an unstable path to x from T . In the first two cases, xcomes before v in the unstable active path to v.

Case 1: T . . . x . . . unstable . . . v. x does not have anunstable path to T . If it did, the path would be of the formT . . . unstable . . . x . . . unstable . . . v. Thus, x wouldhave been removed from Z in a previous iteration be-cause the unstable path is of length ≤ k and its activestatus does not depend on v.

Case 2: T . . . unstable . . . x . . . v. This is an unstablepath to x of length ≤ k. x cannot be in Z since it wouldhave been removed in a previous iteration as the activestatus of this path does not depend on v.

Case 3: T . . . unstable← v → . . . x. Creates new activepath of length > k.

Case 4: T . . . unstable→ v → . . . x. Creates new activepath of length > k.

Case 5: T . . . unstable← v ← . . . x. Creates new activepath of length > k.

Case 6: T . . . unstable→ v ← . . . x. We remove v fromthe conditioning set (so it is now considered unobserved).Thus, this collider path is not active. If a descendent of vis conditioned on, then this is an unstable active path oflength > k.

In all cases, either x would have been removed from Zbefore iteration k or the new unstable active path would beof length > k. This is a contradiction since we assumedx ∈ Z and that the procedure would result in a new activeunstable path of length ≤ k.

A.2 Proof of Theorem 2

We must show that Algorithm 3 will not activate any un-stable paths with respect to the initial stable set Z. Whileconsidering each vulnerable variable v ∈ V, the resultingset Z′ must remain stable.

Proof. We assume the initial set Z is stable. The onlyway to activate a path (stable or unstable) by adding toa conditioning set is if the new variable being added is acollider or descendant of a collider.

Algorithm 3 only adds to Z in branches 2 and 3 of theif-else. We consider each branch in turn.

In branch 2, the vulnerable variable v ∈ V has no activeunstable paths to T . Thus, by adding v to Z, no unstablepath from v to T is active. Next we consider the possi-bility of v being a collider or descendant of a collider.In these cases, if v activates a path all branches of theactivated path · · · → collider ← . . . will be reachableby v. Since v has no unstable active paths to T , none ofthe branches of the collider can be unstable paths to T .Thus, the collider path is not unstable.

In branch 3 of the if-else, the active unstable paths of thevulnerable variable v ∈ V all go through some subset ofthe observed parents of v. Denote this subset of parentsof v as P ⊆ pa(v). We intervene on P and node-split toadd the counterfactual v(P = ∅) to the modified graph.The modified graph contains the path v(P = ∅)→ v ←P. Conditioning on v(P = ∅) does not allow for pathsthrough the collider v. We know the paths through theparents of v(P = ∅) are stable because we intervened onall parents on unstable paths from v. Further, there areno active unstable paths through children of v otherwisev would not be considered in branch 3. As in the caseof branch 2, if v(P = ∅) is a collider or descendant of acollider, then since all parts of any activated collider pathare reachable from the counterfactual variable through itsparents, they must be stable.

Thus, when Algorithm 3 adds a variable or counterfactualvariable to the stable set Z, no unstable paths are activatedand the set remains stable.

B Linear Gaussian Experiment Details

B.1 Simulation Details

We generate the data from the following linear GaussianSEMs:

D = εD

T = w1D + εT

C = w2C + εC

Y = w3T + w4C + εY

εD, εT , εC , εY ∼ N (0, 0.12)

In the training domain, we set w2 = 2 and w1, w3, w4 ∼N (0, 1). We simulated 100 test domains by varying w2

from −3 to 7 in equally spaced increments. In all do-mains we generated 30000 samples. We fit all models

(structural equation of Y , naive model, ideal model, andcounterfactually normalized model) on the training datausing least squares then applied them to all test domains.

B.2 Counterfactual Normalization

C

D

Y

T

Y (∅)

Figure 8: Modified graph after node-splitting.

D is unobserved and p(C|D) varies between domains.Thus, C is vulnerable becomes conditioning on C resultsin the unstable active path C ← D → T . Y , as a descen-dant ofC, is also vulnerable because without conditioningonC, conditioning on Y results in the unstable active pathY ← C ← D → T . The only shared child of vulnerablevariables and T is Y . This means we need to performnode-splitting on Y to generate an intermediate counter-factual version, Y (C = ∅) for which we have removedthe effects of the vulnerable parent C. The graph afternode-splitting is shown in Figure 8.

The counterfactual variable Y (C = ∅) inherits the parentsof Y from the original graph that we did not interveneupon (i.e., set to null). In this case, the parents it inheritsare T and the unpictured εY . The SEMs in the modifiedgraph are:

D = εD

T = w1D + εT

C = w2C + εC

Y (C = ∅) = w3T + εY

Y = Y (C = ∅) + w4C

εD, εT , εC , εY ∼ N (0, 0.12)

Importantly, note that the counterfactual Y (C = ∅) is nowa random quantity, while Y is a deterministic function ofY (C = ∅) and C. The modified SEMs are observation-ally equivalent to the original SEMs (marginalizing overthe latent counterfactual yields the same joint as in theoriginal system).

We can recover Y (C = ∅) by observing Y and C:Y (C = ∅) = Y − w4C. This makes clear the roleof the intermediate counterfactual variable: it isolates theeffect of the target on the outcome from the effects ofvulnerable variables (or any other parents we set to null)on the outcome.

C Cross Hospital Transfer ExperimentDetails

C.1 Simulation Details

We generate data at the source hospital as follows:

D ∼ Bernoulli(0.5)T |D = 1 ∼ Bernoulli(0.7)T |D = 0 ∼ Bernoulli(0.1)C|D = 1 ∼ Bernoulli(0.9)C|D = 0 ∼ Bernoulli(0.1)A|C = 1 ∼ 24 ∗Beta(0.5, 2.1)A|C = 0 ∼ 24 ∗Beta(0.7, 0.2)Y ∼ N (−0.5T +−0.3C + f(A), 0.22)

f(A) = 2 exp(−0.08A)

At the target hospital, we change p(C|D) and p(A|C):

D ∼ Bernoulli(0.5)T |D = 1 ∼ Bernoulli(0.7)T |D = 0 ∼ Bernoulli(0.1)C|D = 1 ∼ Bernoulli(0.1)C|D = 0 ∼ Bernoulli(0.9)A|C = 1 ∼ 24 ∗Beta(1.7, 1.1)A|C = 0 ∼ 24 ∗Beta(1.7, 1.1)Y ∼ N (−0.5T +−0.3C + f(A), 0.22)

f(A) = 2 exp(−0.08A)

We generate 2000 patients from the source hospital, using1600 for training and holding out 400 to evaluate perfor-mance on the source hospital. We evaluate cross hospitaltransfer on 1000 patients generated from the second hos-pital.

C.2 Class Conditional Densities

D Real Data Experiment Details

Our posited structural equation for INR (Y ) is a linearregression of the parents of Y in Figure 5. The sevenconditions (C) we include are liver disease, sickle celldisease, chronic kidney disease, any immunodeficiency,any cancer, diabetes, and stroke. In the statistical un-certainty quantification community, one technique forparameter calibration when the computer model is mis-specified is to jointly estimate model parameters with anexplicit discrepancy function that captures model inade-quacy (Kennedy and O’Hagan, 2001). The discrepancyfunction has a Gaussian process prior. The parameters

Figure 9: The distribution of factual (solid line) and es-timated counterfactual (dashed line) blood pressures atthe source hospital in the simulated experiment. It is eas-ier to discriminate T from counterfactual BP than fromobserved BP due to decreased overlap in the distributions.

to estimate are the linear regression parameters β, theobservation noise scale σ, the RBF kernel output scale γ,and the kernel lengthscales `.

We placed the following priors on parameters:

γ ∼ HalfN (1)

σ ∼ HalfN (1)

` ∼ Gamma(4, 4)β0 ∼ N (0, 1)

β1,β(1,2,3,5)2 ,β

(1)3 ∼ N (1, 0.1)

β(4)2 ,β

(2)3 ∼ N (−1, 0.1)

β4 ∼ N (0, 0.1)

We used the PyMC3 FITC sparse GP approximation im-plementation with 20 inducing points initialized by k-means.