85
Presentations in this series 1. Overview and Randomization 2. Self-matching 3. Proxies 4. Intermediates 5. Instruments 6. Equipoise Avoiding Bias Due to Unmeasured Covariates Alec Walker
Presentations in this series1. Overview
and Randomization2. Self-matching3. Proxies4. Intermediates5. Instruments6. Equipoise
Avoiding Bias Due toUnmeasured Covariates
Alec Walker
T D
XSelf-matchingProxies Proxies
Randomization
IntermediatesIntermediates
Instruments
T D
XSelf-matchingProxies Proxies
Randomization
IntermediatesIntermediates
Instruments
T D
XSelf-matchingProxies Proxies
Randomization
IntermediatesIntermediates
Instruments
When strong proxies for the possible confounding determinants of exposure indicate no effect on exposure, there is correspondingly strong evidence for an absence of confounding.
T D
X
Proxies
Other, possibly unmeasured, non-confounding determinants of exposure are the sole determinants of treatment variation.
UT
Where we’re goingWhen different doctors give different treatments to similar patients, there must be a variety of opinions about therapy in the clinical community.
The presence of differing therapeutic opinion has been termed “clinical equipoise” and is a permissive condition for conducting an ethical clinical trial.
Clinical equipoise, identified in observational data as “empirical equipoise,” is also a permissive state for comparative effectiveness research.
Σ
ΣProvider
ΣProvider
ΣProvider
Rx
ΣProvider?
ΣProvider?
Patient
?
?complaint
Σ?complaint
Σ?complaint
training norms colleagues experience
Σ?complaint
symptoms
training norms colleagues experience
Σsigns?
complaintsymptoms
training norms colleagues experience
Σsigns?
complaintsymptoms
history training norms colleagues experience
Σsigns?
complaintsymptoms
historyillnesses
training norms colleagues experience
Σsigns?
medications
complaintsymptoms
historyillnesses
training norms colleagues experience
Σsigns?
medications
complaintsymptoms
test results
historyillnesses
training norms colleagues experience
Σsigns?
medications
complaintsymptoms
test results
historyillnesses
training norms colleagues experience
Σsigns?
medications
complaintsymptoms
test results
historyillnesses
training norms colleagues experience
Rx A
Σsigns?
medications
complaintsymptoms
test results
historyillnesses
training norms colleagues experience
Rx A
Σ
Σsigns?
medications
complaintsymptoms
test results
historyillnesses
training norms colleagues experience
Rx A
Σtraining norms colleagues experience
Σsigns?
medications
complaintsymptoms
test results
historyillnesses
training norms colleagues experience
Rx A
Σtraining norms colleagues experience
Rx B
Σsigns?
medications
complaintsymptoms
test results
historyillnesses
training norms colleagues experience
Rx A
Σtraining norms colleagues experience
Rx BΣtraining norms colleagues experience
Rx C
Σsigns?
medications
complaintsymptoms
test results
historyillnesses
training norms colleagues experience
Rx A
Σtraining norms colleagues experience
Rx BΣtraining norms colleagues experience
Rx C
Σtraining norms colleagues experience
Rx D
Σsigns?
medications
complaintsymptoms
test results
historyillnesses
training norms colleagues experience
Rx A
Σtraining norms colleagues experience
Rx BΣtraining norms colleagues experience
Rx B
Σtraining norms colleagues experience
Rx A
Σsigns?
medications
complaintsymptoms
test results
historyillnesses
training norms colleagues experience
Rx A
Σtraining norms colleagues experience
Rx BΣtraining norms colleagues experience
Rx B
Σtraining norms colleagues experience
Rx A
When the same patient has an equal chance of getting Rx A or Rx B, depending only on which doctor he happens to visit, the treating community is in EQUIPOISE.
Personal ambivalence
“If a physician knows that these treatments are not equivalent, ethics requires that the superior treatment be recommended. “Following Fried, I call this state of uncertainty about the relative merits of A and B ‘equipoise.’ ”
Reference is to Charles Fried. Medical Experimentation: Personal Integrity and Social Policy. Amsterdam: North-Holland Publishing, 1974
Benjamin Friedman. Equipoise and the ethics of clinical research. N Engl J Med 1987;317:141-145.
Shortcomings of ambivalence in RCTs
Threats to personal equipoise• Studies that motivated the trial• Early results of the trial itself• Findings on secondary endpoints: QoL• Change in experimenter’s understanding of– Relevant data external to the RCT– Views of other competent observers
Benjamin Friedman. Equipoise and the ethics of clinical research. N Engl J Med 1987;317:141-145.
Failed rescue attempts for personal equipoise
Conceal results from investigators through a Data and Safety Monitoring Committee
Give the patient responsibility for valuing the relative merits of the treatments
Informed ConsentFrankly admit the social need for reliable health research.
Medical Conscription
Benjamin Friedman. Equipoise and the ethics of clinical research. N Engl J Med 1987;317:141-145.
Retained for other reasons
Conceal results from investigators through a Data and Safety Monitoring Committee
Give the patient responsibility for valuing the relative merits of the treatments
Informed Consent
Benjamin Friedman. Equipoise and the ethics of clinical research. N Engl J Med 1987;317:141-145.
Autonomy
Conceal results from investigators through a Data and Safety Monitoring Committee
Give the patient responsibility for valuing the relative merits of the treatments
Informed ConsentFrankly admit the social need for reliable health research.
Medical ConscriptionBenjamin Friedman. Equipoise and the ethics of clinical research. N Engl J Med 1987;317:141-145.
When we do RCTs
Benjamin Friedman. Equipoise and the ethics of clinical research. N Engl J Med 1987;317:141-145.
“The standard treatment is A, but some evidence suggests that B will be superior.”“Or there is a split in the clinical community, with some clinicians favoring A and others favoring B …” “… an honest, professional disagreement among expert clinicians …”
When we do RCTs“The standard treatment is A, but some evidence suggests that B will be superior.”“Or there is a split in the clinical community, with some clinicians favoring A and others favoring B …” “… an honest, professional disagreement among expert clinicians …”
Benjamin Friedman. Equipoise and the ethics of clinical research. N Engl J Med 1987;317:141-145.
“At this point a state of ‘clinical equipoise’ exists … A state of clinical equipoise is consistent with a decided treatment preference on the part of the investigators.”
Clinical equipoise and community
Benjamin Friedman. Equipoise and the ethics of clinical research. N Engl J Med 1987;317:141-145.
“… there is a split in the clinical community ” “There is no consensus within the expert clinical community about the comparative merits…”“… clinical equipoise persists as long as [available] results are too weak to influence the judgement of the community of clinicians.”“As Fried has emphasized, competent (hence, ethical) medicine is social rather than individual in nature.”
Reference is to Charles Fried. Medical Experimentation: Personal Integrity and Social Policy. Amsterdam: North-Holland Publishing, 1974
What prescribers think, what they doIn the absence of knowledge of prescribers’ beliefs about alternative treatments, we might assume that each prescriber’s behavior reflects belief about best treatment for the individual patient, given the constraints of their shared environment.
Σsigns?
medications
complaintsymptoms
test results
historyillnesses
training norms colleagues experience
Rx A
Σsigns?
medications
complaintsymptoms
test results
historyillnesses
training norms colleagues experience
Rx A
Σtraining norms colleagues experience
Rx BΣtraining norms colleagues experience
Rx B
Σtraining norms colleagues experience
Rx A
When the same patient has an equal chance of getting Rx A or Rx B, depending only on which doctor he happens to visit, the treating community is in EMPIRICAL EQUIPOISE.
A community of prescribers
Σsigns?
medications
complaintsymptoms
test results
historyillnesses
training norms colleagues experience
Rx A
Σtraining norms colleagues experience
Rx BΣtraining norms colleagues experience
Rx B
Σtraining norms colleagues experience
Rx A
We will take empirical equipoise in the population of prescribers as evidence of clinical equipoise in the prescriber community, in just the sense meant by Friedman.
A community of prescribers
Modeling the counterfactual
We can’t observe what different doctors would do with the same patient.
Σtraining norms colleagues experience
Rx A
Σtraining norms colleagues experience
Σtraining norms colleagues experience
Σtraining norms colleagues experience
Rx ARx B Rx B
In general, the patients who go to different doctors are not identical.
Modeling the counterfactual
Σtraining norms colleagues experience
Rx A
Σtraining norms colleagues experience
Σtraining norms colleagues experience
Σtraining norms colleagues experience
Rx ARx B Rx B
But what if switching patients between doctors made no difference to the treatment assignment?
Modeling the counterfactual
Σtraining norms colleagues experience
Rx A
Σtraining norms colleagues experience
Σtraining norms colleagues experience
Σtraining norms colleagues experience
Rx ARx B Rx B
We’d say that the treatment choices are probably most reflective of prescriber beliefs.
Modeling the counterfactual
The patients can’t be switched between doctors in an observational study.
Modeling the counterfactual
The patients can’t be switched between doctors in an observational study.
Instead of asking whether the patient per se is determinative of doctor’s treatment choice, we can ask whether any observed patient characteristics tend to predict treatment.
Modeling the counterfactual
Pr(A|A or B)=f(test results, medications, concurrent illnesses, history, signs, symptoms)
Instead of asking whether the patient per se is determinative of doctor’s treatment choice, we can ask whether any observed patient characteristics tend to predict treatment.
Modeling the counterfactual
Pr(A|A or B)=f(test results, medications, concurrent illnesses, history, signs, symptoms)
If we estimate from the data whether any observed patient characteristics tend to predict treatment, we’ve calculated a propensity score.
Propensity
Pr(A|A or B)=f(test results, medications, concurrent illnesses, history, signs, symptoms)
If doctor preference is the major determinant of treatment choice, most fitted values from the propensity model estimating Pr(A) will fall near the overall prevalence of A in “A or B” population.
Propensity
To make different A-B comparisons easier to compare, we’ve subtracted out the grand mean and renamed the result “Preference.”
Preference
F Preference S Propensity P Prevalence of A
Alternative treatments in Community Acquired Pneumonia
Rela
tive
freq
uenc
y
Patients sorted by the preference score corresponding to their clinical and demographic characteristics
Alternative treatments in Community Acquired Pneumonia
Rela
tive
freq
uenc
y
Patients sorted by the preference score corresponding to their clinical and demographic characteristics
A B C D
Alternative treatments in Community Acquired Pneumonia
Rela
tive
freq
uenc
y
Patients sorted by the preference score corresponding to their clinical and demographic characteristics
E F G H
Alternative treatments in Community Acquired Pneumonia
Rela
tive
freq
uenc
y
Patients sorted by the preference score corresponding to their clinical and demographic characteristics
Empirical Equipoise
Most people have covariate patterns that are similarly represented in the two treatment groups.
Alternative treatments in Community Acquired Pneumonia
Rela
tive
freq
uenc
y
Patients sorted by the preference score corresponding to their clinical and demographic characteristics
Empirical Equipoise
Most people have covariate patterns that are similarly represented in the two treatment groups.
A B C D
Alternative treatments in Community Acquired Pneumonia
Rela
tive
freq
uenc
y
Patients sorted by the preference score corresponding to their clinical and demographic characteristics
Patient-driven preferences
Most people have covariate patterns that are differently represented in the two treatment groups.
Alternative treatments in Community Acquired Pneumonia
Rela
tive
freq
uenc
y
Patients sorted by the preference score corresponding to their clinical and demographic characteristics
Patient-driven preferences
Most people have covariate patterns that are differently represented in the two treatment groups.
E F G H
Empirical Equipoise
Levofloxacin Azithromycin0%5%
10%15%20%25%30%35%40%45%
Trea
tmen
t Fa
ilure
Levofloxacin Azithromycin0%5%
10%15%20%25%30%35%40%45%
Trea
tmen
t Fa
ilure
Levofloxacin Azithromycin0%5%
10%15%20%25%30%35%40%45%
Trea
tmen
t Fa
ilure
Levofloxacin Azithromycin0%5%
10%15%20%25%30%35%40%45%
Trea
tmen
t Fa
ilure
Levofloxacin appears to have fewer failures
Empirical equipoise means that the prescribers appear to have diverse opinions. The patients themselves formed very similar groups, so differences in treatments are most likely ascribable to the doctors.
If one treatment has 28% fewer failures, the difference is large enough to be important.
Empirical equipoise means that th prescribers appear to have diverse opinions.
Why should balance on observed variables imply balance on the unobserved?
Why should balance on observed variables imply balance on the unobserved?
One answer is narrative.
The observed variables give a clue as to how prescribers respond to variations in patient characteristics. If we know that they respond little to medical facts that we can account for, we guess that they are insensitive to facts that we have not measured.
Why should balance on observed variables imply balance on the unobserved?
Another answer is in the form of a syllogism:• Different patients get different treatments.• Patient characteristics in empirical equipoise
are not determinative of drug choice.• Possibly unmeasured factors external to the
patient and to his/her prognosis are determinative of drug choice
These unmeasured factors are Instruments, whose presence is the only justification for observational CER)
More plausible than “no unmeasured confounding”
Treatment variation due to uncontrolled Instruments
is much larger thanTreatment variation due to uncontrolled
Confounders
Where we’ve beenWhen different doctors give different treatments to similar patients, there must be a variety of opinions about therapy in the clinical community.
The presence of differing therapeutic opinion has been termed “clinical equipoise” and is a permissive condition for conducting an ethical clinical trial.
Clinical equipoise, identified in observational data as “empirical equipoise,” is also a permissive state for comparative effectiveness research.
?medications
complaint
signssymptoms
test results
historyillnesses
A note of caution
?medications
complaint
signssymptoms
test results
historyillnesses (x)
(y)
(z)
A note of caution
?medications
complaint
signssymptoms
test results
historyillnesses (x)
(y)
(z)
A note of caution
?medications
complaint
signssymptoms
test results
historyillnesses (x)
(y)
(z)
?medications
complaint
signssymptoms
test results
historyillnesses (a)
(b)
(c)
A note of caution
?medications
complaint
signssymptoms
test results
historyillnesses (x)
(y)
(z)
?medications
complaint
signssymptoms
test results
historyillnesses (a)
(b)
(c)
A note of caution
?medications
complaint
signssymptoms
test results
historyillnesses (x)
(y)
(z)
?medications
complaint
signssymptoms
test results
historyillnesses (a)
(b)
(c)
Because some patient characteristics known to the doctor may not be recorded in the database, Empirical Equipoise might be only Apparent.
A note of caution
?medications
complaint
signssymptoms
test results
historyillnesses (x)
(y)
(z)
?medications
complaint
signssymptoms
test results
historyillnesses (a)
(b)
(c)
Because some patient characteristics known to the doctor may not be recorded in the database, Empirical Equipoise might be only Apparent.
!
A note of caution
?medications
complaint
signssymptoms
test results
historyillnesses (x)
(y)
(z)
?medications
complaint
signssymptoms
test results
historyillnesses (a)
(b)
(c)
We’ve made arguments as to why the vectors of unmeasured covariates[x,y,z] and [a,b,c] ought to have similar distributions.
!But it is surely a good idea to worry, to check whenever you can, and to be modest.
A note of caution
