Towards a Complete Solution for Cost/Effectiveness in Oncology: Handling Heterogeneity, Variability and Censoring

Gerhardt PohlEli Lilly and Company

Objective

We view a “complete” solution to the problem of calculating the Incremental Cost Effectiveness Ratio (ICER) in oncology as one simultaneously addressing the key issues of (1) Heterogeneity (2) Variability and (3) Censoring.The talk will discuss a stratified, bootstrap approach and draw links to the concept of “local propensity.”

What is an ICER??

• Incremental Cost Effectiveness Ratio

• Ratio of difference in mean cost divided by difference in mean effectiveness.

where T denotes the new treatment and C the control

Impact

• ICER is the primary tool used in cost-effectiveness comparisons by HTA (Health Technology Assessment) bodies around the world.

• ICER’s allow comparisons of treatments between therapies and across disease states to allow appropriate choices in national health expenditures.

ICER Graphically

QALY

$

ICE Plane

Cheaper and BetterCheaper but Less Effective

More Expense for More Effectiveness

More Expensive and Less Effective

(Quality Adjusted Life Years)

NICE Thresholds

£50,000/QALY (E

nd of Life

Status)

£30,000/QALYNot Approvable

Approvable

“End of Life Status” is granted only for treatments which are life-extending (>3months) for patients (<7,000) with short life expectancy (<24 months)

National Institute for Health and Clinical Excellence

Analysis GoalCreate a bootstrapped display of variability in the ICE plane where each iteration is based on risk-adjusted and censored estimates of the cost and survival.

Frick KD, et al. “Modeled cost-effectiveness of the experience corps Baltimore based on a pilot randomized trial.” Journal of Urban Health 2004; 81:106-117.

Addressing Heterogeneity

Propensity Score Pictorially

Age

Com

orbi

ditie

s

Propensit

y Sco

re St

rata

More likely to receive red treatment

More likely to receive blue treatment

A Downside of Propensity Scoring

• Patients with the identical propensity score may have very different covariate levels.

Age

Com

orbi

ditie

s

Young and Sick

Old But Healthy

Blocking

• Grid the factor space into blocks (unordered strata) of similar patients.

• This may be thought of as a many-to-many matching directly in the covariate space.

Age

Com

orbi

ditie

s

Stratum 1 Stratum 2 Stratum 3

Stratum 4 Stratum 5 Stratum 6

Stratum 7 Stratum 8 Stratum 9

General Approach

• Whatever the original dimension of the covariate space, this reduces the problem to cross-classification of treatments versus strata.

Stratum 1 Stratum 2 Stratum 3

Stratum 4 Stratum 5 Stratum 6

Stratum 7 Stratum 8 Stratum 9

Stratum 1 Stratum 2 … TotalTreatment nT1 nT2 NT

Control nC1 nC2 NC

Total N1 N2 N

Calculate Within-Stratum Treatment Differences

Stratum 1 Stratum 2 Etc. TotalTreatment nT1 nT2 NT

Control nC1 nC2 NC

Total N1 N2 N

Cost:

Effectiveness:

How to pool across strata?

Stratum Weighting

For overall mean difference, pool relative to size of strata:

Definition of Stratified ICER:

Pros and Cons

Blocking• Non-parametric• Provides better matches of

underlying covariates• Able to capture complex

interactions of covariates and likelihood of treatment

Propensity Score Matching• Reduces complexity of

covariate space down to one dimension

• Can maintain structure/ordering of covariate levels

• Can borrow information across blocks

• Potentially uses full richness of continuous data

Addressing Variability

Variability in the ICER Estimate• Many methods have been proposed to incorporate variability into ICER

based inference:

– Univariate Sensitivity Analyses (Tornado Diagrams)– Confidence Intervals (Fieller’s Theorem, 2-dimensional boxes, ellipses, wedges,

bootstrapping…)– Simulation– Cost-Effectiveness Acceptability Curves– Net Monetary Benefit– And various combinations thereof…

• Fundamental technical issue is that the ratio of 2 normal variables is not normal. (Nor very tractable! For example, what if confidence interval for the denominator includes zero.)

• Evidence in literature suggests that Bootstrapping provides robust and consistent inference. Bootstrapping is also relatively assumption free and easily, explained heuristically .

Bootstrapping the Stratified ICER

• Pre-specify strata (factors and cutoff levels).• Sample individual patient cost/effectiveness pairs.• Draw samples with replacement proportional to the

size of the treatment groups -- Ignore stratification in drawing the samples.

• Use the fixed, pre-specified boundaries to divide each sample into strata.

• Calculate stratified difference in cost, stratified difference in effectiveness and stratified ICER for each sample.

Interpreting the Bootstrap Samples

+Upper

95% Co

nf. In

t.

Lower 95% Conf. Int.

ICER

QALY

$

50.5%48.0%

0.5%1.0%

+

QALY

$

Windshield Wiper Regions Proportion in Each Quadrant of ICE Plane

Effects of Bootstrapping against Fixed Strata Boundaries

• The fraction of patients treated with one or the other treatment within a stratum changes from sample to sample.

• This captures (some of) the component of variability due to estimating the propensity.

• Fixed boundaries prevent technical issues that violate assumptions necessary to assure convergence of the bootstrap. (Abade and Imbens. “On the failure of the bootstrap fro Matching Estimators”. Econometrica, Vol. 76, Issue 6, pp. 1537-1557, Nov. 2008.)

Stratification Redux

• Whatever the original dimension of the covariate space, reduce the problem to cross-classification of treatments versus strata.

Stratum 1 Stratum 2 Stratum 3

Stratum 4 Stratum 5 Stratum 6

Stratum 7 Stratum 8 Stratum 9

Stratum 1 Stratum 2 … TotalTreatment nT1 nT2 NT

Control nC1 nC2 NC

Total N1 N2 N

Local Propensity Score

• Define the “Local Propensity Score” as the fraction of patients treated with treatment A within each stratum.

• Blocking can be thought of as fitting a step function for the estimated propensity.

Stratum 1 Stratum 2 … TotalTreatment nT1 nT2 NT

Control nC1 nC2 NC

Total N1 N2 N

Propensity pj = nT1/N1 nT2/N2

Inverse Propensity Weighting

A Curious Equivalence

Some Algebra

Sum by blocks

Within-block Average

Sum of Weights isTotal Sample Size

Consequences of Equivalence

• (For a class of statistics…)• Consider two individuals from different blocks but

with same propensity score. They enter IPW statistic identically with same weight.

• Therefore, even if matched via propensity score, the summary statistic remains the same.

• Stated another way, matching within strata then calculating local propensity is equivalent to deriving local propensity and then matching via that propensity.

The Downside of PS Does Not Apply to Stratified Local Propensity Scoring

• Underlying differences in covariates are irrelevant to the summary statistic.

Age

Com

orbi

ditie

s

Young and Sick

Old But Healthy

Three Possible WeightingsStratum 1 Stratum 2 Etc. Total

Treatment nT1 nT2 NT

Control nC1 nC2 NC

Total N1 N2 N

Marginal wrt Treatment:

Marginal wrt Control:

Marginal wrt Population:

Weighting Within-Stratum Differences

Three Possible WeightingsStratum 1 Stratum 2 Etc. Total

Treatment nT1 nT2 NT

Control nC1 nC2 NC

Total N1 N2 N

Marginal wrt Treatment:

Marginal wrt Control:

Marginal wrtPopulation:

Average Treatment Effect (ATE)

Average Treatment amongControls (ATT for Control Group)

Average Treatment amongTreated (ATT)

Weighting Within-Stratum Differences IPW/Causal Inference

Relationship among Weightings

Average Treatment Effect (ATE)

Average Treatment amongControls (ATT for Control Group)

Average Treatment amongTreated (ATT)

I.e., ATE is a convex combination of the ATT weightings for Treatment and Control proportional to the size of the groups.

Implications

ATT and ATE weightings are the same,

if and only if, there is uniform propensity to treat with regard to strata.

Therefore, differences under ATT and ATE weightings inform on the impact of directed prescribing (a.k.a. propensity).

Population-Based Weightings Are Not the Same as ANOVA Weightings

• SAS Type I:

• SAS Type II:

• SAS Type III:

Differ, so not collapsible to function of

Inversely proportional to variance of

Uniform across strata

Addressing Censoring

Censored Survival• A recent paper reviewed the survival component of 45 Health Technology

Assessments (HTA) submitted to National Institute for Health and Clinical Excellence (NICE) in the cancer disease area.

• Nicholas R. Latimer. “Survival Analysis for Economic Evaluations alongside Clinical Trials—Extrapolation with Patient-Level Data: Inconsistencies, Limitations, and a Practical Guide”. Medical Decision Making, published online 22 January 2013.

• A variety of methods were noted as having been used to estimate mean survival in the assessments– restricted means analyses, i.e., area under the K-M curve up until a certain point– parametric modeling (exponential, Weibull, Gompertz, etc.)– Partial Likelihood Regression/Proportional Hazards (which accounts for heterogeneity)– Reliance on estimates external to the study

• STRONG PREFERENCE of author for parametric modeling : “a lifetime horizon is usually advocated, particularly for interventions that affect survival. Therefore, in the presence of censoring, extrapolation is required to predict the complete survival impact of the new intervention, which may be summarized as the mean survival benefit.

Is Censoring Really So Bad?

• “In 17 (38%) [H]TAs, extrapolation was not performed, with the survival analysis based purely on the observed trial data (restricted means analysis). Appropriately, this was generally only the case when there was relatively little censoring in the survival data from the trial.”

• Consider a study with 1,000 patients followed until death. Now, add data from 1,000 more patients that have incomplete censored data. The supplemented data has 50% censoring but more information content. Is it really worse than a study one half the size with no censoring?

Our Approach

• The restricted K-M mean provides the advantage of using all data and being non-parametric. So long as time horizon is adequately long to fully characterize survival function, censoring should not be a problem.

• We calculate the K-M mean within each stratum as area under the curve up until the last death. This is conservative as regards the ICER as it tends to underestimate the denominator.

Censored Costs

• D. Y. Lin; E. J. Feuer; R. Etzioni; Y. Wax. “Estimating Medical Costs from Incomplete Follow-Up Data”. Biometrics, Vol. 53, No. 2. (Jun., 1997), pp. 419-434.

• Basic concept is to calculate expected cost via conditioning, i.e., as the sum all intervals of the probability of survival to start of an interval times the average cost incurred during the interval by patients alive at the start of the interval.

Calculation of Censored Cost

PatientEvent Time

Death or

Censor Cost1 Cost2 Cost3 …1 22 1 $1,533 $3,742 $4,899 2 18 1 $5,426 $2,538 $3,745 3 39 0 $6,792 $4,407 $3,890 …

Input Dataset

One Record per Patient

Time to Death or Censoring

Total Cost in Interval (E.g., Weekly Cost)

Calculation of Censored Cost

PatientEvent Time

Death or

Censor Cost1 Cost2 Cost3 …1 22 1 $1,533 $3,742 $4,899 2 18 1 $5,426 $2,538 $3,745 3 39 0 $6,792 $4,407 $3,890 …

Week S1 0.982 0.873 0.66…

Week Avg Cost1 $4,583 2 $3,562 3 $4,178 …

Kaplan-Meier Calculate Averages

Input Dataset

Probability ofSurvival untilTime Period

Average Period Cost among Patients Alive at

Start of Time Period

One Record per Time Interval

One Record per Patient

Calculation of Censored Cost

PatientEvent Time

Death or

Censor Cost1 Cost2 Cost3 …1 22 1 $1,533 $3,742 $4,899 2 18 1 $5,426 $2,538 $3,745 3 39 0 $6,792 $4,407 $3,890 …

Week S1 0.982 0.873 0.66…

Week Avg Cost1 $4,583 2 $3,562 3 $4,178 …

$10,348

Kaplan-Meier Calculate Averages

Input Dataset

Probability ofSurvival untilTime Period

Average Period Cost among Patients Alive at

Start of Time Period

One Record per Time Interval

One Record per Patient

Expected (Censoring-Adjusted) Cost

Pulling It All Together

• To adjust for heterogeneous risk stratify.• Address censoring by calculating censored mean

survival and expected cost within strata.• Pool strata relative to stratum size.• Bootstrap at individual patient level against the pre-

specified strata boundaries to incorporate variability arising from outcomes and propensity estimates.

• All together, this yields a complete method to calculate ICER in oncology.