Study Design Considerations for Observational Comparative
Effectiveness Research
Prepared for:Agency for Healthcare Research and Quality (AHRQ)
www.ahrq.gov
This presentation will: Provide a rationale for study design choice
and describe key design features Define start of followup Define inclusion and exclusion criteria at
start of followup Define exposures of interest at start of
followup. Define outcome(s) of interest Define potential confounders
Outline of Material
Conventional designs Cohort Case-control Case-cohort
Self-controlled designs Case-crossover Case-time-controlled Self-controlled case series
Study Design Overview
Exposures or treatments are not assigned, a situation which leads to challenges ensuring internal validity, that is, the absence of bias. To ensure internal validity, treatment groups compared
must have the same underlying risk for outcome within subgroups definable by measured covariates (e.g., no unmeasured confounding). Confounding by indication leads to higher propensity
for/more intensive treatment in those with the most severe disease.
With confounding by frailty, frail patients (close to death) are less likely to be treated with preventive treatments.
Ensuring a study’s internal validity is a prerequisite for its external validity or generalizability.
Issues of Bias in ObservationalComparative Effectiveness Research
Cohorts are defined by their exposure at a certain point in time (i.e., baseline date) and are followed over time for the occurrence of the outcome.
Advantages: Has a clear timeline separating potential confounders from
the exposure and the exposure from the outcome Allows estimation of actual incidence (risk or rate) Can assess multiple outcomes Is easy to conceptualize
Limitations: Is inefficient for ad hoc studies when the incidence of the
outcome is low
Study Design: Cohort Study
Identifies all incident cases that develop an outcome and compares exposure history to controls
Samples controls at random from cohort members at risk for developing an outcome
Advantages: Oversampling cases increases computational efficiency
of ad hoc studies when compared with a cohort study Can assess multiple exposures
Limitations: Is difficult to conceptualize Has potential for recall bias in ad hoc studies
Study Design: Case-Control Study
Cohorts defined as in a cohort study Cohort members followed for incidence of
outcomes Additional information required for analysis
collected for a random sample of the cohort and all cases
Increased efficiency, when compared with a full-cohort design, if additional information needs to be collected
Decreased efficiency, when compared with a nested case-control design, unless studying multiple outcomes or estimating risk
Study Design: Case-Cohort Study
Prior exposure history of cases used as the control Removes confounding effect of measured and unmeasured
characteristics that are stable over time (e.g., genetics) Appropriate for studying acute effects of transient
exposures Advantages:
Self-controlled Ability to assess short-term reversible effects Ability to inform about the time window for these
effects Limitations:
Assumes constant prevalence of treatments over time Does not allow estimation of treatment effect in a
population
Study Design: Case-Crossover Study
Adjusts for calendar time trends in the prevalence of treatments, which can bias the case-crossover design
Divides the case-crossover odds ratio by the equivalent odds ratio estimated in controls
Advantages: Not dependent on assumption of no temporal
changes in the prevalence of treatment Limitations:
Need for controls adds complexity Control for time trend can introduce confounding
Study Design: Case-Time-Controlled Study
Estimates the immediate effect of treatment in those treated at least once
Dependent on cases that have changes in treatment during a defined observation period
Advantages: Controls for factors that are stable over time Cohort design has the potential to increase efficiency Well suited for rare adverse events in vaccine safety
studies Limitations:
Limited applicability in many comparative effectiveness research studies
Study Design: Self-Controlled Case-Series
Study setting Consideration of the study population and data
source(s) Inclusion and exclusion criteria
Should be clearly defined Include details about the study time period
Choice of comparators Reduces potential for confounding by comparing
treatment of interest with a different treatment for the same indication or an indication with the same potential for confounding
Study Design Features
New-User Design The conventional prevalent user design is prone to
confounding and selection bias as a result of changes in treatment effects over time. Including only new users reduces bias and confounding
associated with inclusion of prevalent users. There must be a clear starting point for followup under
similar conditions of medicalization. Immortal Time Bias
Occurs as a result of defining the exposure during the followup time rather than before followup
New-user design and use of comparator treatments reduce potential for this bias
Other Study Design Considerations
Knowledge of study design options is essential to increase internal and external validity of observational comparative effectiveness research.
Biases introduced by suboptimal study design cannot usually be removed by statistical analysis.
Cohort design is preferred when data have already been collected; the validity of a nested case-control study is equivalent, given proper control selection and timing of exposures and covariates.
It is important to define the start of followup, inclusion and exclusion criteria, outcome of interest, and potential confounders at the outset.
Conclusions
Summary Checklist (1 of 3)
Guidance Key ConsiderationsProvide rationale for study design choice and describe key design features
• Cohort study proposals should clearly define the cohort entry date (baseline date), employ a new-user design (or provide a rationale for including prevalent users), and plans for reporting losses to followup.
• Case-control study proposals should clearly describe the control sampling method, employ a new-user design (or provide a rationale for assessing confounders at the index date), and assess potential for recall bias (if applicable).
• Case-cohort study proposals should include how the sampling scheme will be accounted for during analysis.
• Case-crossover study proposals should discuss the potential for confounding by time-varying factors and clearly state how the resulting effect estimate can be interpreted.
• Case-time-controlled study proposals should clearly weigh the pros and cons of accounting for calendar time trends in the prevalence of exposure.
Summary Checklist (2 of 3)
Guidance Key Considerations
Define start of followup (baseline)
• The time point for start of followup should be clearly defined, meaningful, and ideally anchored to the time of a medical intervention (e.g., initiation of drug use).
• If alternative approaches are proposed, the rationale should be provided and implications discussed.
Define inclusion and exclusion criteria at start of followup
• Exclusion and inclusion criteria should be defined at the start of followup (baseline) and solely based on information available at this point in time (i.e., ignoring potentially known events after baseline).
• The definition should include the time window for assessment (usually the same for all cohort members).
Summary Checklist (3 of 3)
Guidance Key Considerations
Define exposure (treatments) of interest at start of followup
• Use an active comparator (indicated as alternative treatment at same stage of disease progression) when possible.
Define outcome(s) of interest
• Provide information on measures of accuracy if possible.
Define potential confounders
• Potential confounders known to be associated with treatment and outcome should be prespecified when possible.
• Confounders should be assessed before exposure or treatment initiation to ensure they are not affected by the exposure.
• Approaches to empirical identification of confounders should be described if planned.