Date post: | 17-Dec-2015 |
Category: |
Documents |
Upload: | anthony-fletcher |
View: | 215 times |
Download: | 0 times |
Can administrative data increase the practicality
of clinical trials?An example from the Women’s Health Initiative
Garnet AndersonFred Hutchinson Cancer Research Center
Pragmatic trial (coined by Schwartz and Lellouch in 1967)• A randomized controlled trial designed to inform decisions about
practice• Used to describe a trial designed to test the effectiveness of the
intervention in a broad routine clinical practice setting (as opposed to testing the efficacy in an ideal setting)
Pragmatic methods and motivations
• Broaden eligibility to improve generalizability, increase recruitment yield and perhaps reduce costs
• Test interventions/delivery mechanisms that better emulate clinical practice and thus improve estimates of population level impact and reduce costs
• Limit data collection to minimize participant burden and reduce costs
Was the original Women’s Health Initiative a pragmatic trial?
DM HT
CaD
os
48,835 27,347
36,282
93,676
CT=68,132 Total WHI =161,809
WHI transitions in post-intervention phase• Protocol streamlined to annual mail follow-up• In 2010, documentation and central adjudication of outcomes limited
to African American, Hispanic and former HT participants (Medical Records Cohort, n~22,000)
• Outcomes for remaining participants (Self-Report Cohort, N~71,000) limited to self-report or passive follow-up sources (NDI, Medicare)
A Pragmatic Trial : Physical Activity to Improve CV Health in Women
Marcia L. Stefanick, Ph.D.Charles L. Kooperberg, Ph.D.
Andrea Z. LaCroix, Ph.D.
Women’s Health Initiative Strong & Healthy Trial
1 U01 HL122280-01
Primary Hypothesis
To assess whether aerobic physical activity combined with muscle strengthening, balance and flexibility exercises, and reduced sedentary behavior, will reduce major CV events (MI, stroke, CV death) in older women, compared to “Usual Activity” (Control) over 4-5 years of follow-up
Based on: Report of the Physical Activity Guidelines Advisory Committee, 2008. (Chapter 5: Active Older Adults) http://www.health.gov/paguidelines/guidelines/chapter5.aspx
Implemented through: National Institutes on Aging (NIA) http://go4life.nia.nih.gov/
Eligibility
• WHI participant, alive and in active follow-up• In Medical Records Cohort or in Self Report cohort and enrolled in
Medicare Part A/B• Exclusions
• Inability to walk• Dementia• Residing in nursing home
Eligible based on existing data
Intervention(n ~ 26,000)
Consent
WHISH PA (Go4Life®) Interventiondeliver mail-based [+ website, etc.] ± IVR** (phone) + live advisor, PRN
Follow, per WHI protocolno
yes
yes
Randomize
Follow, per WHI protocol
Follow, per WHI protocol
Control(n ~26,000)
** Interactive Voice Response System (Consent)
Opt Out: no
Study Design: Zelen’s randomized consent design
Zelen, M. The New England Journal of Medicine 1979; 300: 1242–1245.
Sources of outcomes data
• Self-reported health events annually, for all WHI participants• Fully documented, adjudicated outcomes for the Medical Records
Cohort• Outcomes derived from Medicare claims and NDI among women from
the Self-Report Cohort
Considerations in using Medicare claims data for outcomes assessment• Coverage• Types of data available for outcomes assessments• Quality of inference• Logistics
Medicare data routinely available
• Denominator• Inpatient (MedPar)• Outpatient• Home Health• Skilled Nursing• Hospice• Durable Medical Equipment• Carrier• Part D—Prescription Drug
WHI Medicare linkage
• Submitted 151,116 names, all with valid SSN• 140,471 (93%) were age eligible as of 12/31/07 or had already died at age>65 • 142,195 names returned by Center for Medicare/Medicaid Services (CMS)• 132,109 perfect matches on 5 identifiers (94% of submitted, age-eligible)• 3,203 “fuzzy” matches (2% of submitted age eligible)
• Small discrepancies in one identifier
• 90% linkage of submitted participants • 96% linkage of age eligible participants
Note: All WHI participants are now over age 65
Linkage to Medicare does not mean claims data are available• Medicare claims not available for Managed Care (MC) members
• Included in denominator file but because reimbursement is capitated, MC organizations don’t submit claims
• MC penetrance varies over time and geographic region • Individuals may change their coverage type over time
WHI participants’ Medicare enrollment status by calendar year
1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 20120
20000
40000
60000
80000
100000
120000
140000
Any CMSFFS A+BMC
Calendar Year
WHI
Par
ticip
ants
Individuals can change their type of coverage• ~98% kept the same type of coverage over a one year period
(1/1/2011-1/1/2012)• 64% had Fee-For-Service plans (Part A or A+B)• 34% were in Managed Care
• Of the 2.12% that change types of coverage,• 0.55% went from Managed Care to Fee-For-Service• 1.57% went from FFS to MC
Determining “events” in claims data
• Claims include up to 10 diagnosis codes based on ICD-9-CM• One “primary” diagnosis• Up to 9 secondary diagnosis
• “Primary” designation and number of secondary diagnoses included is determined by the submitting institution
Assessing agreement rates between WHI adjudicated events and Medicare claims-based diagnoses
• Eligibility:• Age ≥ 65 at WHI enrollment• Linked to Medicare and continuously enrolled in fee-for-service Medicare Part
A (in-patient)
• WHI events• Documented, centrally adjudicated (hospitalized) cardiovascular outcomes• Only first post-enrollment event of each type considered
Chronic Disease Warehouse provides a catalog of algorithms for diagnoses• Myocardial infarction: hospital discharge codes (410.x0, 410.x1) as
either primary or one of 9 secondary discharge diagnosis codes. • Coronary revascularization: ICD-9-CM procedures codes for CABG
(36.1x, 36.2) or PCTA (00.66, 36.0, 36.00, 36.01, 36.02, 36.05, 36.97)
• Stroke: ICD-9-CM codes (430.xx, 431.xx, 433.x1, 434.x1, 436.xx, 437.1x, and 437.9x) in any diagnostic position
• Abdominal aortic aneurysm ICD-9-CM diagnosis (441.3-441.5, 441.9) or procedure (38.34,38.44,39.25,39.52,39.71) or CPT codes (35081, 35082, 35102, 35103, 35091, 0001T, 0002T, 35800, 34802-34805, 34830-34832)
Research Data Assistance Center (ResDAC.org) is an excellent resource for working with CMS data
Agreement rates for acute MI using principle diagnosis code
WHIMedicare Yes No TotalYes 914 281 1195No 431 35771 36202 K (95% CI)Total 1345 36052 0.71 (0.69-0.73)
Sens 68%
Spec 99.2%
PPV 76%
NPV 98.8%
Overall accuracy
98.1%
Hlatky et al., Circulation: Cardiovasc Qual Outcomes (2014)
Sensitivity improves but PPV declines using principle or secondary diagnosis codes for aMI
WHIMedicare Yes No TotalYes 1062 439 1501No 283 35613 35896 K (95% CI)Total 1345 36052 0.74 (0.72-0.75)
Sens 79%
Spec 98.8%
PPV 71%
NPV 99%
Overall accuracy
98.0%
Hlatky et al., Circulation: Cardiovasc Qual Outcomes (2014)
Agreement rates for procedures are higher: Coronary bypass graft surgery
WHIMedicare Yes No TotalYes 795 96 891No 53 36549 36506 K (95% CI)Total 848 36549 0.91 (0.90-0.93)
Sens 94%
Spec 99.7%
PPV 89%
NPV 99.9%
Overall accuracy
99.6%
Hlatky et al., Circulation: Cardiovasc Qual Outcomes (2014)
Agreement rates: Stroke events
WHIMedicare Yes No TotalYes 478 318 796No 104 30817 30603 K (95% CI)Total 582 30817 0.91 (0.90-0.93)
Sens 82%
Spec 99.0%
PPV 60.1%
NPV 99.7%
Overall accuracy
98.7%
Lakshminarayan et al., Stroke (2014)
Medicare up-coding or missing WHI events
WHI No/ Medicare Yes
PPV after Adjustment
All 318 60.1%No self-reported, hospitalized event
182 (57%) 77.9%
No adjudication 57 (18%) 85.8%Adjudication did not identify stroke
79 (25%)
Lakshminarayan et al., Stroke (2014)
Sensitivity and PPV improve when looking at agreement rates at the person level: Stroke
WHIMedicare Yes No TotalYes 505 240 745No 77 30817 30603 K (95% CI)Total 582 30817 0.91 (0.90-0.93)
Sens 86.8%
Spec 99.0%
PPV 67.8%
NPV 99.7%
Overall accuracy
98.7%
Lakshminarayan et al., Stroke (2014)
Agreement rates for cancer incidence
Site WHI-Yes CMS-Yes
WHI-No CMS-Yes
WHI-No CMS-Yes
WHI-No CMS-No
Sens (%)
Spec (%)
PPV (%)
NPV (%)
Overall
Breast 3451 1562 56 98302 98.4 98.4 68.8 99.94 98.4
Colorectal 1145 613 39 105947 96.7 99.4 65.1 99.96 99.4
Endometrial 621 146 17 61642 97.3 99.8 81.0 99.97 99.7
Lung 1236 634 63 105468 95.2 99.4 66.1 99.99 99.4
Melanoma 357 785 22 105355 94.2 99.3 31.3 99.98 99.2
Ovarian 353 372 19 86168 94.9 99.6 48.7 99.98 99.6
Medicare derived cancers based on presence of relevant ICD-9 cods in MedPar (in patient data), any position, or the first occurring combination of 2 outpatient or carrier claims containing these codes that are 1-365 days apart.
Analyzing outcomes found in claims but not in WHI
Breast (n=1562)
Colorectal (N=613)
Endometrial (N=146)
Ovarian (N=372)
N(%) N(%) N(%) N(%)Related diagnosis 888 (56.9) 210 (34.3) 62 (42.5) 199 (53.5)Other cancer orhysterectomy reported
292(18.7) 130 (21.2) 47 (32.2) 91 (24.5)
Other non-cancer hospitalization reported
199 (12.7) 148 (24.1) 11 (7.5) 41 (11.0)
Cancer self-reported & denied, or not adjudicated
95 (6.1) 85 (13.9) 19 (13.0) 24 (6.5)
No information 88 (5.6) 40 (6.5) 7 (4.8) 17 (4.6)
Correspondence of event dates
aMI Stroke AAA LE PAD CASExact same date 82% 83% 89% 75% 90%Within ± 1 day 88% 89% 79% 93%Within ± 7 days 95% 89% 82% 96%Within ± 30 days 94% 96% 92% 85% 96%Exactly ± 365 days 0.7% 0.4%
Hlatky et al., Circulation: Cardiovasc Qual Outcomes (2014)Lakshminarayan et al., Stroke (2014)Mell et al., J of Vascular Surgery (2014)
Summary of comparisons between claims-based and protocol-defined events in sample with continuous FFS Medicare enrollment
• More events counted in Medicare than in WHI across all conditions• Agreement rates ranged from good to excellent for clinical diagnoses• Excellent agreement rates for procedures• Accuracy of dates is acceptable for most failure time analyses• Errors in both sources contribute to the disagreements
Existing methods for mismeasured outcomes focus on discrete proportional hazards• Halloran ME and Longini IM. Using validation sets for outcomes and exposure to
infection in vaccine field studies. Am J of Epimiol 2001• Meier AS, Richardson BA, and Hughes JP. Discrete proportional hazards models
for mismeasured outcomes. Biometrics 2003• Magaret AS. Incorporating validation subsets into discrete proportional hazards
models for mismeasured outcomes. Statist in Med 2008
Comparing RCT results using adjudicated and claims based outcomes• Sample: Women > 65 years of age at randomization in either the HT
trial component• ITT analyses comparing HT to placebo
• WHI adjudicated outcomes, with censoring at last-follow-up date or death from other causes
• Claims-based outcomes, with censoring at end of enrollment in FFS Medicare, death from other causes, or 12/31/2007 (last available claims data)
Effect of misclassification of failure time events on inference in a RCT setting: WHI HT trial
Click icon to add picture
Hlatky et al., Circulation: Cardiovasc Qual Outcomes (2014)
Medicare found fewer clinical diagnoses, more procedures, similar hazard ratios and overall inference
Assumptions:
Independent measurement error: • Errors in outcomes data are comparable across treatment arms• May not be defensible if treatment leads to symptoms the result in greater
physician contact/diagnostic procedures, etc.
Outcomes information from secondary source (claims) does not affect the failure risk, given information on true failure time and treatment assignment
Events data collection is not defined by protocol• Study outcomes primarily limited to ICD-9 diagnoses or procedures• Diagnoses represent the community standard(s)• Changes in diagnostic procedures/codes occur outside of the
protocol, may be driven by economic factors• Data availability is at the mercy of another agency, its policies and
practices and those of the institutions who submit data to it
Trial monitoring complexities
• Will data be available soon enough for monitoring purposes?• Annual installments, based on calendar year• Timeline for release is not guaranteed
• Changes in data file structure over time• Requires additional processing time• Difficulty in locking the code for definitions
Outcomes collection and analysis plan
• For participant in the Medical Records Cohort, document and adjudicate all key outcomes
• For the remaining participants in the Self Report cohort, use Medicare Part A/B
• Documenting/adjudicating events for those who switch to managed care
• Analyses based on a stratified Cox Proportional Hazards model with source of data as one of the stratification factors
Monitoring plan considerations
• Intervention is currently available to all (including the comparison group)
• Ethical requirements to stop for efficacy are reduced• Need a full-scale evaluation of this public health program
• Safety concerns are dominant• Medicare may be too tardy, insensitive to monitor safety
• Options: Supplement Medicare data with self-report, particularly in the early phases
Conclusions
• Claims data provide a key source of selected outcomes data that are standardized and available across the nation for a large segment of the population
• The correspondence between claims derived outcomes and traditionally documented and adjudicated outcomes are good to excellent but vary by type of outcome
• Randomized trials using claims data for outcomes may derive valid inference if measurement error assumptions are met
• Timeliness of obtaining claims data may not be adequate for trial monitoring