Designing Cluster Randomized Trials for Health Policy Research
8th International Conference on Health Policy Statistics [ICHPS]
Washington, DC: January 20, 2010
Thomas E. Love, Ph.D. [email protected] Neal V. Dawson, M.D. [email protected] Randall D. Cebul, M.D. [email protected]
Center for Health Care Research and Policy Case Western Reserve University at MetroHealth Medical Center
Cleveland, Ohio www.chrp.org
Workshop Schedule
3:30 – 3:45 Introduction to EMR-Centered Decision Support and Design of a Cluster Randomized Trial [Love, for Cebul]
3:45 – 4:00 Group Task: Designing a Cluster Randomized Trial
4:00 – 4:15 Discussion of Group Task / Strategic Concerns [Group]
4:15 – 4:40 Ethics and Implementation / “When Stuff Happens” [Dawson]
4:40 – 5:05 A Glimpse at Some Analytic Concerns [Love]
5:05 – 5:15 Additional Group Discussion and Workshop Evaluation
8th International Conference on Health Policy Statistics (Washington, D. C.) Designing Cluster Randomized Trials for Health Policy Research
RD Cebul NV Dawson TE Love – Case Western Reserve U. – January 2010 – Front Materials – Page ii
Group Task
Study: Improving the care and outcomes of patients with diabetes
Background: A health care system with a well-established electronic medical record
(EMR) capable of providing various Clinical Decision Support (CDS) functions decides
to undertake a controlled trial to determine whether CDS can improve the care and
outcomes of its approximately 9,000 patients with diabetes. Selected characteristics of
the system are described below, including the proportion of its patients by insurance
class. The system asks for your help in planning and designing the study.
Sites Primary Care Providers Diabetes Patients
10 115 8,804
Primary Insurance Class (%) Commercial / Medicare Medicaid Uninsured / Self-Pay
59 26 15
Task: As a group, you’ll have 15 minutes to select a reporter/scribe, discuss the key
features of the trial that you would recommend they undertake, and then prepare a one-
minute oral report
for the rest of the workshop which describes your group’s views
on the three concerns below, and any additional issues you come up with.
Specifically, it would be helpful to address the following concerns:
a. What would the intervention(s) be (i.e; what would you be comparing)?
b. What would the unit of assignment (to the interventions) be?
c. Identify other information that you would request in order to improve
your recommendations.
8th International Conference on Health Policy Statistics (Washington, D. C.) Designing Cluster Randomized Trials for Health Policy Research
RD Cebul NV Dawson TE Love – Case Western Reserve U. – January 2010 – Front Materials – Page iii
A Few Terms/Acronyms We May Use
CCHIT www.cchit.org/ Created to establish national standards in ambulatory and inpatient EMRs (functionality, interoperability, and security), CCHIT is a coalition of private sector organizations (the American Health Information Management Association, the Healthcare Information and Management Systems Society and The National Alliance for Health Information Technology; released list of first certified ambulatory EMR systems in July, 2006.
Certification Commission for Healthcare Information Technology.
CDS Clinical Decision Support
. In the context of this course, CDS is understood to mean "real-time" clinical decision support that is integrated with functions of electronic medical records systems
CRT Cluster Randomized Trial
. The subject of this course! In contrast to conventional RCT, in which the unit of assignment is typically the same (e.g., a patient) as the unit of analysis.
CPOE Computerized Physician Order Entry
. A function of electronic medical records systems, may be integrated with clinical decision support functions to improve quality, safety, etc (e.g., warning of potential drug interactions)
ICC Intra-cluster Correlation Coefficient
. Also described as rho (ρ), the ICC is the main metric of "clustering"; the amount of variance in a measure that is attributable to differences between clusters; ranges from 0-1. ICCs near zero imply virtually complete statistical independence of members across clusters in a CRT.
MRC United Kingdom's Medical Research Council
. Established a classification system and made recommendations for types of informed consent in CRTs.
8th International Conference on Health Policy Statistics (Washington, D. C.) Designing Cluster Randomized Trials for Health Policy Research
RD Cebul NV Dawson TE Love – Case Western Reserve U. – January 2010 – Front Materials – Page iv
A Very Partial Bibliography on Methods for Cluster-Randomized Trials Compiled by TE Love
Textbooks
1. Donner A Klar N (2000) Design and Analysis of Cluster Randomization Trials in Health Research. London: Arnold.
2. Murray DM (1998) Design and Analysis of Group-Randomized Trials. New York: Oxford University Press.
3. Cochrane Handbook for Systematic Reviews: Section on Cluster-Randomized Trials. http://www.mrc-bsu.cam.ac.uk/cochrane/handbook/chapter_16/16_3_cluster_randomized_trials.htm
Review and General Articles on CRTs
4. Cebul RD Dawson NV Love TE (2007) Cluster Randomized Trials in Health Care Research. Textbook of Clinical Trials, 2nd Edition, edited by Machin D Day S Green S Chapter 36. Wiley.
5. Campbell MK Elbourne DR Altman DF for the CONSORT group (2004) CONSORT (Consolidated Standards of Reporting Trials) Statement: Extension to cluster randomized trials. BMJ 328: 702-708.
6. Cluster Randomised Trials: Methodological and Ethical Considerations, Clinical Trials Series. London: Medical Research Council (2002).
7. Donner A Klar N (2004) Pitfalls of and controversies in cluster randomization trials. Am J Public Health 94: 416-422.
8. Eldridge SM, Ashby D, Feder GS, Rudnicka AR, Ukoumunne OC. Lessons for cluster randomized trials in the twenty-first century: a systematic review of trials in primary care. Clin Trials (2004) 1(1): 80–90.
9. Grimshaw JM Eccles M Campbell MK Elbourne D Cluster randomised trials of professional and organizational behaviour change interventions in health care settings. [Online at http://www.campbellcollaboration.org/Bellagio/Papers/crt_grimshaw.pdf]
10. MacLennan GS Ramsay CR Mollison J Campbell M Grimshaw J Thomas R (2003) Room for improvement in the reporting of cluster randomized trials in behavior change research. Controlled Clinical Trials 24: 69S-70S.
11. Murray DM Varnell SP Blitstein JL (2004) Design and analysis of Group-Randomized Trials: A review of recent methodological developments. Am J Public Health 94: 423-432.
12. Puffer S Torgerson D Watson J (2003) Evidence for risk of bias in cluster randomized trials: review of recent trials published in three general medical journals. BMJ 327 (7418): 785-9.
13. Simpson JM, Klar N, Donner A. Accounting for cluster randomization: a review of primary prevention trials, 1990 through 1993. Am J Public Health (1995) 85(10): 1378–83.
8th International Conference on Health Policy Statistics (Washington, D. C.) Designing Cluster Randomized Trials for Health Policy Research
RD Cebul NV Dawson TE Love – Case Western Reserve U. – January 2010 – Front Materials – Page v
Designing CRTs: Epidemiologic Concerns (Power, etc.)
14. Campbell MK Mollison J Grimshaw JM (2001) Cluster trials in implementation research: Estimation of intracluster correlation coefficients and sample size. Statistics in Medicine 20: 391-399.
15. Campbell MK Thomson S Ramsay CR MacLennan GS Grimshaw JM (2004) Sample size calculator for cluster randomised trials. Comput Biol Med 34:113-125. [Note that the calculator is found at the University of Aberdeen’s Health Service Research Unit Web Site: http://www.abdn.ac.uk/hsru/epp/cluster.shtml which also contains an ICC database.]
16. Campbell MK, Fayers PM, Grimshaw JM. Determinants of the intracluster correlation coefficient in cluster randomized trials: the case of implementation research. Clin Trials (2005) 2(2): 99–107.
17. Campbell MK, Grimshaw JM, Steen N. Sample size calculations for cluster randomised trials. Changing Professional Practice in Europe Group (EU BIOMED II Concerted Action). J Health Serv Res Policy (2000) 5(1): 12–16.
18. Carter BR Hood K (2008) Balance algorithm for cluster randomized trials. BMC Med Res Methodology 8:65 http://www.biomedcentral.com/1471-2288/8/65
19. Eldridge SM Cryer C Feder GS Underwood M (2001) Sample size calculations for intervention trials in primary care randomizing by primary care group: An empirical illustration from one proposed intervention trial. Statistics in Medicine 20: 367-376.
20. Gail MH Mark SD Carroll RJ Green SB Pee D (1996) On design considerations and randomization-based inference for community intervention trials. Statistics in Medicine 15: 1069-1092.
21. Giraudeau B Ravaud P (2009) Preventing bias in cluster randomized trials. PLoS Medicine 6(5): e1000065. http://clinicaltrials.ploshubs.org/article/info:doi%2F10.1371%2Fjournal.pmed.1000065
22. Glynn RJ Brookhart MA Stedman M Avorn J Solomon DH (2007) Design of cluster-randomized trials of quality improvement interventions aimed at medical care providers Med Care 45: S38-S43. Available at http://effectivehealthcare.ahrq.gov/repFiles/MedCare/s38.pdf
23. Guittet L Ravaud P Giraudeau B (2006) Planning a cluster randomized trial with unequal cluster sizes: practical issues involving continuous outcomes. BMC Medical Research Methodology 6:17 http://www.biomedcentral.com/1471-2288/6/17
24. Godwin M Ruhland L Casson I et al. (2003) Pragmatic controlled clinical trials in primary care: The struggle between external and internal validity. BMC Medical Research Methodology 3: 28.
25. Kerry SM Bland JM (1998) Sample size in cluster randomization. BMJ 316: 549. 26. Kerry SM Bland JM (1998) The intracluster correlation coefficient in cluster randomization.
BMJ 316: 1455. 27. Lewsey JD (2004) Comparing completely and stratified randomized designs in cluster
randomized trials when the stratifying factor is cluster size: A simulation study. Statistics in Medicine 23: 897-905.
28. Love TE Cebul RD Einstadter D Jain AK Miller H Harris CM Greco PJ Husak SS Dawson NV Electronic medical record-assisted design of a cluster-randomized trial to improve diabetes care and outcomes. J Gen Internal Med 2008; 23(4): 383-391.
8th International Conference on Health Policy Statistics (Washington, D. C.) Designing Cluster Randomized Trials for Health Policy Research
RD Cebul NV Dawson TE Love – Case Western Reserve U. – January 2010 – Front Materials – Page vi
29. Mickey RM Goodwin GD (1993) The magnitude and variability of design effects for community intervention studies. Am J Epidemiol 137: 9-18.
30. Raab GM, Butcher I. Balance in cluster randomized trials. Stat Med (2001) 20(3): 351–65. 31. Raab GM, Butcher I. Randomization inference for balanced cluster-randomized trials. Clin
Trials (2005) 2(2): 130–40. 32. Raudenbush SW (1997) Statistical analysis and optimal design for cluster randomized trials.
Psychological Methods 2, 173-185. [See “Optimal Design” under Sprybook J (2006)] 33. Raudenbush SW Liu X (2000) Statistical power and optimal design for multisite randomized
trials. Psychological Methods 5, 199-213. [See Sprybook J (2006)] 34. Sprybook J Raudenbush SW Liu X Congdon R (2006) Optimal Design for Longitudinal and
Multi-Site Research: Documentation for the “Optimal Design” software. [Documentation and software available at http://sitemaker.umich.edu/group-based/optimal_design_software]
35. Thompson SG Pyke SDM Hardy RJ (1997) The design and analysis of paired cluster randomized trials: An application of meta-analysis techniques. Statistics in Medicine 16: 2063-2079.
Analyzing CRTs: Biostatistical Concerns
36. Berger VW Exner DV (1999) Detecting selection bias in randomized clinical trials. Controlled Clinical Trials 20: 319-327.
37. Borm GF Melis RJF Teerenstra S Peer PG (2005) Pseudo cluster randomization: A treatment allocation method to minimize contamination and selection bias. Statistics in Medicine 24: 3535-3547.
38. Braun TM Feng ZD (2003) Identifying settings when permutation tests have nominal size with paired, binary outcome, group randomized trials. J Nonparametr Stat 15: 653–63.
39. Dobbins TA Simpson JM (2002) Comparison of tests for categorical data from a stratified cluster randomized trial. Statistics in Medicine 21: 3835-3846.
40. Donner A (1987) Statistical methodology for paired cluster designs. Am J Epidemiol 126: 972-979.
41. Donner A Klar N (1993) Confidence interval construction for effect measures arising from cluster randomization trials. J Clin Epidemiol 46: 123-131.
42. Donner A Klar N (1994) Methods for comparing event rates in intervention studies when the unit of allocation is a cluster. Am J Epidemiol 140: 279-289.
43. Giraudeau B (2006) Model misspecification and overestimation of the intraclass correlation coefficient in cluster randomized trials. Statistics in Medicine 25: 957-964.
44. Hansen BB Bowers J (2008) Covariate balance in simple stratified, and clustered comparative studies. Statistical Science 23 (2): 219-236.
45. Kerry SM Bland JM (1998) Analysis of a trial randomized in clusters. BMJ 316: 54. 46. Koepsell TD Martin DC Diehr PH et al. (1991) Data analysis and sample size issues in
evaluations of community-based health promotion and disease prevention programs: A mixed-model analysis of variance approach. J Clin Epidemiol 44: 701-713.
47. Murray DM, Blitstein JL. Methods to reduce the impact of intraclass correlation in group randomized trials. Eval Rev (2003) 27(1): 79–103.
8th International Conference on Health Policy Statistics (Washington, D. C.) Designing Cluster Randomized Trials for Health Policy Research
RD Cebul NV Dawson TE Love – Case Western Reserve U. – January 2010 – Front Materials – Page vii
48. Nixon RM Thompson SG (2003) Baseline adjustments for binary data in repeated cross-sectional cluster randomized trials. Statistics in Medicine 22: 2673-2692.
49. Peters TJ Richards SH Bankhead CR Ades AE Sterne JAC (2003) Comparison of methods for analyzing cluster randomized trials: An example involving a factorial design. Int J Epidemiol 32: 840-846.
50. Rosenbaum PR. Covariance adjustment in randomized experiments and observational studies. Stat Sci (2002) 17: 286–304.
51. Saha KK Paul SR (2005) Bias-corrected maximum likelihood estimator of the intraclass correlation parameter for binary data. Statistics in Medicine 24: 3497-3512.
52. Spiegelhalter DJ (2001) Bayesian methods for cluster randomized trials with continuous responses. Statistics in Medicine 20: 435-452.
53. Turner RM Omar RZ Thompson SG (2001) Bayesian methods of analysis for cluster randomized trials with binary outcome data. Statistics in Medicine 20: 453-472.
54. Ukoumunne OC Thompson SG (2001) Analysis of cluster randomized trials with repeated cross-sectional binary measurements. Statistics in Medicine 20: 417-433.
55. Yudkin PL Moher M (2001) Putting theory into practice: A cluster randomized trial with a small number of clusters. Statistics in Medicine 20: 341-349.
Ethical and Human Subjects Concerns in CRTs
56. Eldridge SM Ashby D Feder GS (2005) Informed patient consent to participation in cluster randomized trials: An empirical exploration of trials in primary care. Clinical Trials 2: 91-98.
57. Hutton JL (2001) Are distinctive ethical principles required for cluster randomized controlled trials? Statistics in Medicine 20: 473-488.
Information on CRTs and Diabetes, CDS, and Related Fields
58. Bland M (2003) Cluster randomized trials in the medical literature. [Online at http://www.users.york.ac.uk/~mb55/talks/clusml.htm]
59. Borgermans L Goderis G van den Broeke C et al. (2008) A cluster randomized trial to improve adherence to evidence-based guidelines on diabetes and reduce clinical inertia in primary care physicians in Belgium: study protocol. Implementation Science 3: 42. Available online: http://www.implementationscience.com/content/3/1/42
60. Cleveringa FG Gorter KJ van den Donk M Rutten GE (2008) Combined task delegation, computerized decision support and feedback improve cardiovascular risk for type 2 diabetic patients: A cluster randomized trial in primary care. Diabetes Care 31: 2273-2275.
8th International Conference on Health Policy Statistics (Washington, D. C.) Designing Cluster Randomized Trials for Health Policy Research
RD Cebul NV Dawson TE Love – Case Western Reserve U. – January 2010 – Front Materials – Page viii
61. Fretheim A Oxman AD Havelsrud K et al. (2006) Rational Prescribing in Primary Care (RaPP): A Cluster Randomized Trial of a Tailored Intervention. PLoS Med 3(6): e134.
62. Glasgow RE Nutting PA King DK et al. (2004) A practical randomized trial to improve diabetes care. J Gen Intern Med 19: 1167-1174.
63. Glasgow RE Nutting PA King DK et al. (2005) Randomized effectiveness trial of a computer-assisted intervention to improve diabetes care. Diabetes Care 28, 33-39.
64. Grant RW Cagliero E Sullivan CM et al. (2004) A controlled trial of population management: diabetes mellitus: putting evidence into practice (DM-PEP). Diabetes Care 27: 2299-2305.
65. Gulmezoglu AM Villar J Grimshaw JK et al. (2004) Cluster randomized trial of an active, multifaceted information dissemination intervention based on the WHO reproductive health library to change obstetric practices: Methods and design issues. BMC Medical Research Methodology 4: 2.
66. Han (2005) Unexpected increased mortality after implementation of a commercially sold CPOE system. Pediatrics 116: 506-12.
67. Hillestad R et al. (2005) Can EMR systems transform health care? Potential health benefits, savings, and costs. Health Aff. 24: 1103-1117.
68. MacLean CD Gagnon M Callas P Littenberg B (2009) The Vermont Diabetes Information system: A Cluster Randomized Trial of a Population Based Decision Support System. JGIM 24 (12). Available online: http://www.springerlink.com/content/j175384173jhx410/fulltext.pdf
69. Meigs JB Cagliero E Dubey A et al. (2003) A controlled trial of web-based diabetes disease management. Diabetes Care 26: 750-757.
70. Montori VM Dinneen SF Gorman CA et al. (2002) The impact of planned care and a diabetes electronic management system on community-based diabetes care: the Mayo Health System Diabetes Translation Project. Diabetes Care 25: 1952-1957.
71. Moore H Summerbell CD Vail A Greenwood DC Adamson AJ (2001) The design features and practicalities of conducting a pragmatic cluster randomized trial of obesity management in primary care. Statistics in Medicine 20: 331-340.
72. Parker DR, Evangelou E, Eaton CB. Intraclass correlation coefficients for cluster randomized trials in primary care: the cholesterol education and research trial (CEART). Contemp Clin Trials (2005) 26(2): 260–7.
73. Samore MH Bateman K Alder SC et al. (2005) Clinical decision support and appropriateness of antimicrobial prescribing. JAMA 294: 2305-2314.
74. Sequist TD Gandhi TK Karson AS et al. (2005) A randomized trial of electronic clinical reminders to improve quality of care for diabetes and coronary artery disease. J Amer Med Informatics Assoc 12: 431-437.
75. Slymen DJ Elder JP Litrownik AJ Ayala GX Campbell NR (2003) Some methodologic issues in analyzing data from a randomized adolescent tobacco and alcohol use prevention trial. J Clin Epidemiol 56: 332-340.
76. Sturt J Hearnshaw H Farmer A Dale J Eldridge S (2006) The Diabetes Manual trial protocol – a cluster randomized controlled trial of a self-management intervention for type 2 diabetes. BMC Fam Pract 7:45.
8th International Conference on Health Policy Statistics – Washington DC – January 2010
Cluster Randomized Trials in Health Policy Research
Part One: Setting Up The Task of Designing A Study to Evaluate Clinical Decision Support
Randell D. Cebul, M. D. [email protected] Part 1: Page 1
Cluster Randomized Trials (CRTs)in Health Policy Research
(with a focus on CRTs of clinical decision support)
Thomas E. Love, PhD filling in for
Randall D. Cebul, MDJanuary 20, 2010
Center for Health Care Research & PolicyCase Western Reserve University at
MetroHealth Medical [email protected]
Cluster Randomized Trials?
As compared to individually randomized trials (RCTs),
• CRTs are more complex to design
2
• CRTs require more participants to obtain equivalent statistical power
• CRTs require more complex analysis
Overview of Workshop
• Introduction and Context– Electronic medical records (EMRs) to provide and examine clinical decision support (CDS)
• Group Task: Design a Study• Group Task: Design a Study• Task Review• CRTs: Why they are necessary and challenging to execute
• Analytic and Strategic Concerns• Discussion
8th International Conference on Health Policy Statistics – Washington DC – January 2010
Cluster Randomized Trials in Health Policy Research
Part One: Setting Up The Task of Designing A Study to Evaluate Clinical Decision Support
Randell D. Cebul, M. D. [email protected] Part 1: Page 2
Clinical Decision Support (CDS)
Consensus CDS Definition (AMIA):“Providing clinicians, patients, or individuals with knowledge and person‐specific or population information intelligently filteredpopulation information, intelligently filtered or presented at appropriate times, to foster better health processes, better individual patient care, and better population health.”
Clinical Decision Support, informally
Giving the right informationto the right person
at the right time and placeat the right time and place,
making the right decision the easy decision.
Illustrative RealIllustrative Real‐‐time CDS time CDS ––designed to change (?improve) behaviordesigned to change (?improve) behavior
• Provider centered CDS
• Inpatient • Computerized Physician Order Entry, “Smart tools”, guidelines, etc.
• Outpatient • Alerts, linked orders/referrals, links to patient/MD education, etc.
• Patientcentered• access to webbased records, reminders, input/feedback, etc.
8th International Conference on Health Policy Statistics – Washington DC – January 2010
Cluster Randomized Trials in Health Policy Research
Part One: Setting Up The Task of Designing A Study to Evaluate Clinical Decision Support
Randell D. Cebul, M. D. [email protected] Part 1: Page 3
EncounterEncounter‐‐based Alert to Improve care for based Alert to Improve care for Diabetic Patients with Leaking Kidneys Diabetic Patients with Leaking Kidneys
What do we know about this patient?What do we know about this patient?
{Links to Automated Order Set}
• She has diabetes and is visiting her PCP• Her kidneys are leaking protein.• She is not on an ACE inhibitor or ARB and has no documented allergies to them.
• She has no other contraindications (K, Cr)•• There are several alternative drugs/dosesThere are several alternative drugs/doses
Automated Order Set Automated Order Set Linked to ACE/ARB AlertLinked to ACE/ARB Alert
Patient name
Patient name
Re-cap of indications
Choice of Rxs/doses
Follow-up testing
Comparative Performance FeedbackComparative Performance Feedback
“My panel” vs. Comparator
8th International Conference on Health Policy Statistics – Washington DC – January 2010
Cluster Randomized Trials in Health Policy Research
Part One: Setting Up The Task of Designing A Study to Evaluate Clinical Decision Support
Randell D. Cebul, M. D. [email protected] Part 1: Page 4
CDS for Patients: Viewing their information,CDS for Patients: Viewing their information,inputting data, getting feedback at homeinputting data, getting feedback at home
CDS with EMRs: Untapped Promise?CDS with EMRs: Untapped Promise?
“we conclude that effective EMR implementation…could save more than $81 billion annually and that HITenabled management of chronic disease could eventually double those savings while increasing health benefits.”
– Hillestad R et.al. (2004)
Or… increased mortality (with CPOE)?Or… increased mortality (with CPOE)?
• Pittsburgh Children’s (Cerner)– 13 months preCPOE– 5 months postCPOE
• 1,942 children admitted for specialty care75 deaths– 75 deaths• 39 / 1394 preCPOE (2.80%)(2.80%)• 36 / 548 post (6.57%)(6.57%)
–– Multivariate OR 3.28 (CI 1.94 Multivariate OR 3.28 (CI 1.94 –– 5.55)5.55)• But: retrospective, short postimplementation…
• Han (2005)
8th International Conference on Health Policy Statistics – Washington DC – January 2010
Cluster Randomized Trials in Health Policy Research
Part One: Setting Up The Task of Designing A Study to Evaluate Clinical Decision Support
Randell D. Cebul, M. D. [email protected] Part 1: Page 5
Need for EvidenceNeed for Evidence
• What do we need to know?– Will providers adopt CDS? – Safety of Implementation, System Effects– Safety and Quality Implications of Alerts– FPs, Alert Fatigue– Cost implications
• Will they result in more costeffective care?• What is most effective? Over what time horizon?
– Etc – the possibilities are endless…
• How do we study CDS?
Group TasksGroup Tasks
1. Describe key features of a CDS‐catalyzed trial to improve the care and outcomes of patients with diabetes.
2 Identify other information needed to2. Identify other information needed to improve the trial.
One‐Minute Oral ReportsA health care system with an established electronic medical record (EMR) capable of providing various clinical decision support (CDS) functions decides toundertake a controlled trial to determine whether CDS can improve care and outcomes of its 8804
diabetic patients
a. What would the intervention be?b. What should be the unit of assignment?c. What other information do you want?
diabetic patients. 10 sites, 115 primary care providers, 8804 DM patients
Insurance: 59% commercial or Medicare, 26% Medicaid, 15% Uninsured
8th International Conference on Health Policy Statistics – Washington DC – January 2010
Cluster Randomized Trials in Health Policy Research
Part One: Setting Up The Task of Designing A Study to Evaluate Clinical Decision Support
Randell D. Cebul, M. D. [email protected] Part 1: Page 6
Task Review – Part A: InterventionWhat Should We Be Comparing?
Part B: Unit of Assignment?
Task Review –Part C: Other Useful Information
• RCT (Randomized Controlled Trial)– unit that is randomized is the unit of analysis– for large n, important attributes are likely to be distributed similarly across groups
• CRT (Cluster‐Randomized Trial)
Comparing RCTs and CRTs
• CRT (Cluster‐Randomized Trial)– interventions focus on systems, prevention, behavior
– useful when “contamination” is likely to be a problem and/or blinding is impossible
– unit randomized is not the sole unit of interest – “clustering” within units is the critical issue
8th International Conference on Health Policy Statistics – Washington DC – January 2010
Cluster Randomized Trials in Health Policy Research
Part One: Setting Up The Task of Designing A Study to Evaluate Clinical Decision Support
Randell D. Cebul, M. D. [email protected] Part 1: Page 7
Patients within Physicians within Practices within Study Groups: A 4‐Level CRT
19
Cebul RD Dawson NV Love TE (2007) Textbook of Clinical Trials, 2nd Ed., Wiley.
CRTs: Impact of Clustering
If there are important cross‐group differences in important factors at baseline, this affects:
• Study Power [Effective Sample Size]
• Study Design: y g– Pre‐randomization cluster “balancing” will improve balance of important prognostic factors
• Study Analysis:– Need analytic techniques that account for clustering: e.g. GEE, hierarchical models, etc.
8th International Conference on Health Policy Statistics – Washington– January 2010
Cluster Randomized Trials in Health Policy Research
Part Two: Ethics and Implementation of Cluster Randomized Trials
Neal V. Dawson, MD [email protected] Part 2: Page 1
Ethics and Implementation
When CRT = Creative Responses to Trying Circumstances
1
“Stuff Happens”
Neal V. Dawson, MDJanuary 2010
Ethical Issues in CRTs
• Wide spectrum of CRT designs
–Can resemble individual RCTs where each subject decides whether to participate
–May involve the randomization of whole practices, communities, or countries
UK MRC Clinical Trials Series
• Type A trials – are structured such that they do not allow participation decisions by individuals– Often CRTs
• Type B trials – allow individual subjects to decide about participation– Usually individual RCTs
8th International Conference on Health Policy Statistics – Washington– January 2010
Cluster Randomized Trials in Health Policy Research
Part Two: Ethics and Implementation of Cluster Randomized Trials
Neal V. Dawson, MD [email protected] Part 2: Page 2
Usual Ethical Concerns Apply
• Potential to produce findings that can improve human health or welfare
• Favorable balance of risks and benefits• Conflicts: subjects’ welfare prevails over Conflicts: subjects welfare prevails over
interests of science and society• Voluntary informed consent when possible
– Alternative safeguards – cluster representation
• Alternative consent mechanisms• Review by independent ethics review
committee
Ethical Concerns for Type A Trials
• Mechanisms for representing the interests of the cluster – Representative (person or group)– Sufficient knowledge of circumstances, beliefs,
and values of cluster– Delegated authority from or for the cluster– No conflicts of interest
• Cluster representative: ‘individual rights’– Suitably informed– Able to withdraw without adverse impact
Specific Trial Examples
• Many quality improvement studies– Interventions on groups rather than individuals
– Interventions targeted at health care professionals
Ad i i t ti t i l• Administrative trials– Studies that do not intrude into physician-patient
decision making
– Studies of aspects of health care activities about which patients never make decisions (e.g., patient scheduling schemes to improve patient flow)
8th International Conference on Health Policy Statistics – Washington– January 2010
Cluster Randomized Trials in Health Policy Research
Part Two: Ethics and Implementation of Cluster Randomized Trials
Neal V. Dawson, MD [email protected] Part 2: Page 3
Written Informed Consent• May be impractical and produce important amounts
of bias
– Tu et al. NEJM 2004;350:1414-21
– Attempted written informed consent of consecutive patients for stroke registry in Canadaconsecutive patients for stroke registry in Canada
– Participation: Phase 1=39% (4285 eligible), Phase 2=51% (2823 eligible); Many died or were discharged before they could be approached
– Selection bias: Inhospital mortality, Enrolled=6.9%; Not enrolled=21.7%;
– Registry patients not representative
Alternative Consent Mechanisms
• Goldberg. Medical Care 1990;28:822-33• Administrative trials: Firm trials and many CRTs
where doctor-patient decision making is not compelled or constrained‘P i N ifi i ’ b di h b • ‘Prior Notification’ – about studies that may be done to improve care; similar to commonly used notifications about possible use of patient records for epidemiologic or biomedical research (HIPPA issues are separate)
• “You and your physician will always be able to determine which tests and treatments you will receive.”
Opt-out Strategies
• Mechanism for passive consent• Littenberg and MacLean. JGIM 2006;21:207-11
– Quality improvement intervention: Vermont Diabetes Information SystemMulti state randomized trial– Multi-state randomized trial
– Patients were notified by mail that they were eligible – Were able to opt-out by calling a toll-free number– Of 7558 patients invited to participate, 210 (2.8%) opted-
out
• Recruitment of a ‘broad and representative’ sample– Maintains appropriate protections for study subjects
8th International Conference on Health Policy Statistics – Washington– January 2010
Cluster Randomized Trials in Health Policy Research
Part Two: Ethics and Implementation of Cluster Randomized Trials
Neal V. Dawson, MD [email protected] Part 2: Page 4
Implementing CRTs: “Observational Studies”
• Some implementation challenges can make CRTs look like observational studies– Political issues– Temporal trendsp
• Challenges regarding – Randomization– Susceptibility– Performance– Detection– Transfer
10
Randomization
• Not all sites will allow randomization• e.g. they ‘need’ to be in one group or
the other
• Sites may need to be randomized • Sites may need to be randomized together
• May be functionally linked• e.g. several clinicians may work
regularly at 2 sites
11
Susceptibility to Outcome at Baseline
• Differential temporal trends• May threaten comparability across
intervention/non-intervention clusters
• e.g. in a study of patient health behaviors, a contract is lost and many patients with private insurance move to another health system leaving a disproportionate number of uninsured patients at one site
12
8th International Conference on Health Policy Statistics – Washington– January 2010
Cluster Randomized Trials in Health Policy Research
Part Two: Ethics and Implementation of Cluster Randomized Trials
Neal V. Dawson, MD [email protected] Part 2: Page 5
Performance
• Intervention fidelity – e.g. after the study starts, a third of
physicians at one intervention site decide th l i h t ti i tthey no longer wish to participate
• Co-interventions– e.g. after the study starts, at only one site a
pharmaceutical company provides at no cost a new efficacious but expensive drug that influences the outcome of interest
13
Detection of the Outcome
• e.g. study of ACE inhibitor use to slow diabetes related renal function decline – after the study starts, one group installs
software that automatically records data that leads to more appropriate performance of the gold standard test; at other sites the recording is dependent on clinicians remembering to order the gold standard test
14
Transfer
• Transfer– Drop outs and crossovers
• Sites
• Subjects within a site
• Statistical comparisons that do not appropriately consider hierarchies and clusters
15
8th International Conference on Health Policy Statistics – Washington– January 2010
Cluster Randomized Trials in Health Policy Research
Part Two: Ethics and Implementation of Cluster Randomized Trials
Neal V. Dawson, MD [email protected] Part 2: Page 6
Summary
• When studying interventions designed to affect groups, individual-level RCTs risk contamination and Type II errors.– CRTs are often preferable for interventions to affect CRTs are often preferable for interventions to affect
behavior
– Clustering – differences in subjects across groups –affects power, design, and analyses of CRTs
– EMRs facilitate CRTs – pre-assignment “balancing” of important prognostic attributes.
• Implementing CRTs – “stuff happens” and agile designs and analytic plans are called for...
16
8th International Conference on Health Policy Statistics – Washington DC– January 2010
Designing Cluster Randomized Trials in Health Policy Research
Part Three: A Brief Overview of Some Analytic and Strategic Concerns
Thomas E. Love, Ph. D. [email protected] Part 3: Page 1
Designing Cluster Randomized Trials in Health Policy Research :
A Few Analytic d St t i C
1
and Strategic Concerns
Thomas E. Love, Ph. D.January 2010
Cluster Randomized Trials?
As compared to individually randomized trials (RCTs),
• CRTs are more complex to design
2
p g
• CRTs require more participants to obtain equivalent statistical power
• CRTs require more complex analysis
Key Concerns when Doing CRTs
• Unit of randomization (assignment) is different than the unit of analysis
• Clustering has design (sample size) and
3
analytic implications– Need larger samples than individual RCT
– Need better pre-trial data for balancing
– Need sophisticated statistical methods
8th International Conference on Health Policy Statistics – Washington DC– January 2010
Designing Cluster Randomized Trials in Health Policy Research
Part Three: A Brief Overview of Some Analytic and Strategic Concerns
Thomas E. Love, Ph. D. [email protected] Part 3: Page 2
The ICC(Intra-Cluster Correlation Coefficient)
Variance in the outcome attributableto differences BETWEEN clusters
TOTAL variance in the outcome=
ICC for an outcome
4
• ICC = 0.01 means 1% of the variance in the outcome is attributable to differences between clusters
• ICC is interpreted as if it were “R2”
Toy Example #1
r 1
Clu
ster
2
n=48, mean = 109.0, sd = 14
n=51, mean = 108.7, sd = 16
Estimated ICC < 0.00001
5
90 100 110 120 130 140 150
Clu
ster
Systolic BP (in mm Hg)
Source SS df MS F PBETWEEN Clusters 1.8 1 1.8 0.01 0.93
WITHIN Clusters 22063 97 227.4
STATA1-way
RandomEffects
LONEWAY
er 1
Clu
ste
r 2
Toy Example #2
n=49, mean = 118.4, sd = 13.2
n=50, mean = 99.4, sd = 9.8
Estimated ICC = 0.569
6
90 100 110 120 130 140 150
Clu
ste
Systolic BP (in mm Hg)
Source SS df MS F PBETWEEN Clusters 8961 1 8961 66.3 < .0001
WITHIN Clusters 13102 97
STATA1-way
RandomEffects
LONEWAY
8th International Conference on Health Policy Statistics – Washington DC– January 2010
Designing Cluster Randomized Trials in Health Policy Research
Part Three: A Brief Overview of Some Analytic and Strategic Concerns
Thomas E. Love, Ph. D. [email protected] Part 3: Page 3
“Basic” Design Effect Formula
• Average Cluster Size: Mean # of subjects per cluster =
• ICC estimate for relevant outcome
n
ICC
7
Design Effect =
ICCn 11
This is the D. E. for continuous data. For comparing two proportions, there isan additional correction due to Cornfeld (1978). See Campbell (2001).
Design Effect
• Design Effect ≥ 1.0
Total subjects requiredunder CLUSTER randomization
Total subjects requiredunder INDIVIDUAL randomization
=
8
g ≥
– If DE > 1.0, CRT requires more patients than would a RCT with the same power.
• For larger ICC, Design Effect increases– CRT requires increasingly larger n relative
to individually randomized trial (RCT).
A Behavioral Intervention in General Practice to Lower Serum Cholesterol
• Practices randomized into two clusters– Intensive dietary intervention vs usual care
– Suppose we recruit 50 patients per practice
Between practice estimate 2 0 046s
9
– Between practice estimate
– Within practice estimate
• ICC estimate for this outcome …
0.0460.0036
0.046 1.28ICC
Kerry SM Bland JM (1998)
0.046Bs 2 1.28Ws
8th International Conference on Health Policy Statistics – Washington DC– January 2010
Designing Cluster Randomized Trials in Health Policy Research
Part Three: A Brief Overview of Some Analytic and Strategic Concerns
Thomas E. Love, Ph. D. [email protected] Part 3: Page 4
Intervention to Lower Serum Cholesterol
• Practices randomized into two clusters– With n = 50 pts / practice, and ICC = 0.0036,
design effect is
1 1 1 50 1 .0036 1.17DE n ICC
10
• Individual RCT requires n = 5,364 pts to detect 0.1 mmol/L difference (90% power and α = 0.05). So the CRT needs…
Kerry SM Bland JM (1998)
1.17 5364 6276 6300 pts
= 126 clusters of 50 patients each
Detecting a 0.10 mmol/L cholesterol Difference (90% power, = 0.05, same ICC)
# needed with individual randomization (i.e. RCT)
5,364 1.00
ICC Pts / Practice Practices PatientsDesignEffect
11
.0036
10 558 5,580 1.04
25 234 5,850 1.09
50 126 6,300 1.17
100 74 7,400 1.38
500 32 16,000 2.98Kerry SM Bland JM (1998)
Are there fancy, free tools to estimate the sample size in CRTs?
• http://www.abdn.ac.uk/hsru/epp/cluster.shtml
– University of Aberdeen sample size
calculator with instructions
12
– Database of ICCs for use in planning
• http://sitemaker.umich.edu/
group-based/optimal_design_software
– “Optimal Design” from U. of Michigan
8th International Conference on Health Policy Statistics – Washington DC– January 2010
Designing Cluster Randomized Trials in Health Policy Research
Part Three: A Brief Overview of Some Analytic and Strategic Concerns
Thomas E. Love, Ph. D. [email protected] Part 3: Page 5
Patients within Physicians within Practiceswithin Study Groups: A 4-Level CRT
13
Common CDS Setting:# of Practices (clusters) is fixed# of MDs/Pts per Practice is variable
Do I Have Enough Practices?
• Many studies: 5-10 practices / arm
– ICC estimates vary quite a bit from study to study - consider 95% CI bounds?
DIG IT bl f ll ICC
14
– DIG-IT: reasonable power for small ICCs• System A: 2 study groups of 5 practices
• System B: 3 groups: 4, 6, and 4 practices
– Design Effect increases with larger practice sizes, if ICC stays constant
Designing a CRT: More Patients Per Practice or More Practices?
# of Practicesper Treatment Arm
ICC(rho)
Required # of Patients (Total)
Needed # of Pts/Practice
40.0
0.01
0.02
200
396
∞
50
99
∞
15
0.03 ∞ ∞
100.0
0.01
0.02
0.03
200
248
320
486
20
25
32
49
200.0
0.01
0.02
0.03
200
220
246
278
10
11
13
14
8th International Conference on Health Policy Statistics – Washington DC– January 2010
Designing Cluster Randomized Trials in Health Policy Research
Part Three: A Brief Overview of Some Analytic and Strategic Concerns
Thomas E. Love, Ph. D. [email protected] Part 3: Page 6
Allocating Practices to Study Groups:The DIG-IT study
• Want minimal differences across study groups on important predictors of:– CDS adoption
16
p– Response to the CDS intervention
• Balancing of practices across study groups (pre-randomization) is critical.– EMR plays a large role here– Most important for small # of practices
EMR-Based Balancing Procedure
• For all feasible clusterings of 10 practices into 2 study groups…– Assemble practice-level clinical and demographic
data from EMR
17
data from EMR
– Identify clusterings which appear to balance an array of baseline characteristics / trends
• (Blinded) consensus as to clustering with best balance. Intervention allocated by coin flip to one study group from that clustering.
Love TE Cebul RD Dawson NV et al. (in press) J Gen Internal Med special issue on Health IT
LDLIncome
VisitsBMI
LDL>=130Hospital12m
ACEARBmalbStatinLDL
CHFCADCVA
RenalPsych
CommInsUnIns
DIG-IT selectionGeographical clustering
Which Table 1 do you want?
18
0 50 100 150 200Absolute Standardized Difference
SlopeA1cA1c>=9A1c<=7LastA1cFemale
Afr-AmerHispanic
InsulinPneumVax
FluVaxSmoker
SBPNoShow
EDAgeLDL
8th International Conference on Health Policy Statistics – Washington DC– January 2010
Designing Cluster Randomized Trials in Health Policy Research
Part Three: A Brief Overview of Some Analytic and Strategic Concerns
Thomas E. Love, Ph. D. [email protected] Part 3: Page 7
Flu Vaccine
Systolic BP
Last A1c <= 7
Last A1c >= 9
Last A1c
Slope A1c
DIG-IT clustering
336 “clusterings” which split 10 practices into 2 study groups of sizes 4 & 6 or 5 & 5
19
0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40
Intra-Cluster Correlation Coefficient
Female
African-American
Hispanic
Current Smoker
Insulin
ED visit
No Show
Pn Vaccine
Pre-Trial Balancing of Practices Using EMRs
• Result: 5 intervention practices and 5 “usual care” practices with excellent balance across study groups.
20
• EMRs provide new opportunities for state-of-the-art study design. – Could use this approach to create study
groups for lots of community-based therapeutic or health care delivery trials.
ANALYSIS: Problems with Individual Level Analysis of Clustered Data
• Lack of independence among members of a cluster (cluster effect)– Need for larger sample sizes: Standard
sample size formulas will lead to
21
sample size formulas will lead to underpowered studies
– Need for sophisticated statistical methods: Standard approaches will tend to bias p-values downward risking spurious claims of statistical significance
Wojdyla D (2005) Cluster Randomized Trials and Equivalence Trials. WHO available online.
8th International Conference on Health Policy Statistics – Washington DC– January 2010
Designing Cluster Randomized Trials in Health Policy Research
Part Three: A Brief Overview of Some Analytic and Strategic Concerns
Thomas E. Love, Ph. D. [email protected] Part 3: Page 8
Analyzing a Cluster Randomized TrialThe Simple Approach
• Construct a summary statistic for each cluster, then analyze these summary values– As in repeated measures or meta-analyses– Simple, but doesn’t allow for covariates
22
• Example: 34 practices referring patients to St George’s Hospital for X-ray exams – 17 practices got nothing, 17 got a one-page
laminated copy of referral guidelines
• Outcome: % of x-rays requested which conformed to the guidelines
Kerry SM Bland JM (1998)
The Wrong Approach• Act as if we had randomly assigned
individual patients to intervention groups• Calculate difference in proportion of requests
in each group that conform.
Group Total Requests % Conforming
23
p q g
Intervention 429 79.5
Control 702 72.5
Difference is 7.0 % points, with SE of 2.6
95% CI is (2, 12) percentage points, P value = 0.0008 (χ2 test)
WRONG!!
This is just wrong, despite what you see in many meta-analyses
X-Ray Requests Conforming to
Guidelines
Two-Sample TMean (%C) Int. = 81.6
Mean (%C) Ctl. = 73.6
SE (diff) = 4.3, 32 df
Intervention Control
ID Req’s % Conf. Req’s % Conf.1 20 100 7 100
2 7 100 37 89
3 16 94 38 84
4 31 90 28 82
5 20 90 20 80
6 24 88 19 79
7 7 86 9 78
24
( ) ,
95% CI: (-1, 17)percentage points
P value = 0.07Weights each practice
equally
8 6 83 25 76
9 30 83 120 75
10 66 80 88 73
11 5 80 22 68
12 43 77 76 68
13 43 74 21 67
14 23 70 126 66
15 64 69 22 64
16 6 67 34 62
17 18 56 10 40
8th International Conference on Health Policy Statistics – Washington DC– January 2010
Designing Cluster Randomized Trials in Health Policy Research
Part Three: A Brief Overview of Some Analytic and Strategic Concerns
Thomas E. Love, Ph. D. [email protected] Part 3: Page 9
X-Ray Requests Conforming to
Guidelines
Two-Sample TEstimated
Mean diff = 7.0SE = 3 3
Intervention Control
ID Req’s % Conf. Req’s % Conf.1 20 100 7 100
2 7 100 37 89
3 16 94 38 84
4 31 90 28 82
5 20 90 20 80
6 24 88 19 79
7 7 86 9 78
25
SE = 3.3
95% CI: (0.2, 14)percentage points
P value = 0.04Weights practices
by # requests
8 6 83 25 76
9 30 83 120 75
10 66 80 88 73
11 5 80 22 68
12 43 77 76 68
13 43 74 21 67
14 23 70 126 66
15 64 69 22 64
16 6 67 34 62
17 18 56 10 40
X-Ray Request ResultsMethod 95% CI P value
WRONG – Treat as if individual RCT
(2, 12) 0.008
CRT – T test (equal practice weights)
(0.2, 14) 0.07
26
• Ignoring clustering results in CIs which are too narrow and P values which are too small.
• Reported ICC in this trial turned out to be 0.019
practice weights)
CRT – T test (weight by # of requests)
(1, 17) 0.04
Kerry SM Bland JM (1998)
Other Analytic Approaches
• Adjust standard errors using the design effect – an approximation
• Robust variance estimates
GEE
27
• GEE
• Multi-level modeling
• Bayesian hierarchical models
• And much, much more …
8th International Conference on Health Policy Statistics – Washington DC– January 2010
Designing Cluster Randomized Trials in Health Policy Research
Part Three: A Brief Overview of Some Analytic and Strategic Concerns
Thomas E. Love, Ph. D. [email protected] Part 3: Page 10
Eight Methods Of Analyzing A CRTCRT of 2 interventions designed to increase
breast screening attendance.Outcome: log (OR) of attendance for two
intervention effects and their interaction
28
Three cluster-level analyses1. Unweighted regression of practice log odds
2. Log odds Regression weighted by inverse variance
3. Random-effects meta-regression of log odds with practice as a random effect
Peters TJ et al. (2003)
Eight Methods Of Analyzing A CRT
CRT of 2 interventions designed to increase
breast screening attendance.
Five individual-level analyses
4 S d d l i i i (i l i )
29
4. Standard logistic regression (ignore clustering)
5. Logistic regression: robust standard errors
6. Generalized Estimating Equations
7. Random-Effects logistic regression
8. Bayesian random-effects logistic regression
Peters TJ et al. (2003)
Results of the Eight Analyses
• [4] was highly anti-conservative (i.e. wrong).
• The other (more valid) methods showed …– Large differences in parameter estimates
– Large differences in standard errors
30
Large differences in standard errors
– Some weren’t computationally stable
– Some were more sensitive than others to between-cluster variation
– GEE doesn’t work well with small # of clusters
• More Guidance Is Needed!
8th International Conference on Health Policy Statistics – Washington DC– January 2010
Designing Cluster Randomized Trials in Health Policy Research
Part Three: A Brief Overview of Some Analytic and Strategic Concerns
Thomas E. Love, Ph. D. [email protected] Part 3: Page 11
Additional Reporting Requirements for CRTs (CONSORT Statement)
1. Rationale for adopting a cluster design
2. How clustering was incorporated into the sample size calculations (include ICCs)
31
3. How clustering effects were incorporated into the analysis
4. The flow of both clusters and individuals through the trial, from assignment to analysis.
Campbell MK et al. for the CONSORT Group (2004)
Repeating Myself:Key Concerns when Doing CRTs
• Unit of randomization (assignment) is different than the unit of analysis
• Clustering has design (sample size) and
32
analytic implications– Need larger samples than individual RCT
– Need better pre-trial data for balancing
– Need sophisticated statistical methods