Using the PCORI Methodology
Standards to Generate Robust,
Relevant, and Timely Evidence for
Patient-Centered Outcomes Research
Joe V. Selby, David Hickam, Brian Mittman, and
Sean Tunis
June 4, 2013
The audio and slide presentation will
be delivered directly to your
computer Speakers or headphones are required to hear the
audio portion of the webinar.
If you do not hear any audio now, check your
computer’s speaker settings and volume.
If you need an alternate method of accessing audio,
please submit a question through the Q&A pod.
Technical Assistance
Live technical assistance:
– Call Adobe Connect at (800) 422-3623
Refer to the ‘Technical Assistance’ box
in the bottom left corner for tips to
resolve common technical difficulties.
To submit a question:
1. Click in the Q&A box on the left side of your screen
2. Type your question into the dialog box and click the Send button
Questions may be submitted at
any time during the presentation
Welcome
Erin Holve, Ph.D., M.P.H.,
M.P.P.
– Senior Director of Research
& Education, AcademyHealth
A Collaborative Effort to
Improve Training in PCOR
PCORI Mini-Course Two morning panels
– Developing a Program of Comparative Effectiveness
Research at the Patient Centered Outcomes Research
Institute” (9:45-11:15am)
• Chair: Joe Selby, PCORI
• Faculty: Chad Boult, Rachael Fleurence, Romana Hasnain-
Wynia, and David Hickam, PCORI
– “PCORI Methods Pilot Projects: Learning from Engagement”
(11:30am-1:00pm)
• Chair: Sue Sheridan, PCORI
• Faculty: Katherne Bevans, Children’s Hospital of Philadelphia;
Renee Robinson, Southcentral Foundation; and Tiffany
Haynes, University of Arkansas for Medical Sciences
PCORI Mini-Course Afternoon workshop (June 25, 2:00-6:00pm ET)
– Heterogeneity of Treatment Effects
• Chair: Brian Mittman, PhD, Department of Veterans Affairs
• Faculty: Ravi Varadhan, PhD, Jodi Segal, MD, MPH, Johns Hopkins
University School of Medicine
– Adaptive and Bayesian Trial Designs
• Chair: Sally Morton, PhD, University of Pittsburgh
• Faculty: Jason Connor, PhD, Berry Consultants
– Data Registries
• Chair: Robin Newhouse, PhD, RN, NEA-BC, FAAN, University of
Maryland
• Faculty: Priscilla Velentgas, Ph.D., Quintiles Outcome
Learning Objectives Provide an overview of the PCORI Methodology
Standards and the process used to develop the
standards;
Provide general guidance on using the standards
when responding to PCORI funding announcements;
and
Review plans for implementing the standards and
updating them in the future.
Webinar Faculty
Joe V. Selby, MD,
MPH
David Hickam, MD,
MPH Brian Mittman,
PhD Sean Tunis, MD,
MSc
Academy Health Webinar June 4, 2013
Joe V. Selby Executive Director, PCORI
Using the PCORI Methodology Standards to Generate Robust, Relevant, and Timely Evidence for Patient-Centered Outcomes Research
PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
About PCORI
• An independent, non-profit organization authorized by Congress.
• Mission is to fund research that will provide patients, their caregivers and clinicians with the evidence-based information needed to make better-informed health care decisions.
• Committed to continuously seeking input from patients and a broad range of stakeholders to guide its work.
PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
What is Patient Centered Outcomes Research?
Patient-Centered Outcomes Research (PCOR) helps people and their caregivers communicate and make informed health care decisions, allowing their voices to be heard in assessing the value of health care options.
PCOR has the following characteristics:
• Actively engages patients and key stakeholders throughout the research process.
• Compares important clinical management options.
• Evaluates the outcomes that are most important to patients.
• Addresses implementation of the research finings in clinical care environments.
PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
Approximate PCORI Funding
* 20% of this amount goes to AHRQ/HHS each year for dissemination and capacity-building
PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
Current Funding Opportunities and Events
http://www.pcori.org/funding-opportunities/funding-announcements/funding-center/
Announcement Letter of Intent Due
Assessment of Prevention, Diagnosis, and Treatment Options 06/15/13
Improving Healthcare Systems 06/15/13
Communication and Dissemination Research 06/15/13
Addressing Disparities 06/15/13
Improving Methods for Conducting Patient-Centered Outcomes Research
06/15/13
Improving Infrastructure for Conducting Patient Centered Outcomes Research: Phase One —Clinical Data Research Networks
06/19/13
Improving Infrastructure for Conducting Patient Centered Outcomes Research: Phase One —Patient-Powered Research Networks
06/19/13
PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
David Hickam, MD, MPH, PCORI
PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
Developing the PCORI Methodology Standards
http://www.pcori.org/assets/PCORI-Authorizing-Legislation-032310.pdf
• Congressional Requirements- Patient Protection and Affordable Care Act, Subtitle D, Paragraph (6)(C)(i)
PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
PCORI Methodology Committee
http://www.pcori.org/about/governance-and-leadership/leadership/methodology-committee/
• Sherine Gabriel, MD, MSc (chair), Mayo Clinic
• Robin Newhouse, PhD, RN (vice chair), University of Maryland
• Naomi Aronson, PhD, BlueCross BlueShield
• Ethan Basch, MD, MSc, Memorial Sloan-Kettering Cancer Center
• Alfred O. Berg, MD, MPH, University of Washington
• David Flum, MD, MPH, University of Washington
• Steven Goodman, MD, MHS, PhD, Johns Hopkins School of Public Health
• Mark Helfand, MD, MS, MPH, Dept. of Veterans Affairs
• John Ioannidis, MD, DSc, Stanford University
• Michael S Lauer, MD, National Heart, Lung, and Blood Institute
• David O. Meltzer MD, PhD, University of Chicago
• Brian S. Mittman, PhD, Dept. of Veterans Affairs
• Sebastian Schneeweiss, MD, ScD, Harvard Medical School
• Jean R. Slutsky, Agency for Healthcare Research and Quality
• Mary Tinetti, MD, Yale University School of Medicine
• Clyde Yancy, MD, MSc, Northwestern University Feinberg School of Medicine
PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
PCORI Methodology Standards
http://www.pcori.org/research-we-support/methodology/
PCORI Methodology
Committee Formed (2011)
Draft Methodology
Report Posted (July
2012)
Public Comment
Period (July-September
2012)
Proposed Revisions to
Methodology Standards
(November 2012)
Revised PCORI
Methodology Standards
(December 2012)
Final Methodology
Report (Spring 2013)
PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
11 Categories of Methodology Standards 47 Individual Standards
*Bold indicates topics to be discussed in detail at the June 25 in-person workshop at the AcademyHealth Annual Research Meeting
• Formulating Research Questions
• Patient-Centeredness
• Systematic Reviews
• General and Cross-cutting Methods for all PCOR
• Causal Inference
• Heterogeneity of Treatment Effects
• Preventing/Handling Missing Data
• Data Networks
• Adaptive and Bayesian Trial Designs
• Data Registries
• Studies of Diagnostic Tests
PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
Characteristics of the Methodology Standards
• Are minimal standards for performing comparative effectiveness research.
• Are intended to provide helpful guidance to researchers and those who use research results.
• Reflect generally accepted best practices.
• Provide guidance for both project protocols and reporting of results.
• Are used to assess the scientific rigor of funding applications.
• Context of research should drive use of the standards.
PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
Standards that Apply to Most Projects
• Formulating research questions (6 standards)
• Patient-centeredness (4 standards)
• General and cross-cutting (6 standards)
• Missing data (5 standards)
PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
Formulating Research Questions
Identify affected populations and health decision(s), and how study results will inform the health decision (RQ-3)
Identify and assess participant subgroups (RQ-4)
Select appropriate interventions and comparators and make explicit how each were selected (RQ-5)
PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
Patient-Centeredness
Engage the population of interest and other relevant stakeholders
as appropriate and necessary in a given research context (PC-1)
Identify, select, recruit, and retain study participants representative of the population of interest (PC-2)
Use patient-reported outcomes when patients or people at risk of a condition are the best sources of information. (PC-3)
Support dissemination and implementation of study results (PC-4)
PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
General and Cross-cutting Methods
• Assess the suitability of the data source (GM-1)
• Document data analysis plans for major aims (GM-3)
• Document validated scales and tests
• Include characteristics of the scales and psychometric properties (GM-4)
PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
Missing Data
• Use Validated Methods to Deal with Missing Data that Properly Account for Statistical Uncertainty Due to Missingness (MD-3)
• Record and Report All Reasons for Dropout and Missing Data, and Account for All Patients in Reports (MD-4)
PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
Standards that Apply to Specific Data Sources and Study Designs
• Systematic reviews (1 standard)
• Causal inference (6 standards)
• Heterogeneity of treatment effects (4 standards)
• Data networks (2 standards)
• Adaptive trials (5 standards)
• Registries (3 standards)
• Diagnostic tests (5 standards)
PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
Causal Inference
• Precisely Define the Timing of the Outcome Assessment Relative to the Initiation and Duration of Exposure (CI-3)
• Measure Confounders before Start of Exposure. Report data on confounders with study results (CI-4)
PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
Diagnostic Tests
• Assess the Effect of Factors Known to Affect Diagnostic Performance and Outcomes (DT-3)
PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
Brian Mittman, PhD, Department of Veterans Affairs
PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
How to Use the Standards
• The standards represent “best practice” recommendations for improving the quality and value of PCOR and CER.
• The standards aim to achieve greater transparency, rigor and other improvements to strengthen the science and methods of PCOR and CER.
PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
Implementing the standards: improving PCOR research practices • “Research practices” include decisions and actions by numerous
stakeholders (funders, reviewers, researchers, etc.)
• Changing these practices is challenging
– Researchers value independence and autonomy
– Researcher actions are guided by knowledge, experience, judgment
• Analogue: the challenge of improving research practices through standards is comparable to the challenge of improving clinical practices through evidence-based clinical practice guidelines.
PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
Implementing research standards
• Professional practices are influenced by multiple factors operating at
multiple levels
• Implementation gaps must be thoroughly diagnosed to identify “root causes”
• Implementation campaigns must include multiple, coordinated components targeting the full spectrum of determinants of gaps
• Influences on research practices – and points of leverage to change practices – should be depicted in a causal, conceptual model
There are “no magic bullets”: examples of dramatic change in research practices (e.g., trial registration) are rare and exceptional.
PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
Conceptual model of the research process
• Research question identification and prioritization (researchers, funding agencies)
• Study design and methods selection (researcher judgment guided by training, professional norms, published standards)
• Peer review (reviewer judgment guided by training, professional norms, published standards)
• Study governance and conduct (researcher judgment, possibly guided by reporting requirements, IRB review, DSMB review)
• Documentation and reporting (researcher judgment guided by reporting guidelines, journal requirements, peer review)
PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
Dissemination and Future Directions
• Initial set of standards released in December 2012.
• PCORI Methodology Report will be released in Summer 2013.
• No change in the set of standards.
• Provides further guidance on interpretation and use.
• Responding to future PCORI Funding Announcements (PFAs).
• All future submissions must demonstrate adherence to/adoption of/consideration of the standards.
• PCORI will continue to develop tools and resources for researchers, reviewers and other stakeholders.
PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
Sean Tunis MD, MSc - CMTP
PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
Purpose of Standards: PMC draft report 7/12
“Departures from basic good research practices are partially responsible for the mismatch between the quality and relevance of the information research provides and the information patients need to make informed clinical decisions.”
“The initial range of topics was chosen to reflect areas in which the Committee believed that there were either substantial deficiencies or inconsistencies in how available methods were applied in practice, or for which there was specialized knowledge in how best to conduct research that had not been effectively disseminated.”
PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
Required by Law! (ACA)
“Methodological standards shall provide specific criteria for”:
• Internal validity (robust)
• Generalizability (relevant)
• Timeliness (timely)
• Feasibility
PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
Where to Focus
• Craft an important, relevant research question
• Informed by gaps in systematic reviews and meaningful input from patients and other decision makers
• Patient-centered care requires attention to subgroups and PROs
• Apply best practices in application of selected methods
• Provide full transparency
Clear, detailed protocol – explain deviations from standards
Follow existing reporting standards
PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
Asking the Right Question
“To produce information that is meaningful and useful to people when making specific health decisions, research proposals and protocols should describe: 1) the specific health decision the research is intended to inform; 2) the specific population for whom the health decision is pertinent; and 3) how study results will inform the health decision.”
• Requires dialogue with decision makers (more is better)
• Methods should serve research question; not vice versa
PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
Use of Patient Reported Outcomes
“Select outcomes based on input directly elicited from patient informants, people representative of the population of interest, either in previous studies or in the proposed research.”
• Use existing core outcomes measures sets when available
• COMET (Core Outcomes Measures in Effectiveness Trials)
• Effectiveness Guidance Documents (Green Park Collaborative)
PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
Feasibility: An Unresolved Issue
Standard for Studies of Dx Tests:
“Studies of clinical outcomes after diagnostic testing should use a prospective randomized study design when possible. If a non-randomized design is proposed, the reason for using an observational study (or modeling and simulation) should be addressed and efforts to minimize confounding documented.”
PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE
Looking Forward
• Specifics of implementation of standards not yet clear
• Initial emphasis of methods standards on patients and caregivers as decision makers. Intent to expand scope to clinicians, payers/purchasers, policy makers, others.
• PCORI thinking continues to evolve based on experience and public feedback from researchers and other stakeholders
• How best to achieve optimal balance of internal validity, generalizability, timeliness and feasibility remains an important challenge
To submit a question:
1. Click in the Q&A box on the left side of your screen
2. Type your question into the dialog box and click the Send button
Submitting Questions
Expert Q&A
Erin Holve (facilitator)
Dave Hickam
Brian Mittman
Sean Tunis
Still time to sign up for the PCORI Mini-Course
• In-person events at AcademyHealth’s Annual Research Meeting in Baltimore, MD (June 25)
Two morning panels
• “Developing a Program of Comparative Effectiveness Research at the Patient Centered
Outcomes Research Institute” (9:45-11:15am)
• “PCORI Methods Pilot Projects: Learning from Engagement” (11:30am-1:00pm)
• Afternoon workshop (June 25, 2:00-6:00pm ET)
• Heterogeneity of Treatment Effects
• Chair: Brian Mittman, PhD, Department of Veterans Affairs
• Faculty: Ravi Varadhan, PhD, Jodi Segal, MD, MPH, Johns Hopkins University School of
Medicine
• Adaptive and Bayesian Trial Designs
• Chair: Sally Morton, PhD, University of Pittsburgh
• Faculty: Jason Connor, PhD, Berry Consultants
• Data Registries
• Chair: Robin Newhouse, PhD, RN, NEA-BC, FAAN, University of Maryland
• Faculty: Priscilla Velentgas, Ph.D., Quintiles Outcome
• Click here to register: $50 for AcademyHealth members, $100 for nonmembers
Thank You
Please take a moment to fill out the
brief evaluation which will appear in your browser.