PCORI Advisory Panels Kickoff & Training

Identifying and Prioritizing

Patient-Centered Research

Questions: Software Training

Kara Odom Walker, MD, MPH, MSHS

Program Officer, Improving Healthcare Systems

April 19, 2013

1

Agenda

Background

Decision Analysis Web Tool

Web Survey Tool

2

What is ExpertChoice?

3

ExpertChoice is a decision analysis software that is based

on multi-criteria decision making that uses mathematical

theory called analytic hierarchicial process

Created in 1983 by Thomas Saaty and Ernest Forman

Serves over 100 Fortune 500 companies such as IBM, used

by 30 Federal Agencies, and is taught in over 100

universities and used in 60 countries worldwide

Analytic Hierarchy Process

Information is broken into a hierarchy of

alternatives and criteria

Information is then synthesized to determine

relative ranking of alternatives

Both qualitative and quantitative information can be

compared using informed judgments to derive

weights and priorities

4

Hierarchical Tree Example

5

Style Reliability Fuel Economy

Selecting

a New Car

- Civic

- Saturn

- Prius

- Ferrari

- Civic

- Saturn

- Prius

- Ferrari

- Civic

- Saturn

- Prius

- Ferrari

• Hierarchy corresponds to

decision maker values

• No right answer

• Group decision must be

negotiated

We’ve made it simple!

6

Software for Ranking Activity:

Expert Choice

7

You’ll receive a personal link via e-mail which you will retrieve at

the time your panel decides to begin the ranking exercise

Please use Internet Explorer or Safari browsers if possible;

Mozilla Firefox is the best alternate

You will score each topic based on the 5 criteria described earlier,

using a 5 (high) to 1 (low) scale

Scale

8

Intensity of Importance Definition

1 Low Importance

2 Moderate Importance

3 Strong Importance

4 Very Strong Importance

5 Extreme Importance

ExpertChoice Welcome Screen

9

If a Login Screen Appears…

10

Using the Topic Briefs in the Software

12


Accessing Notes on the Criteria

13

Accessing Notes on the Criteria


15

Rating Topic Briefs Using the Criteria


17

When You Make it Through the Scoring..

18

These scores were inserted at random for training purposes

only and do not in any way reflect PCORI’s preference.

Need to Go Back?

19

Your Screen May Appear Like This..

20

Your Ranked List in Priority Order

21

Expected Time Commitment

22

Online tools should be easy to use, each program

area has adopted a different approach for how and

when you will complete ranking exercises

Software experts will be floating throughout the day

After all panelists in your group have completed the

exercise, PCORI program staff will generate the

overall list for use in the discussion

Final Ranking

Discussion about important features of each

potential research topic will guide the ranking

All of the discussion is used as important input

We want to document the groups final ranking

using a consensus building approach

23

Final Ranking Tool: Survey Gizmo

24

Link to Online Survey

25

Rank your list: from 1 (your first choice topic) to 5

(or more—depending on how many topics your

group decides—for your lowest)

Your group facilitator will provide the link and final

number of topics

Final Results:

Combination of All Participant Rankings

Topic #1

Topic #2

Topic #3

Topic #4

… and more

26

Thank You for Your Time and Support of

This Important Process

27

Q&A Session Anne Beal, MD, MPH Deputy Executive Director, Chief Operating Officer and Chief Officer for Engagement, PCORI Advisory Panel Kickoff and Training April 19, 2013

1

Wrap-Up and Announcements Anne Beal, MD, MPH Deputy Executive Director, Chief Operating Officer and Chief Officer for Engagement, PCORI Advisory Panel Kickoff and Training April 19, 2013

2

Agenda

!  The remainder of the event will take place in breakout sessions by panel

!  Room Assignments:   Advisory Panel on Addressing Disparities (Hickory)   Advisory Panel on Improving Healthcare Systems (Chestnut)   Advisory Panel on Assessment of Prevention, Diagnosis, and

Treatment Options (Poplar)   Advisory Panel on Patient Engagement (Walnut)

3

Consent Notice

!  Please keep in mind that the breakout sessions will be recorded and open to the public

4

Administrative Items

Please note that the COI form will be disclosed on our website !   Complete and submit the following forms at the registration desk:

  Audio/Visual Release Form   PCORI Conflict of Interest (COI) Form

!   Panelist will receive a $1,500 stipend, and Chairpersons will receive $2,000 stipend

!   Complete all forms for payment. Finance team members are located in the Dogwood room (downstairs)

5

Payment Schedule

May 2013 October 2013

$750 $750 Panelist $1,250 Chair

Appointing Panel Chairpersons

The chairperson may assemble subcommittees composed of members to examine special issues and facilitate activities

related to the scope of work in a panel’s charter.

!   Nominations and self nominations may be submitted to [email protected] by 5:00 PM (ET) Monday, April 22

!   Identify the name of the nominee, panel, and rationale in one brief paragraph no longer than 500 words

!   Staff will review nominations and make recommendations !   The PCORI Board of Governors will select a chairperson to

facilitate panel activities in conjunction with PCORI’s designated staff leader

6

Future Meetings

!   Advisory Panels will convene no more than four times a year, with the option of monthly conference calls

!   The Chairperson will play a key role in scheduling future meetings

!   Program Directors will discuss future meeting dates during the breakout sessions

!   Proposed dates for face-to-face meetings:   September 20–21, 2013   December 6–7, 2013   March 14–15, 2014

7

Advisory Panel Group Photographs

!  We will be taking group photos of each Advisory Panel   Immediately after this session (12:00 PM):

•  Advisory Panel on Improving Healthcare Systems •  Advisory Panel on Assessment of Prevention, Diagnosis, and Treatment Options

  After lunch (12:50 PM): •  Advisory Panel on Patient Engagement •  Advisory Panel on Addressing Disparities

8

Thank you and enjoy!

9

Improving Healthcare Systems Program

Advisory Panel Meeting April 19-20, 2013

1

Chad Boult, MD, MPH, MBA Director

PCORI’s Mission

!  Affordable Care Act (ACA) 2010 says PCORI shall fund research that:   Discovers new information   Allows patients/families/clinicians and health system

leaders to make better decisions between alternative approaches

  Leads to better patient-centered outcomes of care

2

Engagement

!  The ACA also says that PCORI’s research must be conducted by scientists who are engaged with patients and stakeholders (such as clinicians, executives of provider organizations, administrators of insurance companies, and representatives of manufacturers)

!  Significant engagement extends throughout the research process

3

National Priorities for Research and Research Agenda

• Comparisons of alternative clinical options to support personalized decision making and self-care • Identifying patient differences in response to therapy • Studies of patient preferences for various outcomes

• Improving support of patient self-management • Focusing on coordination of care for complex conditions and improving access to care • Comparing alternative strategies for workforce deployment

• Understanding and enhancing shared decision making • Alternative strategies for dissemination of evidence • Exploring opportunities to improve patient health literacy

• Understanding differences in effectiveness across groups • Understanding differences in preferences across groups • Reducing disparities through use of findings from PCOR

• Improving study designs and analytic methods of PCOR • Building and improving clinical data networks • Methods for training researchers and patients to participate in PCOR • Establishing methodology for the study of rare diseases

4

What Is a Healthcare System?

!   Through the patient’s lens—all the providers of the care I receive, as well as my insurer

!   Through the provider’s lens—all the providers with whom I interact, as well as the insurers

!   Through the insurer’s lens—all the providers I pay to care for my insured lives

!   Through the economist’s lens—all the patients, providers, and insurers in a geographic area

!   Through your lens?

5

How Can Healthcare Systems Be Improved?

!  Changes in the deployment of personnel !  Changes in the use of information !  Changes in operating policies !  Changes in patients’ behaviors !  Changes in payment policies !  Changes in linkages to community agencies !  Other changes?

6

Vision

PCORI’s IHS program will produce new scientific information to: !  Transform the quality and efficiency of

critical components of the US healthcare system

!  Improve health-related outcomes that matter most to Americans

7

How Can Patients Benefit?

!  Processes—Engagement in self-care, coordination of care, improved access to care, better quality of care

!  Outcomes—Improved quality of life, greater satisfaction with care, greater functionality in life roles

8

IHS’s Traditional Path for Selecting Topics for Research Contracts

9

IHS’s Novel Path for Selecting Topics for Research Contracts

10

IHS Method

!  PCORI’s IHS program enters into contracts under which selected organizations perform specified health systems–related research, and PCORI pays them an agreed amount

11

The Roles of Patients and Stakeholders on Research Teams

Engaged patients and stakeholders:

  Participate in the formulation of the research questions to be answered

  Help define essential characteristics of study participants, the comparators, and the outcomes to be measured

  Help monitor the conduct and the progress of the study

  Help disseminate the study’s results

12

Patient-Centered Outcomes

!  Patient-centered outcomes (PCOs) are outcomes people care about, for example:   Symptoms   Unwanted events   Health   Quality of life   Function   Safety   Survival

13

What Kinds of Features Should Be Studied?

!  Deployment of personnel !   Information technology (IT) !   Incentives !  Special programs, such as navigators who link

medical care to community services that promote patient self-management

!  Others?

14

Topic Briefs

15

Outline for Our Topic Brief Discussion

!  Primary discussant(s) briefly describes the topic to orient the panel ~2 minutes

!  Secondary discussant(s) can add any other information ~1 minute

16

Review of Agenda

!  2:00-3:30 PM: Topics 1–7 !  3:30 BREAK !  3:45-5:15 PM: Topics 8–15

17

Rating Topics Using Criteria

18

Highest (Meets criteria)

Lowest (Does not meet criteria)

Criteria for Research Prioritization

!   Patient-centeredness   Is the research of specific interest to patients and caregivers?

!   Impact of the condition on individual/population health   Prevalence, incidence, morbidity, productivity, mortality

!   Options for addressing the issue   What could new research contribute toward patient-centered

outcomes? !   Likelihood of implementing research results into

practice   How likely are study findings to change clinical practice?

!   Durability of information   How long will the information resulting from this research be

valuable?

19

#1 Compared to usual care, what are the effects of accountable care organization care on patient-centered outcomes among patients with chronic conditions?

20

Introduction: •  The accountable care organization (ACO) model is patient-centered, with quality measures falling into four domains outlined by CMS: patient experience, care coordination and patient safety, preventive health services, and attention to at-risk populations

•  ACOs are designed to change practice patterns by aligning incentives

Burden: •  Chronic conditions affect >50% of US population •  Accounts for >75% of healthcare spending •  Seven out of 10 deaths in the US are related to chronic disease •  Prevalence of chronic conditions in younger people has increased •  Chronic conditions account for disability and diminution in quality of life

Options for addressing issue:

•  ACOs •  There are no guidelines or systematic reviews that address the impact of

ACOs on patient-centered outcomes (PCOs) •  The steps each is taking to implement an accountable care model are varied

Potential for new information to improve care and PCOs rapidly:

•  Identifying best practices for assigning patients to ACOs •  Developing measures to assess ACOs with respect to PCOs •  Creating a taxonomy of implementation strategies and critically assessing the

merits of each

#2 What are the relative effects of different models of chronic care on PCOs?

21

Introduction: •  Much of the occurrence and many of the complications of chronic diseases are also preventable

•  Failures of the disease management approach have led to an increased focus on patient-centered management, which includes individualized treatment

Burden: •  Chronic conditions affect >50% of US population •  Account for >75% of healthcare spending •  Seven out of 10 deaths in the US are related to chronic disease •  Prevalence of chronic conditions in younger people has increased •  Chronic conditions account for disability and diminution in quality of life


•  Traditional medical model: disease management focusing on single condition •  Newer models have focused on individual patients and their multiple needs •  Outcomes that extend beyond clinical outcomes •  A multidisciplinary team that extends beyond medical care


•  Identifying best models and the relative effects of those models on PCOs •  Identifying which models maximize PCOs in various patient groups •  Identifying elements of a model that make it more or less successful •  Identifying the appropriate PCOs being evaluated •  Target model selection to the correct patients to maximize PCOs

#3 Compared to usual care, what is the effect of care management (designed to optimize care coordination and continuity) on PCOs among patients with COPD?

22

Introduction: •  COPD includes chronic bronchitis and emphysema •  Patients have diverse needs arising from the disease itself •  Practice guidelines offer little guidance for these patients with complex and

variable needs

Burden: •  >12 million Americans affected •  Fourth leading cause of morbidity and mortality in the United States •  Direct cost estimated to be $30 billion in United States and indirect $20 billion •  Evidence shows practitioners adhere poorly to guideline recommendations


•  >40 guidelines released in last five years focusing on diagnostic strategies and treatment recommendations

•  The failure of traditional approaches suggests the need for management strategies that are patient-centered and include individualized treatment


•  Benefits of specialists, co-management, or referral for different patient groups •  Structure of provider teams •  How to structure care management in solo or small practices •  Care management to reduce ED usage, hospitalizations, and readmissions

#4 Compared to usual care, what is the effect of care management (designed to optimize care coordination and continuity) on PCOs among patients with cancer?

23

Introduction: •  Common approach is based in clinical practice guidelines, focusing on diagnostic criteria and treatment recommendations

•  Treatment must also manage the patient’s individual needs arising from associated comorbidities and treatment complications

Burden: •  Cancer affects >1 million Americans •  Cancer is the second leading cause of mortality in the United States •  Direct costs of cancer were $124 billion in 2010 •  Incidence rates have remained stable or risen slightly in the last 10 years


•  Hundreds of cancer guidelines have been issued in the last five years, focusing on diagnosis and treatment

•  Guidelines on screening and early diagnosis exist for some cancers; however, adherence is low

•  Survivorship care plans exist to address the needs of cancer survivors, but lack evidence on whether they improve patient outcomes


•  Impact of new developments in diagnostics and treatment on patients •  How to facilitate the transition to life after cancer •  Little research has been done on fear of recurrence

#5 Compared to usual care, what is the effect of care management (designed to optimize care coordination and continuity) on PCOs among patients requiring palliative care?

24

Introduction: •  Palliative care is patient- and family-centered care with the goal of optimizing quality of life, by focusing on pain and symptom management, communication about goals and care planning, and psychosocial and spiritual support

•  PCOs may extend to family-caregiver–centered outcomes

Burden: •  Despite the benefits, hospice care usually does not occur in the United States •  As much as 1/3 of healthcare utilization occurs at the end of life •  Disparities exist related to race and income in areas such as pain

management, communication, and use of hospice care


•  Care management is one type of palliative care intervention •  Moderate evidence for the effectiveness of palliative care interventions, but

results vary depending on the outcomes, population, and settings •  Limited evidence for other types of interventions, such as advance care

planning, quality improvement, and policy initiatives


•  Larger, more inclusive, better quality studies that include a comprehensive range of patient populations, types of conditions, and outcomes to target and prioritize outcomes, increase access to care, and reduce disparities

•  Impact of improving communication about goals and care planning in settings other than intensive care units

#6 Compared to usual care, what is the effect of care management (designed to optimize care coordination and continuity) on PCOs among pregnant women?

25

Introduction: •  Care management models include expanding the scope of prenatal care beyond prevention of adverse outcomes to include emphasis on quality of care, continuity of care, maternal education, and social support

•  These models generally rely on the addition of nurses and other trained personnel to the care team

Burden: •  Maternal mortality has increased in recent years (13 deaths/100,000 births) •  Adverse infant outcomes may lead to long-term health consequences and

may pose a financial and emotional burden to caregivers


•  Standard care: aims to optimize medical outcomes by providing regular screening and medical care

•  Care management: seeks greater continuity, more communication, and more pregnancy education, while minimizing delays in screening and treatment

•  Care management models are generally individual to institutions and difficult to generalize


•  Determine the impact of care management on clinical and patient-centered outcomes

•  Identify which models deliver the best outcomes •  Determine whether care management helps with the early identification and

care for high-risk pregnancies

#7 Compared to usual care, what is the effect of care management (designed to optimize care coordination and continuity) on PCOs among patients with multiple chronic conditions?

26

Introduction: •  Patients with two or more chronic conditions are said to have multimorbidity or multiple chronic conditions (MCCs)

Burden: •  75 million people have MCCs •  MCC patients are more likely to take multiple medications, have a higher rate

of adverse events, more psychological distress, more rates of disability, and poorer quality of life than the rest of the population


•  There are no standard management options for people with MCCs •  Application of multiple single-condition clinical practice guidelines (common,

but not optimal) •  Individualized treatment plans, focused on the patient and his or her needs,

with the patient an active participant in his or her own care •  Effects of care management on PCOs for MCCs are not well understood


•  Optimal mix of providers structure of the teams providing care management •  Optimal frequencies and modalities of interaction with care management team •  Which PCOs should be targeted for improvement with care management •  Determining whether approaches to care management need to be modified

based on the intended outcomes

BREAK

27

#8 Compared to care management provided by insurance companies, what is the effect of care management provided by medical homes on PCOs among patients with MCCs?

28

Introduction: •  Care management includes coordination of care and educational activities to help people with chronic conditions understand their condition and to achieve optimal health and quality of life

Burden: •  75 million people have MCCs •  Patients with MCCs are more likely to take multiple medications, have a

higher rate of adverse events, more psychological distress, more rates of disability, and poorer quality of life than the rest of the population


•  Patient-centered medical homes (PCMHs): provide care management with a primary care physician as the leader of patient care and care coordination activities

•  Care management by insurance companies: utilizes non-physician personnel, generally offsite, to coordinate care

•  Insufficient evidence to compare clinical outcomes of care coordination in patients with MCCs in medical homes versus insurance companies


•  Effect of PCMHs on PCOs for patients with MCCs •  Implementation methods for PCMHs •  Identifying elements that have significant effect on outcomes on patients with

MCCs

#9 Compared to usual care, what is the effect of care from a non-physician PCMH on care quality and PCOs?

29

Introduction: •  PCMHs may be directed and staffed by providers other than physicians, such as physician assistants (PAs), nurse practitioners (NPs), nurses, or other specially trained staff

Burden: •  ~1/3 of current physicians practice primary care, but only 1/4 of current medical school graduates plan on careers in primary care

•  Within 10 years, the deficit of primary care physicians will be 40,000


•  The American College of Family Physicians has stated that PAs should be recognized as primary care providers in the PCMH model

•  There is speculation that NPs could also fulfill roles as primary care providers •  No published trials have evaluated PCMHs led by PAs, NPs, nurses, or other

non-physicians


•  Evaluate the effectiveness of PCMHs led and staffed by PAs and NPs on care quality and PCOs

•  Identify elements required for rapid uptake of this model

#10 Compared to primary care alone, what is the effect of primary care co-located with mental health services on mental health symptoms, medication use, and other PCOs?

30

Introduction: •  WHO recommends integrating mental health care with primary care services and the promotion of mental health along with general health

Burden: •  Half of all Americans will develop some mental illness in their lifetime •  17% of US adults have both a physical and mental health condition •  Mental illness is associated with diminished well-being, unhealthy behaviors,

and reduced overall quality of life •  Health disparities tend to be exacerbated for mental health care due to access

and other social factors


•  Collaborative care: includes mental health services in the primary care setting •  Some evidence for improvement on specific outcomes, such as the mental

condition itself, patient satisfaction, and quality of life •  Lacking evidence on outcomes such as medication adherence, self-

management, symptom burden, and clinical outcomes relevant to the medical condition


•  Comparison of different models: treatment by primary care physician, co-managed, or referred

•  Identifying the effect on a comprehensive range of patient outcomes •  Optimal structure of provider teams •  Identifying whether mental health disease also increases risk for other

medical problems

#11 Compared to direct transportation to a regional trauma center, what is the effect of stabilization at a local hospital (followed by transfer to a regional trauma center) on survival and other PCOs?

31

Introduction: •  A trauma center is a hospital that has resources and equipment needed to care for severely injured patients

•  Local hospitals can provide initial assessment and treatment of trauma but are not designated as trauma centers if they do not have trauma surgeons and other necessary hospital resources

Burden: •  Injuries are the leading cause of death for children and adults ages 1 to 44 •  Nearly one in five (45 million) Americans live in regions without access to a

Level I or II trauma center within one hour of where they are injured


•  Trauma systems: includes both trauma centers and non-trauma centers for the care of injured individuals in a region

•  Inclusive system: all hospitals within a region participate in the trauma system •  Exclusive system: all injured patients are preferentially sent to the few trauma

centers


•  Optimal triage and management of patients within trauma systems •  Effect of different triage models on PCOs

#12 Compared to usual care, what is the effect of information technology (e.g., EHRs, PHRs, and decision support) on providers’ compliance with guidelines and chronically ill patients’ adherence to treatment plans?

32

Introduction: •  Clinical decision support (CDS) systems enhance compliance to clinical practice guidelines (CPGs) by providers and increase adherence to treatment by patients

•  ACA provides large incentives for adoption of CDS-enabled electronic health records (EHRs) and provides patient health records (PHRs) to patients

Burden: •  <50% of Americans receive the recommended treatment based on CPGs •  Due to poor communication across systems and providers, patients with

chronic conditions, especially those with multiple conditions, are at risk of having duplicated tests and more adverse events


•  Health information technology (HIT) solutions (e.g., EHRs and PHRs) that are CDS-linked

•  Patient-focused guidelines directly implemented in the PHR


•  New HIT tools (e.g., CDS) •  Strategies for implementing CDS-linked HIT systems •  Proper application of CPGs in planning a treatment while considering patient-

centered preferences and outcomes •  Effect of HIT tools on provider compliance and patient adherence

#13 What are the relative effects of different quality improvement strategies on the quality of preventive services, acute care, chronic care, and rehabilitative services—and on PCOs—for adults and children?

33

Introduction: •  According to the Institute of Medicine, high-quality care is care that is safe, effective, patient-centered, timely, efficient, and equitable

•  Applications of quality improvement (QI) have spanned all components of care, including preventive, acute, chronic, and rehabilitative services

Burden: •  ~1/2 of patients receive suboptimal or unsatisfactory care •  An estimated 1/3 of total healthcare expenditures in the United States

represent waste •  Deficiencies in the quality of care lead to excess morbidity and mortality


•  Good evidence for QI strategies to address preventive and chronic care •  Some evidence for a variety of different multimodal interventions for specific

acute conditions in different contexts •  Insufficient data for strategies for QI efforts for rehabilitative services


•  Research on the effectiveness of QI for acute and rehabilitative care •  Rigorous evaluation of QI efforts (e.g., using randomized designs,

comprehensive outcomes, and generalizable populations) •  Implementation strategies (including developing local leadership and capacity

for management and measurement)

#14 What are the relative effects of different insurance features (e.g., benefit designs, utilization management, cost sharing) on chronically ill patients’ access to care, quality of care, and PCOs?

34

Introduction: •  Fee-for-service payment models are often costly for both the insurer and the patient, and they can reward the provision of unnecessary care

•  Recently, new insurance options have been developed, and their benefits and risks for the chronically ill remain to be seen

Burden: •  Chronic conditions affect >50% of US population •  Accounts for >75% of healthcare spending •  Nearly 2/3 of Medicare beneficiaries have two or more chronic conditions •  Copays and premiums are often unaffordable for chronically ill individuals,

and medical bills contribute importantly to personal debt and bankruptcy


•  Alternatives to fee-for-service insurance designs •  Value-based insurance design: co-payment inversely related to proven benefit •  “Consumer-directed health plan”: offers financial incentive for consumers to

become involved in purchasing decisions for their own health care •  Little evidence for these plans on access, quality, and outcomes


•  Determine whether different insurance features reduce payments for medical services while preserving the health of their beneficiaries

•  Experiment through voluntary participation in alternative designs in Medicare, Medicaid, and private insurance

•  Effect of different designs on access, quality, and PCOs

#15 Compared to usual care, what are the effects of different models of transitional care on patient safety and other PCOs?

35

Introduction: •  Transitional care: strategies designed to ensure the coordination and continuity of health care as patients transfer between different locations or different levels of care

Burden: •  Transitioning from inpatient to outpatient care, as well as transitioning to higher intensity care, are both periods of increased risk for adverse events

•  ~20% of patients have adverse events after discharge, half of which are considered preventable

•  Deficiencies in transitional care are felt largely through costs due to hospital readmissions


•  Transitional care involves three basic elements to guarantee continuity of care and quality: communication, medication reconciliation, and education

•  The evidence to support these interventions is fairly robust


•  How to best implement evidence-based transitional care strategies in real-world settings

•  Effective tools and processes to improve transitions •  Effect of different models and tools on outcomes

DISCUSSION

36

Identifying and Prioritizing Patient-Centered Research Questions: Orientation for Advisory Panels Rachael Fleurence, PhD, Acting Director, PCOR Methods April 19, 2013

Principles to Guide Us: Patients asked for Transparency, Efficiency, Collaboration

Transforming Patient-Centered Research:

Building Partnerships and Promising Models

Washington DC, October 27-28, 2012

Engaging the Wider Community

!  A two-pronged approach to identify high-priority research questions   Investigator-initiated Approach   Patient- and Other Stakeholder-initiated Approach

3

Patient- and Other Stakeholder-Initiated Approach

PCORI receives research topics from patients and other stakeholders

Advisory Panels prioritize research topics based on explicit criteria

PCORI issues specific funding announcements for highest priority topics

Peer review prioritizes applications by level

of alignment with criteria

Diverse research portfolio answering key questions for patients, clinicians, and

healthcare leaders

Researchers and stakeholders develop responsive applications

Identifying Questions

5

Topic Generation

Topic Generation

Confirming Research Gaps

Gap Confirmation

Research Opportunities

6

Topic Generation

Prioritizing Research Questions

Gap Confirmation


7

Research Prioritization


Topic Generation

Creating Funding Announcements

Gap Confirmation


8



Final Selection for Specific PFAs

PCORI’s Research Prioritization Process

9

From Research Questions to Research Studies

PCORI is Building on the Existing Evidence Base and Prior Experience

10

Existing Scientific Work and Literature

Methodology Committee and

Methodology Report

Experience of Other Agencies

Federal Coordinating Council for

Comparative Effectiveness

Research

PCORI Criteria for Ranking Research Topics

11

1. Patient-Centeredness

12

•  Are patients and clinicians asking for this research ?

•  Will research findings make a difference to patients and their clinicians when making healthcare decisions ?

2. Impact

13

•  Burden of disease in terms of prevalence, mortality, morbidity, individual suffering, loss of productivity?

•  Rare disease?

3. Options for Addressing the Issue

14

•  What is known about the relative benefits and harms of the available management options?

•  What could new research contribute?

•  Have recent innovations (eg, a new technology or a new policy) made research on this topic especially compelling?

4. Likelihood of Implementation in Practice

15

•  How likely is it that the research findings will be implemented in practice?

5. Duration of Information

16

• Will research findings be valid by the time the study has concluded?

Piloting the Process

17

•  Piloted from August to November 2012

•  35 Pilot participants •  8 criteria to prioritize 10 topics •  Results •  Feedback

Questions to Pilot from a Diverse Range of Disease Areas

Obesity Back Pain in the Elderly

Indoor Air

Pollution

Falls in the

elderly Prostate Cancer

Anti-psychotics in Young Adults

Breast Cancer

Coronary Artery Disease

Clostridium Difficile

18

Pilot Groups used 2 Different Tools to Prioritize

Group 1 Results Using Two Software Programs

0.00% 2.00% 4.00% 6.00% 8.00% 10.00% 12.00%

Indoor air pollution interventions

Effectivenss of multiple chronic conditions

Mindfulness-based interventions and

obesity

Treatment for C. difficile diarrhea

Efficacy of antipsychotics in adolescents and

Prevention of falls in the elderly

Management of elderly patients with back pain

Treatment of ductal carcinoma in situ (DCIS)

Biomarkers for the prevention of breast

cancer

Treatment of coronary artery disease

7.30%

8.79%

9.55%

9.64%

9.99%

10.20%

10.52%

11.03%

11.21%

11.77%

Expert Choice Survey Gizmo

67

137

145

145

152

156

177

199

201

216

0 50 100 150 200 250

Indoor Air Pollution

Obesity

Preventing Falls

Multiple Chronic Conditions

Antipsychotics in ADHD, bipolar disorder or

schizophrenia

Diarrheal Infection Clostridium Difficile

Treatment of Ductal Carcinoma In Situ

Management of Back Pain in Elderly Patients

Biomarkers for Breast-Cancer

Coronary Artery Disease

Total Score

Group 2 Results

21

0.00% 2.00% 4.00% 6.00% 8.00% 10.00% 12.00%

Indoor air pollution interventions

Treatment for C. difficile diarrhea

Effectivenss of multiple chronic conditions

Efficacy of antipsychotics in adolescents and children

Mindfulness-based interventions and obesity

Management of elderly patients with back pain

Biomarkers for the prevention of breast cancer

Prevention of falls in the elderly

Treatment of coronary artery disease

Treatment of ductal carcinoma in situ (DCIS)

7.28%

9.07%

9.49%

9.53%

9.89%

9.94%

10.69%

10.74%

11.41%

11.96%

Participants Provided Valuable Insights to Improve the Process

22

Emphasize Patient’s

Voice Clarify the

Criteria

Improve Supporting Information

Choose the Tools

Research Topics Submitted this Cycle

!  1393 topics submitted   552 topics excluded   841 topics accepted

23

Research Topics

Accepted Excluded

Source of Accepted Topics

ANA= American Nurses Association ANCC is the American Nurses Credentialing Center

24

AHRQ 25%

ANA and ANCC 3%

Friends of Cancer 2%

IOM 11%

NIH 8%

Web 39%

Stakeholder Workshop

12%

Prioritizing Research Topics

APDTO 594

1393 Research Topics

841 Accepted Topics

552 Excluded Topics

20 Topics 15 Topics 12 Topics

Reviewed in future cycle

IHS 308

AD 47

CD 97

Steps in Ranking Research Topics

  Learn how to use the tool Expert Choice

  Review and discuss topic briefs   Do ranking exercise   Discuss results and conduct final

ranking   PCORI Staff will commission

landscape reviews (Summer 2013) and present their recommendations to the Board (September 2013).

  Funding expected Q1 2014.

26

Expectations of Advisory Panel Participants

!  Challenge !  Experience !  Honoring others’ contributions !  Respectful dissonance !  Feedback to PCORI about

process and improvements!

27

Thank you !

28

Assessment of Prevention, Diagnosis, and Treatment Options

Advisory Panel April 19-20, 2013

1

Program Introduction and Overview 1 pm – 1:30 pm

David Hickam, MD, MPH Program Director Assessment of Prevention, Diagnosis, and Treatment Options PCORI

2

Advisory Panel Topic Selection Process

3

1. Public Input: PCORI received 1,393 research questions via the Web. 594 related to Assessment of Options.

2. Topic Selection: PCORI staff selected 20 topics in the Assessment of Options category (see Orientation guide).

3. Panel Review: 20 topic briefs inform prioritization based on PCORI research criteria.

4. Board Approval: high-priority topics will be considered for funding announcements from PCORI.

Score

Discuss

Rank

Goal: Generate a list of five prioritized topics.

Discussion of Topic Briefs 1:30 pm – 3:30 pm

4

Format (12 minutes per topic) 1. Background (3-5 minutes)

•  What is the important clinical question?

•  What are the gaps in current research?

•  Could research close these gaps?

2. Discussion (7-10 minutes) •  Does the topic meet the 5 PCORI criteria?

Break 3:30 PM – 3:45 PM

5

Discussion of Topic Briefs - continued 3:45 pm – 5:45 pm

6

Format (12 minutes per topic) 1. Background (3-5 minutes)




2. Discussion (7-10 minutes) •  Does the topic meet the 5 PCORI criteria?

Adjourn 5:45 PM

7

Recap of Day 1: Address Questions or Concerns 8 AM

8

Discussion of Topic Briefs: Review Tabled Items 8:15 am – 9:15 am

9

Format 1.  Background




2. Discussion •  Does the topic meet the 5 PCORI criteria?

Ranking of All Topics 9:15 am – 10 am

10

•  Expert Choice •  Objectives are the PCORI criteria •  Alternatives are the short topic

titles •  Scoring: 5 (high) – 1 (low)

Break 10 AM – 10:15 AM

11

Discussion of Future Research Directions 10:15 AM – 12:15 PM

12

Discussion of Top-Tier Topics 1:15 pm – 2:45 pm

13

•  Did the right questions end up in the top 5?

•  What’s missing?

Break 2:45 PM – 3 PM

14

Final Selection of Top-Tier Topics 3 pm – 3:15 pm

15

•  Survey Gizmo •  Rank top-tier items •  1 (top priority) to 5 (lowest

priority)

Break 3:15 PM – 3:30 PM

16

Conclusions and Next Steps 3:30 PM – 4 PM

17

Adjourn 4 PM

18

Welcome and Overview of the Advisory Panels Joe Selby, MD, MPH Executive Director, PCORI Advisory Panel Kickoff & Training April 19, 2013

1

Source: Affordable Care Act. Subtitle D—Patient-Centered Outcomes Research. PUBLIC LAW 111–148—MAR. 23, 2010.

“The purpose of the Institute is to assist patients, clinicians, purchasers, and policy-makers in making informed health decisions by advancing the quality and relevance of evidence concerning the manner in which diseases, disorders, and other health conditions can effectively and appropriately be prevented, diagnosed, treated, monitored, and managed through research and evidence synthesis that considers variations in patient subpopulations and the dissemination of research findings with respect to the relative health outcomes, clinical effectiveness, and appropriateness of medical treatments, services.

Purpose

Of

PCORI

PCORI’s Broad Mandate

Welcome to PCORI

!  PCORI funds research intended to provide patients and those who care for them the information they need to make better-informed healthcare decisions.

!  Our core duties are to:   Establish national research priorities   Establish and carry out a research agenda   Develop and update methodological standards   Disseminate research findings

3

Why PCORI is Necessary

BMJ – Clinical Evidence 2013

Continuum of Medical Research

Basic Science

Clinical Research

Effectiveness Research

T1

 RCTs for Efficacy and Safety

T2

  Implementation/Effectiveness studies

  Molecular discovery   Etiologic and

mechanistic research

PCOR

 Comparative Outcomes Studies

National Priorities for Research and Research Agenda

PCORI Funding – Approximate

2012

2013

2014

2015 . . . .

2019

$150,000,000*

$300,000,000

$500,000,000

$500,000,000

$500,000,000

* 20% each year goes to AHRQ/HHS 7

PCORI’s Key Criteria for Funding

8

!   Potential for Improvement – How likely is the proposed research to lead to changes in patient or clinician practices that lead in turn to meaningful improvement in patient health?

!   Patient-Centeredness – Is the proposed research focused on comparisons and outcomes that matter to patients and their caregivers?

!   Patient and Stakeholder Engagement – Have the researchers included in their team relevant patients and other key healthcare community members, representative of those who would use its information?

Our First Topics for Targeted Research Funding

!   Jump-starts our long-term effort to identify and prioritize specific research topics to study

!   Leverages stakeholder input !   Ad hoc workgroup meetings

recorded and available on the PCORI Web site

Research Topics: Treatment Options for Severe Asthma in African-American and Hispanic/Latino populations

Treatment Options for Uterine Fibroids

Preventing Injuries from Falls in the Elderly

Treatment Options for Back Pain

Obesity Treatment in Diverse Populations

9

Long-term Approach to Targeted Research: PCORI Advisory Panels

10

!   PCORI established the first four Advisory Panels in March 2013

!   1,295 applications were reviewed by PCORI staff, and the final panel slates were approved by the Board of Governors

!   Each panel has 21 members consisting of patients, researchers, clinicians, and other experts with relevant experience and knowledge to inform PCORI’s work

!   Each panel has a unique charter, term duration, and clearly defined scope of work

Addressing Disparities

Assessment of Prevention,

Diagnosis, and Treatment Options

Improving Healthcare Systems

Patient Engagement

Overview of PCORI Advisory Panels

11

What does the law say about advisory panels? Legislative

Authorization •  PCORI can appoint permanent or ad hoc advisory panels to assist in identifying research

priorities and establishing the research project agenda. •  Advisory panelists will include representatives of practicing and research clinicians, patients,

and experts in scientific and health services research, health services delivery, and evidence-based medicine who have experience in the relevant topic, and as appropriate, experts in integrative health and primary prevention strategies.

What’s the purpose of advisory panels? Purpose

•  Advisory panelists work in conjunction with PCORI staff to help identify research priorities and topics

•  Leveraging members’ expertise will help better inform PCORI’s mission and work.

Will panel members be eligible for future PCORI funding? Conflicts of Interest

•  Panel members are not making decisions on either funding or funding announcements. •  PCORI will maintain “firewalls” between Advisory Panel members and internal decisions that

would convey any knowledge advantage to panel members. •  Advisory panel meetings are recorded and publicly available immediately after a meeting. •  Therefore, membership generally does not preclude eligibility for funding.

What has been approved by PCORI’s Board? Panel Charters

•  Charters and panel slates were reviewed and approved for four panels: Assessment of Prevention, Diagnosis, and Treatment Options; Addressing Disparities; Improving Healthcare Systems; and Patient Engagement.

•  Members will initially be appointed for a one-year term, with the possibility of reappointment for a maximum of two terms.

•  Term of the charter will remain in effect for one year beginning on the day of the first meeting. •  Charter is subject to review, reauthorization, amendment, or termination by the Board of

Governors or its designee.

Overview of PCORI Advisory Panels

12

PCORI’s Board of Governors

13

PCORI Board of Governors, March 2012 in Baltimore, MD

Why PCORI Needs You

!   Advisory Panels will assist PCORI in developing and refining our patient-centered comparative clinical effectiveness research portfolio and in disseminating research findings through:

  prioritization of proposed research topics

  provision of input and expertise on questions that arise within the Advisory Panel’s focus area

  periodic evaluation of PCORI’s research portfolio

  Ongoing two-way communication with stakeholder communities

  Ensuring the highest patient engagement standards and a culture of patient-centeredness in both our work and the research that we fund.

14

Distribution of Advisory Panelists

15

20%

41%

18% 21%

Panelists by Stakeholder Group (Self-Identified; N=84)

* Map of census regions and divisions of the United States made available by the U.S. Census Bureau, U.S. Department of Commerce, Economics and Statistics Administration.

Panelists by Geographic Location (Self-Identified; N=84)

General Session Agenda

16

Time Session PCORI Presenters 8:45-9:15 Welcome and Overview of the Advisory Panels Joe Selby, Executive Director

9:15-9:35 Patient Engagement in Research Sue Sheridan, Director, Patient Engagement

9:35-10 Q&A Session Anne Beal, Deputy Executive Director & Chief Officer for Engagement

10:15-11:15 Research Prioritization Training Rachael Fleurence, Acting Director, PCOR Methods

11:15-11:45 Software Training Kara Odom Walker, Program Officer

11:45-12 Wrap-Up and Announcements Anne Beal, Deputy Executive Director & Chief Officer for Engagement

Patient Engagement in Research

Susan Sheridan, MBA, MIM Director, Patient Engagement

Advisory Panel Kickoff & Training April 19, 2012

How PCORI Engages Patients in Research

!   Identifying and Selecting Research Questions

!   Reviewing Research Proposals for Funding

!   Conducting PCORI-Funded Research

!   Matching Patients and Stakeholders with

Researchers

!   Disseminating Research to the Community

!   Evaluating PCORI’s Patient and Stakeholder

Engagement Programs

Transforming Patient-Centered Research: Building Partnerships and Promising Models

!   PCORI’s October 2012 workshop included five breakout sessions to help PCORI implement a patient-centered research agenda through sustained and meaningful engagement.

!   PATIENT ENGAGEMENT WORKSHOP VIDEO

Identifying and Selecting Research Questions

!   How should PCORI identify and select specific research questions for funding that are patient-centered?

Identifying and Selecting Research Questions

  PCORI Web site

  Roundtables   Workshops

  PCORI Methodology Report

  Advisory Panels

Reviewing Research Proposals for Funding

!   How can PCORI effectively engage and use the real-world experience of patients to help evaluate research proposals we receive?

Reviewing Research Proposals for Funding

!  Building a community of informed stakeholder reviewers   Training patients and stakeholders on PCORI review

criteria, process, and instructions on how to score PCORI Funding Applications

  Identifying Reviewer Mentors   Matching expertise with research priority area   Equalizing voices at the reviewer table

Conducting PCORI-Funded Research

!  PCORI Methodology Report   Standards Associated with Patient-Centeredness

!  Review of PCORI Funding Applications   Patient-Centeredness

!  Exemplary models of patient engagement in research   High Plains Research Network - Boot Camp Translation

for Patient-Centered Outcomes (Dr. Westfall & Maret Felzien)

Matching Patients and Stakeholders with Researchers

!   How can PCORI connect patients and stakeholders with researchers for collaborative work that ensures studies reflect patient perspectives?


!  Video Clip: Ming Tai Seale


!  The PCORI Challenge Initiative   Developing a patient/research “matching” system that

will effectively connect potential partners interested in seeking funding from us for rigorous patient-centered outcomes research

Disseminating Research to the Community

!   How do we ensure that patients and those who care for them can access and use PCORI’s research to make more-informed decisions?

Disseminating Research to the Community

!  Speeding the implementation of and use of PCOR !  Becoming a trusted resource for information !  Engaging stakeholders from across the healthcare

community to include PCORI research in training, practice, and standards

Evaluating PCORI Engagement Activities

!   How can PCORI measure the effectiveness of its programs to involve patients and stakeholders throughout its work?

Evaluating PCORI Engagement Activities

!  Maximizing engagement activities to inform programs and understand the impact

!  Building trust in PCORI through transparent communication of the results of engagement efforts

Introduction & Overview of Patient Engagement Sue Sheridan, MBA, MIM Director, Patient Engagement Advisory Panel Kickoff & Training April 19, 2013

1

Who are we as a group?

Stakeholder Group #

Patient, Caregiver & Patient Advocate 13

Researcher 2

Clinician, Physician 2

Clinician, Occupational Therapist 1

Clinician, Nurse 1

Industry, Pharmaceuticals 1

Policymaker 1

2

What brings you here?

Please introduce yourself to the group: Name Where are you from? Why did you decide to support PCORI’s work in this

area?

3

Questions to Consider

Creating a community – what is the role of this advisory panel? What is true patient-centeredness? What does success look like? What is your experience?

4

Agenda for Today

5

Session Objective Introductions & Patient Engagement Overview

Panelist introductions

Patient Engagement Advisory Panel Charge & Work Plan

Share meeting objectives and overall vision for the panel Facilitate the development of a vision document Write the panel scope of work together

Brainstorming Begin considering PCORI-pedia (notecards are available for

idea sharing) Share pre-meeting activity responses

The Roles of Patients in Research

Share and discuss what is meaningful patient engagement in research


Share and discuss best practices in matching patients and researchers

Evaluation of Engagement Efforts Current evaluation strategies The potential role the panel could play in this capacity moving

forward

Advisory Panel on Patient Engagement Charge & Work plan Sue Sheridan, MBA, MIM Director, Patient Engagement Advisory Panel Kickoff & Training April 19, 2013

6

Source: Affordable Care Act. Subtitle D—Patient-Centered Outcomes Research. PUBLIC LAW 111–148—MAR. 23, 2010.

“The purpose of the Institute is to assist patients, clinicians, purchasers, and policy-makers in making informed health decisions by advancing the quality and relevance of evidence concerning the manner in which diseases, disorders, and other health conditions can effectively and appropriately be prevented, diagnosed, treated, monitored, and managed through research and evidence synthesis that considers variations in patient subpopulations and the dissemination of research findings with respect to the relative health outcomes, clinical effectiveness, and appropriateness of medical treatments, services.

Purpose

Of

PCORI

PCORI’s Broad Mandate

Advisory Panel on Patient Engagement Charge

Advise on processes to identify research topics and priorities that are important to patients; Advise on all aspects of stakeholder review of applications for PCORI funding; Provide general recommendations to PCORI and externally on the conduct of patient-centered research; Advise on methods to evaluate the impact of patient engagement in research; Assist and advise PCORI on communications, outreach, and dissemination of research findings; and Provide advice on other questions and areas of interest that may arise that are relevant to PCORI’s mission and work.

8

Meeting Objectives

Familiarize panelists with PCORI’s mission, national priorities, and the purpose and function of the Advisory Panels Inform panelists on all of PCORI’s engagement opportunities Train panelists in research prioritization Review recommendations on best practices in patient engagement in research outlined in the final workshop report and offer additional input Receive advice and recommendations on the Engagement Awards and Ambassadors Program Collaboratively develop a pledge for the panel, as well as a work plan and expectations of panelists for Year 1

9

Meeting Outputs

Patient Engagement Panel vision document

Patient Engagement Panel work plan

Edits to PCORI Ambassadors Program and the

Engagement Awards

Recommendations and enhancements for best

practices in patient engagement in research

PCORI-pedia – definitions for PCORI terms

10

Advisory Panel on Patient Engagement Work Plan

Identify goals to work toward over the next 12 months

11

Brainstorming

Sue Sheridan, MBA, MIM Director, Patient Engagement Advisory Panel Kickoff & Training April 19, 2013

12

Review Responses to Pre-Meeting Activity

What is the biggest hope I have for patient-centered research?

What does success look like for patient engagement in PCORI's research?

13

Idea Capture

This is an opportunity to share any additional ideas, hopes, and goals you may have for the Patient Engagement Advisory Panel

14

The Roles of Patients in Research Sue Sheridan, MBA, MIM Director, Patient Engagement Advisory Panel Kickoff & Training April 19, 2013

15

Putting Patients at the Center

16

http://youtu.be/4OzgXWMNhDk

The Roles of Patients in Research

What is meaningful patient engagement in research?

17

Matching Patients and Stakeholders with Researchers Sue Sheridan, MBA, MIM Director, Patient Engagement Advisory Panel Kickoff & Training April 19, 2013

18

Methodology Standards Associated with Patient-Centeredness

PC-1 Engage people representing the population of interest and other relevant stakeholders in ways that are appropriate and necessary in a given research context. Stakeholders can be engaged in the processes of: Formulating research questions; Defining essential characteristics of study participants, comparators, and

outcomes; Identifying and selecting outcomes that the population of interest notices

and cares about (eg, survival, function, symptoms, health-related quality of life) and that inform decision making relevant to the research topic;

Monitoring study conduct and progress; and Designing/suggesting plans for dissemination and implementation

activities.

19

20

Video Clip: Ming Tai Seale

http://www.youtube.com/watch?v=TVfOkr18y1E&feature=youtu.be

Review Criterion 7: Team and Environment (Current)

Are the investigators appropriately trained and experienced to carry out the planned studies?

Is the work proposed appropriate to the experience level of the principal investigator?

Does the study team have complementary and integrated expertise?

Is their leadership approach, governance, and organizational structure appropriate for the project?

Are relevant patients and other key stakeholders of the study information appropriately involved in the design and implementation of the study?

Do the experiments proposed take advantage of unique features of the scientific environment or employ useful collaborative arrangements?

Is there evidence of institutional or other support?

The 8 Merit Review Criteria:

1. Impact of the Condition

2. Innovation/Potential for Improvement

3. Impact on Healthcare Performance

4. Patient-Centeredness

5. Rigorous Research Methods

6. Inclusiveness of Different Populations

7. Team and Environment

8. Efficient Use of Resources

21

22

Video Clip: David Thorn

http://www.youtube.com/watch?v=rSnQEM5yjGs&feature=youtu.be

Challenge Background

PCORI is committed to meaningful patient, caregiver, and stakeholder engagement as a tool for rigorous research. PCORI funding announcements require that patients be fully engaged throughout research process. Engagement is among the criteria PCORI uses to score applications.

23

The Challenge

Develop a “matching” system that can connect researchers and potential patient partners. Solution could be: A well-articulated conceptual model. An adaptation of existing matching protocol. A prototype or an entirely new Web-based service or

app. Some combination of these approaches, or something

else entirely.

24

The Challenge

Two first-place awards: Conceptual model – $10,000 Prototype of app – $40,000

Winners’ work may be considered for additional PCORI support, depending on outcome of the review process. Submission materials: slide deck (5 slides), overview doc (5 pgs), video demo (5 min), link to working app (optional)

25

Evaluation Criteria

Technical feasibility, usability, and scalability of the proposed conceptual model/prototype. Differences in ways patients, caregivers, and researchers understand, describe and seek answers to problems or issues they face. Maximizing patient-centeredness and scientific rigor. Particular challenges of serving hard-to-reach audiences: ethnic and racial minorities, rural populations, the elderly, physically challenged people, and non-English speakers.

26

Timeline

Submission period began: December 14, 2012 Submission period ended: April 15, 2013 Winners notified: May 15, 2013 Winners announced: at a major national health conference in the spring of 2013

27

Video Clip: Dr. Vinod Bhutani/Kris Schulze

28

http://youtu.be/kqlU3W8VUoM

Evaluation of Engagement Efforts Lori Frank, PhD Director, Engagement Research Advisory Panel Kickoff & Training April 19, 2013

29

Vision for Engagement Research

Evaluation is fundamental to understanding and improving all that PCORI does. Engagement Research ensures that PCORI understands and maximizes the impact of its research funding to help people make informed healthcare decisions and improve healthcare delivery and outcomes.

30

Goals for Engagement Research

Evaluate research engagement by PCORI awardees to identify and support best practice Evaluate impact of PCORI funding Establish a survey research function to inform PCORI programs Evaluate PCORI engagement and outreach programs

31

Elements for Evaluation of Research Engagement from PCORI Workshop

Define success Determine goals of engagement by research phase Establish feedback channels Address patient and researcher parity Assess patient and researcher perceptions of the value and appropriateness of engagement

32

Date post:	28-Jan-2015
Category:	Health & Medicine
Upload:	patient-centered-outcomes-research-institute
View:	111 times
Download:	0 times

PCORI Advisory Panels Kickoff & Training

Health & Medicine