Performance Measurement in Pediatric Emergency Care

Evie Alessandrini, MD, MSCE

Center for Health Care Quality

Division of Emergency Medicine

Cincinnati Children’s Hospital Medical Center

I have no relevant financial relationships with the manufacturers of any commercial products and/or provider of commercial services discussed in this CME activity.

I do not intend to discuss an unapproved or investigative use of a commercial product/device in my presentation.

Disclosure

Performance Measurement

Learning Objectives

• Delineate the different purposes of performance measurement

• Become familiar with existing endorsed performance measures of emergency care

• Develop a framework for organizing pediatric emergency care performance measures for your ED

Measurement Motivators• Health Care System

– Growing complexity and costs– Transparency initiative, external mandates

• Institute of Medicine Reports– To Err is Human, 1999– Crossing the Quality Chasm, 2001– Performance Measurement: Accelerating

Improvement, 2005– The Future of Emergency Care, 2006

• Develop national standards for emergency care performance measurement

Outcomes

Quality

Why Measure Performance?Stakeholder Survey Analysis

Why Measure Performance?• Improve

– Health and Healthcare• For patients and populations• Within one ED or with one practitioner• Within networks of EDs or health systems

• Inform– Transparency, consumer decision-making– Regionalization of care

• Incentivize– Pay for performance– National rankings

Stakeholder Survey Analysis

Performance Measurement:Accelerating Improvement

2005 IOM Report

• The ultimate purpose of performance measurement is to improve the health of everyone in the United States

• Performance measures are yardsticks by which all health care providers and organizations can determine how successful they are in delivering recommended care and improving patient outcomes

• Public reporting of performance data holds health providers accountable to both consumers and purchasers of care; transparency builds trust

• Patients can also learn what the expected professional standards of care are and where they can go to receive it

Performance Feedback

Feedback has proven most effective when • rates of adherence to practice guidelines are low• the information is directly useful for care• practitioners are motivated to change

IOM. Rewarding Provider Performance: Aligning Incentives in Medicine. Washington DC: American Academy Press; 2006.

Feedback Report: Appropriate Use of Amoxicillin for Newly Diagnosed Acute Otitis

Media (AOM)

Your Performance

(December 2009 – February 2010)

81%

Top Performers

(December 2009 – February 2010)

100%

Quality Measure (Numerator): Amoxicillin Prescribed

Quality Measure (Denominator): All Visits for Acute Otitis Media Satisfying Inclusion and Exclusion Criteria

Inclusion Criteria: Visit Diagnosis of Acute Otitis Media Children 2 months to 12 years of age Oral antibiotic prescribed at the visit

Exclusion Criteria: Acute Otitis Media Visits in the Past 14 Days Antibiotics in the Past Month Penicillin/Amoxicillin Allergy Co-infections: Pneumonia, Sinusitis, Conjunctivitis, or Pharyngitis

Improving Otitis Media Care with Clinical Decision Support and Feedback; AHRQ R18 HS017042

Feedback Report: Appropriate Use of Amoxicillin for Newly Diagnosed Acute Otitis Media (AOM)

Appropriate Amoxicillin Use

TimePeriod

You Your Practice Network

# of Visitswith NewlyDiagnosedAOM

AmoxicillinAppropriatelyPrescribed(N, %)

AmoxicillinAppropriately Prescribed (%)

AmoxicillinAppropriatelyPrescribed(%)

September 2009 through November 2009

19 18 (95%) 91% 77%

December 2009 through February 2010

16 13 (81%) 93% 77%

Improving Otitis Media Care with Clinical Decision Support and Feedback; AHRQ R18 HS017042

Transparency“Transparency aims to reduce specific risks or performance

problems through selective disclosure by organizations. The ingeniousness of target transparency lies in its mobilization of individual choice, market forces, and participatory democracy.” Wikipedia

Levels of transparency– Within your department– Within your hospital– Outside your hospital

Examples• http://my.clevelandclinic.org/Documents/Medicine/PEDS283076A_LR.pdf• http://www.dartmouthatlas.org

Pay for Performance“Pay for performance systems link compensation

to measures of work quality or goals”Perfect Asthma Care• The pay-for-performance program rewarded

practices for – participating in an improvement collaborative– achieving network- and practice-level performance

thresholds– building improvement capability

• The percentage of the network asthma population receiving "perfect care" increased from 4% to 88%.

Mandel, KE; Archives of Ped and Adol Med: 161(7): 650-5, 2007 July

Leadership & Performance Measurement

“Turning Doctors into Leaders”• Organize doctors into teams• Measure performance by patient

outcomes, not resource use• Apply financial and behavioral incentives• Dismantle dysfunctional cultures

Harvard Business Review, April 2010; hbr.org

Performance Measurement

Learning Objectives

• Delineate the different purposes of performance measurement

• Become familiar with existing endorsed performance measures of emergency care

• Develop a framework for organizing pediatric emergency care performance measures for your ED

National Quality Forum2008 Measures

Hospital-based Emergency Care• Median time from ED arrival to ED departure for

admitted patients*• Median time from ED arrival to ED departure for

discharged patients*• Admit Decision Time to ED Departure Time for Admitted

Patients*• Door to provider• Left without being seen

* Measures stratified by– Psychiatric diagnoses, observation patients, transfers, all others

www.qualityforum.org

National Quality Forum2008 Measures

Hospital-based Emergency Care (cont.)• Severe Sepsis and Septic Shock: Management Bundle • Confirmation of endotracheal tube placement• Percentage of patients with Chest Pain Symptoms in ED

receiving Early Therapy including IV, Oxygen, Nitroglycerin, Morphine and Chewable Aspirin on Arrival

• Pregnancy test for female abdominal pain patients• Anticoagulation for Acute Pulmonary Embolus Patients• Pediatric Weight documented in kilograms

– AAP is the measure sponsor

www.qualityforum.org

Children’s Health Corporation of America Existing Measures

Whole System Measures

• ED Left Without Being Seen

• ED Length of Stay

www.chca.com

Urgent Matters Robert Wood Johnson Foundations

Urgent Matters Initiative Goals

• Rigorously evaluate the implementation of strategies for improving patient flow and reducing ED crowding

• Advance the development of standard performance measurement in the ED

• Promote spread of promising practices to a wider audience / variety of hospitals

www.urgentmatters.org

Urgent Matters Robert Wood Johnson Foundations

Performance Measures (NQF and CMS)• Throughput for admitted patients• Throughput for discharged patients• Time to pain management for long bone fractures

– Admitted patients– Discharged patients

• Median time to chest x-ray– Admitted patients– Discharged patients

• Admit decision time to ED departure time www.urgentmatters.org

Other Existing Measures and Measurement Organizations

• Joint Commission – ORYX performance measures

• Children’s Asthma Care measures (inpatient)• http://www.jointcommission.org/PerformanceMeasurement

• AHRQ Pediatric Quality Indicators (PDIs)– 18 risk-adjusted measures– Obtained from inpatient administrative data– www.qualityindicators.ahrq.gov/pdi_overview.htm

• Alliance for Pediatric Quality– AAP, American Board of Pediatrics, CHCA, NACHRI– www.kidsquality.org

Other Existing Measures and Measurement Organizations

ACEP National Report Card on the State of Emergency Medicine

• Access to Emergency Care

• Quality and Patient Safety Environment

• Medical Liability Environment

• Public Health and Injury Prevention

• Disaster Preparedness

http://www.emreportcard.org

Performance Measurement

Learning Objectives

• Delineate the different purposes of performance measurement

• Become familiar with existing endorsed performance measures of emergency care

• Develop a framework for organizing pediatric emergency care performance measures for your ED

Rationale for Framework

Limitations of prior work– Single centers or geographic locales – Focus on condition-specific indicators– Preponderance of process-oriented measures– Benchmarks very focused on

• Timeliness (through put)• Satisfaction (ceiling effect)

– Lack of comprehensiveness regarding spectrum of ED care• Lindsay et. al., AEM, 2002• Guttmann et. al., Pediatrics, 2006

Performance Measure Framework

Quality indicator set development process

• Adapted from AHRQ

• “Defining Quality Performance Measures for Pediatric Emergency Care”

– Funded by HRSA/EMSC Targeted Issues Grant H34MC08512

CandidateMeasures

N = 60

ResearchLiterature

Actual Use

Concept

Measure by MeasureEvaluation

for Selection

SelectionCriteria

Measure SetEvaluation forApplication

SOURCES

Performance Measure Development

Adapted from AHRQ PDI development process

Performance Measure Framework

Measure development dimensions• IOM Quality Domains• Donabedian’s framework for quality• PEM disease frequency and severity

Measure evaluation dimensions• National Quality Forum criteria

Institute of Medicine Quality Domains

Built around the core need for health care to be

• Safe

• Effective

• Efficient

• Timely

• Patient-centered

• Equitable

Institute of Medicine Quality Domains

Safe• Health care avoids injuries to patients

from the care that is intended to help them

Effective• Health care provides services based on

scientific knowledge to all who could benefit, and refrains from providing services to those not likely to benefit

Institute of Medicine Quality Domains

Efficient• Health care avoids waste, including waste

of equipment, supplies, ideas and energy

Timely• Health care reduces waits and sometimes

harmful delays for both those who receive and those who give care

Institute of Medicine Quality Domains

Patient - centered• Health care provides care that is respectful of

and responsive to individual patient preferences, need and values, and ensures that patient values guide all clinical decisions

Equitable• Health care provides care that does not vary

because of personal characteristics such as gender, ethnicity, geographic location, and socioeconomic status

Donabedian’s Framework

Structure• Indirect quality-of-care measures related to a physical

setting and resources: Staff, space, supplies, equipment and financial resources

Process• Measures evaluate the method or process by which care

is delivered, including both technical and interpersonal components

Outcome• Outcome elements describe valued results related to

lengthening life, relieving pain, reducing disabilities and satisfying the consumer

PEM Disease Frequency & Severity

• Condition-specific– Proportion of patients with croup receiving

corticosteroids

• General– Proportion of patients returning to the ED within

72 hours of an initial ED visit

• Cross-cutting– Proportion of patients with pain who receive an

analgesic

Choosing Condition-Specific Measures

24.59

19.22

9.97 9.65 9.64

16.1815.33

11.9112.80

11.48

0.00

5.00

10.00

15.00

20.00

25.00

30.00

Trauma ENT, Dental &Mouth Diseases

GastrointestinalDiseases

RespiratoryDiseases

Systemic States

Major Group

% o

f D

iag

no

sis

NHAMCS

PCDP

Alessandrini et.al., Academic Emerg Med; February 2010

www.pecarn.org/tools

Measure Evaluation Criteria

Importance• The measure reflects a priority or high

impact aspect of healthcare• The measure addresses outcomes or is

strongly linked to improving outcomes• The measure addresses an area of

considerable variation or poor performance across providers or population groups

Measure Evaluation Criteria

Scientific Acceptability• There is strong evidence for the specific

measure focus, such as evidence based guidelines

• The measure is reliable, reproducible and accurately represents quality of care

Measure Evaluation Criteria

Usability• The measure provides information that is

actionable and can be used to make decisions that improve the quality of care

• The measure is meaningful and understandable

Measure Evaluation Criteria

Feasibility• Data for the measure is generated during

care delivery and is available in the EHR or other electronic sources

• Data collection for the measure can be implemented

• The information provided outweighs the costs/burdens of collecting the data

Results: Performance Measure Distribution by IOM Quality

Domain

Equitable – measures stratified by gender, age, race, ethnicity and payor

Applicability of Measures to IOM Domains (Measures can apply to more than one IOM domain)

43

34

14

7

17

05

101520253035404550

Effective Safe Patient Centered Efficient Timely

# o

f Mea

sure

s App

licab

le to

IO

M

Dom

ain

Results: Performance Measure Distribution by Donabedian

Framework

Distribution of Measures by Donabedian Classification

13

37

10

0

5

10

15

20

25

30

35

40

Outcome Process Structure

Measure Type

# o

f Mea

sure

s

Results: Performance Measure Distribution by

Diagnosis Type

Cross-cutting measures include pain/sedation, severe illness, diagnostic testing and medication management

Distribution of Measures by Category

27

13

20

0

5

10

15

20

25

30

General Cross-Cutting Condition-Specific

# o

f Mea

sure

s

PEM Balanced Report Card

• Measuring weight in kilograms for ED patients <18 years of age

• All pediatric equipment present in the ED (per ACEP, AAP, ENA policy statement)• Reducing pain in children with acute fractures• Systemic corticosteroids in asthma patients with

acute exacerbation• Medication error rates• Parent/caregiver understanding of ED discharge

instructions• Door to provider• Total ED length of stay

Further Considerations

• Measures valuable to patients– Not minimum level of competency

• Composite measures– Conceptual and analytic issues

• Unit of analysis• Measure target• Locus of control• Balancing measures

– Are we improving parts of our system at the expense of others?

Measurement & Quality

• Quality is central to achieving affordable care that knows patient needs and keeps them healthy 

• It’s a three step process– The first step toward achieving quality is convening expert

members across the healthcare industry, including patients to define quality with uniform standards and measures that apply to the many facets of care patients receive. 

– Second, information gleaned from measuring performance is reported and analyzed to pinpoint where patient care falls short. 

– Third, caregivers examine information about the care they are providing and use it to improve.

Measure. Report. Improve.

References

• AAP Policy Statement: Principles for the Development and Use of Quality Measures– Pediatrics 121 (2), February 2008, pp 411-418

• Pediatric Clinics of North America “Pediatric Quality”: Quality Measures in Pediatrics– Volume 56 (4), August 2009, pp 816-829

References

• Institute of Medicine Report: Performance Measurement, Accelerating Improvement– December 2005– www.iom.edu/Reports/2005/Performance-

Measurement-Accelerating-Improvement.aspx

• Joint Policy Statement—Guidelines for Care of Children in the Emergency Department– Pediatrics 2009;124:1233–1243

Steps in Measure Specification

• Numerator statement• Denominator statement• Denominator exclusions• Data source and collection methods• Sampling• Risk adjustment• Stratification to detect disparities• Level of measurement / analysis

Steps in Measure Specification

Risk Adjustment

• Accounts for patient-associated factors before comparing outcomes across settings

• “Levels the playing field”

• Would be unnecessary if patients were randomly assigned to treatments, settings etc.