Developing Quality Indicators When There is Limited Evidence:

The Example of Injury Care

H. Thomas Stelfox, MD, PhDUniversity of Calgary

February 9, 2012

Objectives

1. Describe a quality indicator development process– Review strategies for determining the need for

quality indicators– Review the value of environmental scans– Discuss the role of consensus methodologies

2. Review lessons learned from the process

Objectives

1. Describe a quality indicator development process– Review strategies for determining the need for quality

indicators• Are there opportunities to improve care?

16 yo Post Motor Vehicle Collision

Decompressive Craniectomy

• Patient scheduled for OR

• Pre-op blood work ordered

• aPTT > 5 X normal

Ordered Administered

Problem – Quality of Care

• 98,000 die in US hospitals from error each year

• Canadian study of adverse events:– 7.5% of hospitalized patients– 1/3 of adverse events judged preventable

Patient Safety Publications Before & After IOM Report

Stelfox et al. IQSH 2006

Personal Experience with Error

Response Physicians(N=831)

Public(N=1207)

P-value

Error made in own or family member’s care

35% 42% <0.001

Health consequences

Serious 18% 24% <0.001 Minor 10% 13% 0.03 None 7% 5% 0.06Blendon et al. NEJM 2002

Quality of Trauma Care• Medicine’s quality problem includes

trauma:– Half of all patients do not receive

recommended care

– Medical errors common in critically ill trauma patients

– 2.5% - 14% trauma deaths in hospital are preventable

What is the Quality of Trauma Care?

The Challenge – Quality Measurement & Improvement

“If you can’t measure it you can’t manage it”

Peter Drucker

To Address This Challenge

Research program to develop population-based & evidence-

based indicators of quality of care in critically injured adult patients

Research ProgramResearch Synthesis International Audit of

QI Practices

Potential Quality Indicators

Quality Indicators for Evaluation of Implementation

Evaluation of Implementation

Final Quality Indicators

Multi-Step QI Development Process

Years 1-2

Year 3

Years 4-5

Objectives

1. Describe a quality indicator development process– Review strategies for determining the need for quality

indicators• Are there opportunities to improve care?• What quality indicators currently exist?

Research Synthesis

To systematically review the literature about quality indicators (QI) for

evaluating trauma care

Scoping Review Quality Indicators

Stelfox et al. Arch Surg 2010

Pediatric Patients

• Deficiencies in care 8%-45% pts.

• 6%-32% deaths judged preventable

• Need for pediatric specific measures

• No evaluations of validity or reliability

Stelfox et al. Crit Care Med 2010

Adult PatientsBest Indicators

• Preventable death

Indicators To Avoid• Time to craniotomy in TBI

Potential Indicators• Complications• Non-fixation of femur fracture• Scene time• ↓ LOC & airway management• Unplanned return to OR• Time to emergency laparotomy• ↓ LOC & time to CT head• Readmission to hospital• Time to basic diagnostics• Reintubation <48 hrs• Preventable morbidity• Missed injury• Statistically unexpected death

Stelfox et al. Crit Care Med 2011

Impact on Quality of CareSource Outcome Result

Before After

Chadbunchachai et al. 200155

Hospital mortalityPreventable deathTreatment pitfalls

Pitfalls → mortality

2.6%3.0%

43 per 1000 patients29 per 1000 patients

2.4%2.0%

23 per 1000 patients12 per 1000 patients

Chadbunchachai et al. 200354

Hospital mortalityPreventable deathTreatment PitfallsPitfalls → mortality

2.4%2.0%

23 per 1000 patients12 per 1000 patients

1.4%1.3%

17 per 1000 patients10 per 1000 patients

Ruchholtz et al. 200256

Assessment criteria

Hospital mortality 17%

Improved both centers 25%*

Improved one centre 45%*

Improved neither centre 30%11%

*p<0.05

Stelfox et al. Crit Care Med 2011

What does the Evidence Tell Us?

• QI literature for evaluating trauma care:

– Adults more than children

– Acute care more than post-acute care

– Supported by limited scientific evidence• A few promising indicators• May be associated with improved care

Objectives

1. Describe a quality indicator development process– Review strategies for determining the need for quality

indicators– Review the value of environmental scans

• How are quality indicators used?

Environmental Scan of Trauma Centres

• Goal – describe real world trauma centre performance improvement activities

• U.S., Canada, Australia & New Zealand

• Electronic surveys to 328 trauma centres– 249 centres responded (76%)

• Follow up interviews of 76 centres

Quality Indicators

Quality Indicators USA(N=198)

CND(N=35)

AUS(N=16)

Structure indicators 88% 66% 75%

Process indicators 98% 91% 75%

Outcome indicators 99% 91% 88%

*

*

** p<0.05

Stelfox et al. Ann of Surg 2012 in press

Performance Improvement Practices

PI Practices USA(N=198)

CND(N=35)

AUS(N=16)

M & M Conference 97% 94% 88%

Quality Audits 93% 89% 88%

Report Cards 53% 26% 31%Internal Benchmark 81% 63% 69%Extern Benchmark 81% 37% 63%

* p<0.05

***

Phase of Care Evaluated by QIs

Prehosp

ital

Hospita

l

Post-Hosp

ital

2° Prev

entio

n

Prehosp

ital

Hospita

l

Post-Hosp

ital

2° Prev

entio

n

Prehosp

ital

Hospita

l

Post-Hosp

ital

2° Prev

entio

n0

5

10

15

20

25

30

USACNDAUS

No. I

ndic

ator

s pe

r Cen

tre

*

* *

Structure Process Outcome

*

IOM Aims Evaluated by QIs

Safe

Effecti

ve

Patien

t-Cen

tered

Timely

Efficien

t

Equitable

0

10

20

30

40

USACNDAUS

No.

Qua

lity

Indi

cato

rs p

er C

entre

* p<0.05

*

* * *

Do Trauma Centres use the Same Indicators?

Santana et al J Trauma 2012 in press

10 Most Common Quality IndicatorsQuality Indicator Percentage of Centres (n=247)Appropriate admission service/MD 53

Hospital mortality 43

Secure airway in comatose patient 40

Time to laparotomy 39

Scene time 38

Time to craniotomy 36

Length of stay 35

Reintubation 34

Non-surgical gunshot wound management 32

Unplanned return to operating room 30

Santana et al J Trauma 2012 in press

How did you decide on your QIs?• Accreditation

“A lot of it is driven by formal regulations as what is expected of us as a trauma center either by

our State authority or the ACS.”

• Local Issues“We had cases with certain issues that were recurrently popping up, and these issues were

chosen for monitoring.”

How have PI activities impacted your trauma program?

• Culture of quality“The whole program is driven by safety and quality.”

• Standardized care“Set standards so we have clear expectations on how

trauma patients should be care for.”

• Improved processes & outcomes of care“It has improved it tremendously, we keep getting better

at what we are doing, it helps us drive down our morbidity and mortalities.”

How has your trauma program evaluated its PI activities?

• No Evaluation“No it’s hard – you can take forever – you

can evaluate the evaluation of the evaluation”

• Informal Evaluation“Nothing formally, but we meet once a week

and go through what’s working, what’s not working, what we want to change.”

What can a program do to improve its quality of trauma care?

• Better measures: “It’s a vicious circle. If we had the stats to prove we make a difference it would be easier to get funding. But how do you get there?”

• Better benchmarking: “In an ideal world I would love to benchmark outcomes, but people don’t use the same data dictionary and it is hard to benchmark when not comparing apples to apples”

• More concurrent: “My dream would be that we would track our indicators in real time, review charts within a day or two of them being flagged and fix problems while still active.”

Summary of Environmental Scan Observations

• Trauma centres spend a lot of energy and time on quality measurement & improvement

• Significant variation exists in how trauma centres measure & manage the quality of care they deliver

• Significant gaps exist within the observed quality improvement processes

What Next?

Objectives

1. Describe a quality indicator development process– Review strategies for determining the need for quality

indicators– Review the value of environmental scans– Discuss the role of consensus methodologies

• How to bridge the gap between the evidence base and the need for developing applied measures?

Quality Indicator Development

Expert Panel ReviewRound 1

Potential Quality Indicators

Quality Indicators for Evaluation of Implementation

Expert Panel ReviewRound 2

Expert Panel Workshop

Expert Panel

Sample Quality Indicator

Rating Scale

Quality Indicator Development Process74 •Round 1: 22 accepted, 52 neutral, 0 rejected, 5 proposed

57 •Round 2: 10 accepted, 49 neutral, 1 rejected, 2 proposed

84 •Workshop: 43 accepted, 14 neutral, 27 rejected, 24 merged

43 •Working Groups: 43 merged into 33, mini SRs for 33 QIs

33 •Final Review: 32 accepted, 1 rejected

32

Research ProgramResearch Synthesis International Audit of

QI Practices

Potential Quality Indicators

Quality Indicators for Evaluation of Implementation

Evaluation of Implementation

Final Quality Indicators

Multi-Step QI Development Process

Years 1-2

Year 3

Years 4-5

Stepping BackAre we on target?

Gruen et al. BJS 2012

Indicators Developed

Phase of care Structure Process Outcome

Prehospital 3 5 1

Hospital 2 14 5

Posthospital X 1 2

2˚ Prevention X 1 1

* 1 to 2 indicators per cell of our conceptual model

Objectives

1. Describe a quality indicator development process– Review strategies for determining the need for quality

indicators– Review the value of environmental scans– Discuss the role of consensus methodologies

2. Review lessons learned from the process

Lesson #1Clear Purpose & Goals for the Quality

Indicator

“...I would submit that the end that we seek here is to try and draw the line between the indicator in question and its measurability and whether or

not the patients are going to have better outcomes as a consequence”

Lesson #2Incorporating Evidence, Expertise & Patient

Perspectives

“The timing I’m not sure about, there’s no class one evidence to support it … At the same time I

think there is some value added to early management … prevention of certain secondary

complications … that’s why I advocate for it.”

Lesson #3Contextual Considerations & Variation

“…again this is going to be a local guidelines issue because different hospitals and systems will triage to their resuscitation room differently

depending on their volumes, …[this difference is] just going to influence the wording [of the

indicator ]…”  

Lesson #4Data Collection & Management

“… anything that encourages the whole trauma system to … improve the data that is submitted … if we’re going to have a measure that forces

the issue that we need better data collection, I’m all for it .”  

 

One Additional Lesson

Take Advantage of Potential Gaps / Opportunities Identified During

the Process

Missed Opportunity

“I discussed the paper with one other editor and we felt that it would be substantially strengthened if it had evidence of program outcomes of

these included”

Dr. Ginny BarbourChief Editor, PLoS Medicine

Captured Opportunities• No patient-centered quality indicators

Research program to develop patient-centered indicators

• Exclusive focus on high income countries Initiated an evaluation of trauma quality

improvement in low & middle income countries

AcknowledgementsMentors• Sharon Straus• David Bates• Bill Ghali• John Kortbeek• Tom Noseworthy• Don Redelmeier

Funding Agencies• CIHR• Alberta Innovates, Health

Solutions

Partners• Trauma Association of Canada• National Trauma Registry

Collaborators• Avery Nathens • Russell Gruen• Andrew Kirkpatrick• Morad Hameed• Anna Gagliardi• Sean Bagshaw• John Tallon

Research Team• Barbara Artiuch• Nancy Clayden• Jamie Boyd• Farah Khandwala• Maria Santana