Comparative hospital performance: new data, borrowed methods, more targeted analysis?

Professor Jonathan Karnon

Why do we measure hospital performance?

1. To ‘sack the Board’ Aggregate, or aggregated performance measure Under direct control of those being assessed

– Costs OR Process indicators Not so interested in magnitude of difference

– Identify general poor performance

2. To inform target areas for quality improvement Condition-specific Not necessarily under direct control

– Costs, outcomes, and processes Value of improvement important

Task: Improve population

health

Option 1: Fund new

technologies

Option 2: Improve use of

existing technologies

Show me the money

Increasing interest in quality improvement

1983 1988 1993 1998 2003 2008 20130

100

200

300

400

500

600

Year

Pu

bM

ed

"q

uali

ty i

mp

rovem

en

t" i

n

titl

e p

ap

ers

Not a new idea

“a programme of work not only to identify causes of variation at specific local level, but also to prioritise those variations and causes that have the most important impact on equity, effectiveness, efficiency and patient health outcomes”?

Variations in health care: The good, the bad and the inexplicable, The King’s Fund 2011

Outline

Evidence of variation; The information: costs, outcomes, and processes The incentives: potential strategies for using the information

Variation in process

RAND Corp (McGlynn et al, NEJM 2003)– 30 acute and chronic conditions and preventive care– 10 to 80% participants receiving recommended care

CareTrack (Runciman et al, MJA 2012)– 22 common conditions– 32 to 86% compliance with appropriate care

Variation in costs

Duckett & Breadon, Controlling costly care: a billion-dollar hospital opportunity, Grattan Institute 2014

$ per admission (2010/11)

Variation in outcomes

*adjusted by age, sex, indigenous origin and Diagnostic Group Hierarchical Condition Category (HCC) risk score

30-Day Readmission Rates by Area Health Service of Residence (NSW: 1 July 2005 – 30 June 2008)*

Policy/Practice Relevance?

Appropriate processes in theory ≈ cost-effective care in practice? – Costs? Timeliness?

Best outcomes? – Meaningful outcomes? At what cost?

Lowest cost? – With what outcomes?

Which providers are providing cost-effective care, and how are they doing it?

Using comparative condition-specific health service data

Systematic review– Feedback of comparative performance data alone does not work

Anecdotal– Service change works best when common recognition of a problem

Theory– Costs and post-discharge outcomes data demonstrates problems

Despite risk adjustment ‘my patients are sicker’ syndrome– Process data provides

Additional rationale for the existence of a problem Starting points for identifying solutions

11

Case study: ED chest pain presentations

Four public hospital in South Australia Clinical context, underlying diagnosis could be:

– ST Elevated MI– Non-ST Elevated MI – Unstable angina– Non-cardiac chest pain

Aims:– Identify benchmark performer(s) on basis of costs and outcomes– Assess the potential value of improved performance at non-

benchmark hospitals– Inform targets for investigation – variation in clinical pathways

12

The data

Clinical data extracted from common data warehouse– key procedures, pathology test results, movement between hospital

departments and wards, etc. – automated linkages to population-based mortality data.

Administrative data, linked to index events to identify other inpatient separation (episodes) at all South Australian hospitals– age, gender, and postcode (SEIFA), co-morbidities

13

Missing data

ED capacity– ED beds, personnel– Other presentations: rates and severity

Inpatient capacity– Bed occupancy rates

Cardiac Non-cardiac

– Staffing ratios

14

The dependent variables

Costs: Bottom-up patient-level costs available for all inpatient separations

Outcomes: 30 day/12 month related admission (unstable angina, MI, or stroke) or mortality

Process, not quality indicators– Pr(admission)– Time to admission– Pr(PCI | angiogram)– Inpatient LoS

15

The analysis

Separate multiple regression models fitted – Cost, outcome, and process variables

Hospital interaction terms tested – Identify patient sub-groups driving variation

Mean covariate values used to generate predicted outputs – Bootstrapping, stratified by hospital, to represent uncertainty

16

Patient characteristics

Hospital 1 Hospital 2 Hospital 3 Hospital 4 Difference (p value)

No. patients 1997 1527 2368 2058Age 60.2 62.5 59.1 57.9 <0.001Male 0.54 0.53 0.53 0.54 0.768SEIFA decile 5.4 4.0 6.0 2.6 <0.001Positive troponin test result 0.17 0.13 0.12 0.11 <0.001Existing circulatory disorder 0.37 0.33 0.37 0.33 0.002Cancer 0.03 0.03 0.03 0.02 0.189COPD 0.03 0.03 0.03 0.03 0.706Renal disease 0.07 0.08 0.09 0.07 0.398Diabetes 0.07 0.06 0.09 0.10 <0.001Dementia/Alzheimer's 0.01 0.02 0.01 0.01 0.716After hours presentation 0.61 0.62 0.59 0.58 0.082Weekend presentation 0.24 0.25 0.23 0.24 0.636

17

SEIFA – SocioEconomic Indicator For Areas: 1=lowest decile

Inpatient costs per presenting patient

Sub-group N Hosp 4 Hosp 1 - Hosp 4 Hosp 2 - Hosp 4 Hosp 3 - Hosp 4

All patients 7950 $2,868 $42 $630($307 to $989)

$510($299 to $713)

Existing circ., Out-of-hours 1706 $4,539 $229 $1,542

($786 to $2365)$510

($299 to $713)

No existing circ., Out-of-hours 1082 $1,579 -$451 $136

(-$73 to $361)$510

($299 to $713)

Existing circ., In-hours 3050 $4,640 $643 $1,265

($513 to $2111)$510

($299 to $713)

No existing circ., In-hours 2112 $1,610 -$6 -$91

(-$318 to $143)$510

($299 to $713)

NHosp 1 Hosp 2 Hosp 3 Hosp 4

Pr(event) RR (95%CI)

12m readmission or death

All patients 7950 0.045 1.59(1.27 - 1.93)

1.06(0.80 - 1.35)

1.16(0.91 - 1.47)

Young males 2281 0.043 1.59(1.27 - 1.94)

1.09(0.75 - 1.51)

1.16(0.91 - 1.47)

Old males 1979 0.102 1.54(1.25 - 1.85)

1.50(1.18 - 1.93)

1.14(0.91 - 1.43)

Young females 1701 0.025 1.61

(1.28 - 1.97)0.76

(0.47 - 1.20)1.16

(0.90 - 1.48)

Old females 1989 0.086 1.55(1.26 - 1.87)

1.09(0.81 - 1.42)

1.15(0.91 - 1.44)

12m mortality

All patients 7950 0.02 1.42 (0.93 to 2.05)

0.82 (0.47 to 1.23)

1.05 (0.83 to 1.2)

12m readmission

All patients 7950 0.03 1.72 (1.27 to 2.27)

1.22 (0.84 to 1.66)

1.30 (1.01 to 1.92)

C-E Acceptability Planes

value_avoiding_death_000s0.00

value_avoiding_readmission_000s

50.00

prho

sp_3

_cos

t_ef

fect

ive

100.000.00

50.00

100.00

0.00

0.17

0.35

value_avoiding_death_000s0.00

value_avoiding_readmission_000s

50.00

prho

sp_1

_cos

t_ef

fect

ive

100.000.00

50.00

100.00

0.02

0.39

0.76

value_avoiding_death_000s0.00

value_avoiding_readmission_000s

50.00

prho

sp_4

_cos

t_ef

fect

ive

100.000.00

50.00

100.00

0.16

0.56

0.97

value_avoiding_death_000s0.00

value_avoiding_readmission_000s

50.00

prho

sp_2

_cos

t_ef

fect

ive

100.000.00

50.00

100.00

0.00

0.01

0.03

Pr(Admitted)

NHospital 1 Hospital 2 Hospital 3 Hospital 4

Pr(admitted) RR (95%CI)

All patients 7950 0.77 0.87 (0.83 - 0.91)

0.92 (0.89 - 0.96)

0.92 (0.88 - 0.96)

Troponin +ive, Existing circulatory condition 842 0.97 0.99

(0.98 - 1.01)0.97

(0.94 - 0.99)0.97

(0.94 - 0.99)Troponin +ive, No existing circulatory condition

216 0.87 0.88 (0.79 - 0.95)

0.89 (0.78 - 0.97)

0.89 (0.78 - 0.97)

Troponin -ive, Existing circulatory condition 1946 0.89 0.98

(0.94 - 1.03)0.97

(0.95 - 0.99)0.97

(0.94 - 0.99)Troponin -ive, No Existing circulatory condition

4946 0.62 0.71 (0.65 - 0.77)

0.90 (0.84 - 0.96)

0.90 (0.83 - 0.96)

21

Pr(PCI | Angiogram)

22

NHospital 1 Hospital 2 Hospital 3 Hospital 4

Pr() RR (95%CI)

All patients 7950 0.14 2.40 (1.71 - 3.36)

1.79 (1.31 - 2.44)

1.57 (1.14 - 2.15)

Troponin +ive, Weekend 279 0.30 2.18

(1.36 - 3.44)1.07

(0.62 - 1.87)0.94

(0.54 - 1.53)Troponin +ive, Weekday 779 0.22 2.03

(1.44 - 2.79)1.22

(0.83 - 1.74)1.72

(1.26 - 2.28)Troponin -ive, Weekend 1632 0.17 2.76

(1.48 - 4.74)1.72

(0.96 - 2.93)0.93

(0.49 - 1.64)Troponin -ive, Weekday 5260 0.12 2.36

(1.56 - 3.41)2.02

(1.35 - 2.92)1.90

(1.32 - 2.64)

Length of Stay as inpatient

23

N Hosp4 (hrs) Hosp1 - Hosp4 Hosp2 - Hosp4 Hosp3 - Hosp4

All patients 5373 37.2 7.4(4.46 - 10.32)

4.5(0.79 - 8.38)

10.5(7.58 - 13.42)

Troponin +ive, Out-hours presentation 561 56.4 3.3

(-7.28 - 14.39)18.0

(2.52 - 34.19)16.3

(5.84 - 27.24)

Troponin +ive, In-hours presentation 325 51.1 7.0

(-3.44 - 17.45)13.8

(-1.55 - 29.42)15.2

(4.33 - 26.11)

Troponin -ive, Out-hours presentation 2702 33.8 6.5

(3.29 - 9.52)3.8

(-0.13 - 7.87)9.9

(6.50 - 13.03)

Troponin -ive, In-hours presentation 1785 35.3 10.2

(5.88 - 14.53)-0.4

(-5.02 - 4.19)8.7

(4.76 - 12.46)

How to compare larger numbers of providers?

Stratify by clinical process and identify best performing strata?– Hypothesis driven?– Process mining driven?

Group hospitals by performance and compare processes?– Empirical stratification

Informing action

“Publicising the existence of unwarranted variations and their causes does not guarantee that they will be tackled.”

“local health organisations… be required to publicly justify and explain in a consistent way their relative position on key aspects of health care variation.

…it may also be necessary to explore the development of harder-edged, locally focused incentives to encourage action to deal with unwarranted variation.”

Variations in health care: The good, the bad and the inexplicable, The King’s Fund 2011

The incentives

26

1. Sticksa) Public Reportingb) Mandated action plans

2. Carrotsa) Pay-for-Performance?b) External Services Improvement Fund

Mindful of the NHS Improvement Fund…

1b + 2b– Externally identified areas for improvement– Externally reviewed and supported applications to fund improvement

projects

Prioritisation criteria

27

Expected Value of Removing Variation (EVRV)– Hospital 2, 1527 patients per annum– Costs per patient could reduce by $630– Mortality could decrease by 1% (x $200,000?)– Readmissions decrease by 2% (x $50,000?)

Annual EVRV = 1527 x $(630 + 2000 + 1000) = $5.5 million x5? x10?

Zombies and Paradoxes

• “political paradox of rationing”

• “the appeal for transparency in medical decision-making is like a zombie, an idea that refuses to die despite its limited utility”

• Do the benefits of open and explicit quality improvement outweigh the: Financial costs and Political risks of unrealistic expectations?

Oberlander et al, Rationing medical care: rhetoric and reality in the Oregon Health Plan, CMAJ, 2001; 164(11):1583-7

Summary

29

Huge scope for service evaluation and improvement– Electronic data systems– Linkage facilities

Post-improvement evaluation: ICER estimates to inform the– Design of future improvement processes– Balance of spending on new technologies and existing services

Acknowledgements

Clarabelle Pham, Andrew Partington, Orla Caffrey, Jason Gordon, Brenton Hordacre, David Ben-Tovim, Paul Hakendorf, Maria Crotty

Funders: SA Health, NHMRC, HCF Foundation