Australian Masterclass

Australian Masterclass

Sally Batley Deputy Director of Analysis,

NHS Modernisation Agency (UK)

Working in partnership with the Patient Flow Collaborative (Victoria AU)

So what are we going to cover

Measurement for Improvement What is Statistical Process Control (SPC) Understanding Variation Benchmarking Build you own SPC charts



Measurement for Improvement

Sir Josiah Stamp

Public agencies are very keen on amassing statistics - they collect them, add them, raise

them to the Nth power, take the cube root and prepare wonderful diagrams. But ...

what you must never forget is that every one of those figures comes in the first instance from the village watchman (or admissions clerk?) -

who puts down what he damn pleases.

There are three kinds of lies:

lies, damned lies and statistics

After Mark Twain

Collecting your data

How good is your data?

Is the routine data you collect and distribute 100% accurate?

Is it complete rubbish? So it must be somewhere in between

Issues

Definitions Accuracy Consistency Timing

The information vicious circle

Information is not used

Information is:

InaccurateIncomplete

LateInconsistent

Task

In groups you have to describe the people in the room so answer these questions …

How many people are there in the room? How many are wearing something red? How many are tall? How many types of footwear are there? Find one word to describe the group?

Issues

Timing Definitions Accuracy Consistency

Data types

Types of data

Routine v special collection Qualitative v Quantitative Soft v hard Descriptive v numeric Example of current performance:

“Patients are satisfied” v waiting time is 4 hours Example of change:

“Communication with patients has improved” v Average X-ray waits reduced by 20 minutes

Which types are you collecting?

Types of measurement

Different types of measurements

measurements for judgement league tables

Performance Indicators

Measure probability not certainty Are better in groups Are better at identifying poorer performance Should not be used for league tables

Different types of measurements

measurements for diagnosis to show where the problems are lots of measurescomparative data useful

measurements for improvement to show if improvement are being made linked to the project objectives and aimsa few specific measures

Measurement for Improvement

or how do we know

that a change is an improvement?

What are we trying toaccomplish?

How will we know that achange is an improvement?What changes can we make

that will result in the improvements that we seek ?

Model for improvement

Act Plan

Study Do





Act Plan

Study Do

project aims





Act Plan

Study Do

project aims

global

measurements





Act Plan

Study Do

project aims

global

measurements

change principles

A

S

P

D

D

PA

S

D

PA

S

AP

DS

Data

Changes thatresult inimprovement

Building Improvement KnowledgeIm

pro

vem

ent

Time

Measurement for improvement:

Answers the question How do we know change is an improvement ?

Is linked to the project objectives or aims usually requires no more than five to seven

measures crosses the whole process of care measures change over time

Change areas, aims and measures should be related

Area - Effective Delivery of Health Care Aim - To improve access to the appropriate

treatment Measure - Reduce the number of days

between referral and first definitive treatment

Example from Action On

programme

Measuring quantitative outcomes

Measuring quantitative outcomes

A descriptive goal eg reduce DNAs

But by how much? Quantify the starting point (baseline) Set an objective (improve by x%) How will you measure that? (methods) Monitor progress

An example - hospital cancellations

Monitor progress to target

0

24

68

10

1214

16

0 1 2 3 4 5 6

Time

Pe

rce

nta

ge

Baseline = 15%

Target = 5%

How are we doing?Setting the baseline

Baseline period must be representative Small numbers issue Baseline period can be greater than

monitoring frequency

Over what period to measure baseline?

DNA rate with large variation

0%

5%

10%

15%

20%

25%

1 2 3 4 5 6

Time

Pe

rce

nta

ge

Average = 8.7%

Over what period to measure baseline?

DNA rate with small variation

0.0%

5.0%

10.0%

15.0%

20.0%

25.0%

1 2 3 4 5 6

Time

Pe

rce

nta

ge

Average = 8.7%

How will we know?Tips on measurement

Measurement periodsCensus point (particular time of day - eg 12pm)Period of time (eg 24 hour period)Don’t mix the two!

Use routine data where possible to allow cross-checking

Specify method preciselyeg process time in hours for patients from triage to

admission onto appropriate ward

How much will we improve? - Expressing the measurement of change

Be realistic in your expectationsDon’t think you can reduce error rate from 50% to 0%

Mostly express values to one decimal placeDNA rate = 5.6% (not 6%)

Express target as a value not as an improvementIf baseline is 5 patients/hour and you want to improve by 10%

then state target as 5.5 patients/hour

Avoid confusion over percentagesBaseline is 10% and you want to improve (reduce) by 25%

then state target as 7.5%

Process Mapping

Understand the process before settling on your measures

Route A - Self-referral

Arrival Arrival in A&Ein A&E

Seen by Seen by o/c teamo/c team

Seen by Seen by A&EA&E

TriageTriage DTADTA

Leave Leave A&EA&E

W1

W4

W5W2

W3

Indicative waits

W1 - 5 minutesW2 - by categoryW3 - 1 hourW4 - 1 hourW5 - 4 hours

Global measure: % patients seen within recommended waiting times at three key identified stages in care

We want to improve the overall patient journey

Global measure: % patients seen within recommended waiting times at three key identified stages in care

But Changes are made at specific points

The Measurement Paradox

We want to improve the whole patient experience/ journey but we make changes at specific points. How do we cope with

measuring the change?

Specific measurescan be temporary to monitor change ideas

Global measures are permanent to monitor overall improvement

Measurement at specific points

In addition to reported global measures plotted, additional measures may be required during changes:

specific measures related to the change results for sub-groups of patients results by consultant groups results for patients experiencing a particular

clinical process

Average waiting times across the care

pathway in days

0

1020

30

4050

60

Jan

Feb Mar Apr

May Ju

n Jul

AugSep

tOct

NovDec Ja

n

Change 1

Change 2

Change 3

Impact of changes on global measures (hopefully!)

Setting the baseline Or how are we doing right now?

Baseline period must be representative Watch out for small numbers! Baseline period can be greater than

monitoring frequency

Measurement guidelines

key measures plotted and reported each month should clarify your project team’s aim and make it tangible.

be careful about over-doing process measures.

consider sampling to obtain data. integrate measurement into the daily routine. plot data on the key measures each month

during the programme

Task: Creating measures for your project aims

Your Project is Improving Patient Flow what is your measurement strategy? what are you aims what quantified measures could be used?

Data collection method what baseline are you going to use? what is the potential performance? frequency of measurement? How are you going to feed it back and to

whom?

Patient experience monitoring

Why?

To use patient feedback to improve services

Agenda

evaluating patient experience quantitative versus qualitative rating versus reporting practical hints and tips

Task

On your table, brainstorm ideas for measuring and monitoring patients’ experience with a service:

How can we measure what patients think of the service?

Approaches to monitoring

quantitative•structured questionnaires•“tick box” surveys

qualitative•semi-structured interviews•questionnaires that combine “tick box” with comment spaces

•unstructured interviews•patient focus groups•critical incident technique

Report experience don’t rate satisfaction

How satisfied were you with the consultation you received with the doctor?

Please answer the questions by ticking the response which most closely matches your experience.

All the treatment options were fully explained to me. I was given as much as much information as I wanted to know Treatment options were very briefly discussed with me The doctor did mention different treatments, but I did not really understand I did not feel that I was given a choice about treatment

very satisfiedquite satisfiedsatisfiedquite dissatisfiedvery dissatisfied

X

Designing a questionnaire or survey

goal of the research research methodquestionnaire designpatient sample frequency of data collectiondata collection methodssystems for analysis reporting systems

What do you want to know?

How will you find out?

What sort of questions?

How many will you ask?

How often will you ask them?

How will you ask them?

How will you analyse the data?

How will you report the results and

to whom?

Designing a questionnaire or survey

keep it simple plain English small patient sample and track changes over

time, little and often (run chart) combine quantitative and qualitative pilot first involve patient / user representatives in

questionnaire design, data collection and analysis of results

Leave room for comments

How satisfied were you with the consultation you received with the doctor?

Please answer the questions by ticking the response which most closely matches your experience.

All the treatment options were fully explained to me. I was given as much as much information as I wanted to know Treatment options were very briefly discussed with me The doctor did mention different treatments, but I did not really understand I did not feel that I was given a choice about treatment

Add any other comments you wish to make in the box below

The power of a good quote

“The best thing was getting the date for the operation, I was given a date that suited me and was given the letter to show my boss .”

“Everything was completed in one morning, I saw the Consultant went to pre-assessment and got my surgery date, this meant that I did not

have to take further time off work”

Back to you measurement strategy…



What do you know?

Task

What do you know about the following: Mean Variation Special causes Standard deviation

What is SPC?

P is for ProcessWe deliver our work through processes

S is for Statisticalbecause we use some statistical concepts to help

us understand our processes

C is for ControlAnd by this we mean predictable

What is SPC for?

A way of thinking Measurement for improvement - a simple tool

for analysing data Better way for making decisions Evidence based management Easy, sustainable

What Can It Do For Me? To identify if a process is sustainable

are your improvements sustaining over time

To identify when an implemented improvement has changed a process and it has not just occurred by chance

To understand that variation is normal and to help reduce it

To understand processes - This helps make better predictions and improves decision making

What about this?

Where have we come from?

Compare to some arbitrary fixed point in the past the average (median) waiting time of those on the

list, at 2.97 months, fell slightly over the month, and remains lower than at March 1997 (3.04 months).

Show percentage change this month and to some arbitrary fixed point in the past the number of over 12 month waiters fell this

month by 3,800 (7.4%) to 48,100, and are now 24,000 (33%) below the peak at June 1998

Comparing this year to last year

Delayed Discharges (w eekly Sitreps)

0

1000

2000

3000

4000

5000

6000

7000

W eeks from October

No. o

f del

ayed

di

scha

rges

2000/012001/02

Waiting time performance

2000Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec85 76 83 58 62 80 53 71 64 82 55 78

2001Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec39 19 31 22 25 51 40 11 31 54 28 16

What can you tell me about the following data?

Is this better?

Average wait in days

0102030405060708090

Jan Mar May Jul Sep Nov Jan Mar May Jul Sep Nov

Or better still?

Average wait in days

0

20

40

60

80

100

120

Jan Mar May Jul Sep Nov Jan Mar May Jul Sep Nov

Common management reactions to data

take 3 different numbers6 possible (& random) sequences

"Upward Trend"?

0

1

2

3

4

1 2 3

"Setback"?

0

1

2

3

4

1 2 3

"Downturn"?

0

1

2

3

4

1 2 3

"Turnaround"?

0

1

2

3

4

1 2 3

"Rebound"?

0

1

2

3

4

1 2 3

"Downward Trend"?

0

1

2

3

4

1 2 3

3 points can give 6 possible (& random) sequences

Unacceptable decision-making

Develop polite impatience with guesswork - single figure decision making shooting from the hip anecdotal data debate “known” solutions ?arbitrary targets and standards

What else can it do for me?

Recognise variation Evaluate and improve underlying process

is it stable? can it meet “targets”?

Help drive improvement has the process really improved or is it just chance? is it sustainable?

Prove/disprove assumptions and (mis)conceptions Use data to make predictions and help planning Reduce data overload

What is a control chart

Upper process limit

Mean

Lower process

limit

0

10

20

30

40

50

60

70

80

F M A M J J A S O N D J F M A M J J A S O N D



What is Benchmarking?

Benchmarking compares practice and performance across organisations in order to identify ways to improve

It is in essence, the identification, understanding, dissemination and implementation of best practice

Benchmarking encompasses…

Regular comparison of aspects of performance (functions and processes) with different practitioners

Identifying gaps in performance Seeking fresh approaches to bring about

improvements in performance Following through with implementation of

improvements Monitoring progress and reviewing the benefits

Why is Benchmarking important?

Benchmarking can be used to improve the overall performance of organisations through sharing and developing different practices

What are the benefits of Benchmarking?

Improving quality and productivity Improving performance measurement Learning from others and greater confidence

in developing and applying new approaches Greater involvement and motivation of staff

Comparing performance of different people or services

Measuring for judgement

The minister has decided that prescribing aspirin for patients on the CHD register is a Good Thing

Non-compliance will henceforth be a hanging offence

But who to hang? He has been given the latest data on several

Health Services

Who’s doing well?

The % patients on CHD register who are being treated with aspirin

February 2002

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

A B C D E F G H I J K

Average

Gold stars to Health Services A & B

Hanging for Health Services I,

J & K

Why not traditional?

Remember who’s doing well?

The % patients on CHD register who are being treated with aspirin

February 2002

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%


Average

Gold stars to Health Services A & B

Hanging for Health Services I,

J & K

A different way of presenting it

The % patients on CHD registerwho are being treated with aspirin

February 2002

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%


Average

Control limits added

The % patients on CHD registerwho are being treated with aspirin

February 2002

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%


AverageLowerUpper



What is Variation?

Everything varies - no two things are alike

Recognising this is a start but not enough: must understand it’s effect on customers and then manage it as appropriate

Task

In pairs think of reasons why your journey driving to work may be delayed on a morning. – Write on post its

You have a few mins and we’ll come back to this later.

Different Types of Variation

Common Cause = Stable in time & therefore relatively predicatable

For example traffic lights which hold us up today would probably hold us up in the next week.

Different Types of Variation

Special Cause = Irregular in time and therefore unpredictable.

For Example a police convoy escorting a wide load

Practical interpretation of the Standard Deviation

Mean - 3s

Mean

Mean + 3s

3s and the Control Chart

6s

3s

3s

UCL

LCL

Mean

Reducing Variation

Walter Shewhart - Statistician 1920’s Bell Telephones: every failure led to an

alteration to the telephones. Good idea? Started to look at limits and Common &

Special Causes

“A phenomenon will be said to be controlled when, through the use of past experience, we can predict, at least within limits, how the phenomenon may be expected to vary in the future”

Shewart - Economic Control of Quality of Manufactured Product, 1931

Task

Back to the Task-Journey to work

Which are common causes of variation? And which are special causes?

0

20

40

60

80

100

120

Consecutive trips

Min.

My trip to work

average

Accident on motorway

tyre had puncture

Borrowed helicopter

Stopped by police for speeding

School holidays

COMMON CAUSE VARIATION - Points within the yellow lines is variation you would expect - normal variation of the process (my trip to work) E.G. traffic lights, pedestrians, rush hour

CONTROLLED VARIATION

stable,consistent pattern of variation

“chance”/constant causes

0

10

20

30

40

50

60

70

80


Upper process

limit

Mean

Lower process

limit

UNCONTROLLED VARIATION

•pattern changes over time

•“assignable”/special causes

0

20

40

60

80

100


2 Ways to improve a process

If controlled variationprocess is stable and predictablevariation is inherent to process therefore, process must be changed

If uncontrolled variationprocess is unstable and unpredictablevariation caused by factor(s) outside processcause should be identified and “sorted”

2 dangers to beware of

Reacting to special cause variation by changing the process

Ignoring special cause variation by assuming “its part of the process”

Pause:

Think of some examples in your own area:

- Common cause variation- Special cause variation


Measurement for Improvement What is Statistical Process Control (SPC)-the

math Understanding Variation Benchmarking Build you own SPC charts

How to interpret the results

Rules for special causes

RULE 1 Any point outside one of the control limits

RULE 2 A run of seven points all above or all below the centre line, or all increasing or all decreasing. RULE 3 Any unusual pattern or trends within the control limits.

RULE 4 The number of points within the middle third ofthe region between the control limits differs markedly from two-thirds of the total number of points.

XX

X

X

X

X

X

X

X

LCL

UCL

MEAN

X

X

X

X

XX

X

X

X

X

LCL

UCL

MEAN

X

Point above UCL

Point below LCL

SPECIAL CAUSES - RULE 1





MEAN MEAN

Seven points above centre line


LCL

UCL

LCL

UCL

XX

X

X

X X

X

XX

XX X

X

XX

X X

X

XX

X

Seven points below centre line

MEAN MEAN

Seven points in a downward direction


LCL

UCL

LCL

UCL

XX

XX

X

XX

X

X X

X

XX X

XX

XX

X

X

X

Seven points in an upward direction






X

X

X

X

X

X

XX X

X

X

X

X

X

X

X

X

X

X

X

Cyclic pattern

X

X X

XX

XX

X

X

X

X

X

X

X

X X

X

X

XLCL

UCL

LCL

UCL

Trend pattern






Considerably less than 2/3 of all the points fall in this zone

X

XX X X

X

X

X

X

X

XX

XX

X

XX

LCL

UCL

X

X

X

X

X

X

XX

X

X

X

XX

X

XX

X

X

XX

X X

X

X

XX

LCL

UCL

Considerably more than 2/3 of all the points fall in this zone

NOW FOR SOME MATHS!

Use individual values to calculate the Mean

Difference between 2 consecutive readings, always positive = Moving Range, MR

Calculate the Mean MR

One standard deviation/sigma = (Mean MR) ÷ d2 * s or σ

Upper Process Limit (UPL) = Mean + 3 s

Lower Process limit (LPL) = Mean - 3 s

* d2 is a constant for given subgroups of size n (n = 2, d2 = 1.128)

H.L. Harter, “Tables of Range and Studentized Range”, Annals of Mathematical Statistics, 1960.

Construction and Interpretation of (X, Moving

R) Chart

Run chart, running record, time order sequenceCalculate the meanCalculate upper and lower process limitsInterpret the chart for process controlFind the causes of real change & act to improve

Calculation of the mean

Σ means “ sum of ”

Mean =

= X1 + X2 + X3 + X4 + X5 + X6 + X7 + X8 + … + X19 + X20 2

0

X

= 5.9 + 0.4 + 0.7 + 4.7 + 2 + 1.3 + 0.8 + … + 2

20Mean = 2.545

X nXΣ=

X19 X20

1.5 2

SPC 33

n = number of results

X1 X2 X3 X4 X5 X6 X7 X8

5.9 0.4 0.7 4.7 2 1.3 0.8 0.7

= 50.9

20

Calculation of mean moving range

Σ means “ sum of ”

Moving Range

= R1 + R2 + R3 + R4 + R5 + R6 + R7 + R8 + … + R19

19

MR

= 5.5 + 0.3 + 4 +2.7 + 0.7+0.5 + 0.1 + 1.8 +0 + 0.8 +0.7 + 3.7 + 0.5 + 3.8 + 0.1 + 0.2 + 0.6 +0.8 + 0.5

19

= 1.437 MR nRΣ=

R18

0.8 0.5

SPC 33

n = number of moving ranges

R1 R2 R3 R4 R5 R6 R7 R8

5.5 0.3 4 2.7 0.7 0.5 0.1 1.8

= 27.3

19

R19

= 27.3

19

R

Calculate

= 1.437

1.128

SPC 33

= 1.274

Calculation of σ = 1 standard deviation

From the formula R

d2

=

d2 is always 1.128 for a sample size of 2 (difference between 2 values)

Never use the standard

deviation key on a

calculator to get this figure

σ

Calculation of control limits

Calculate UCLX (Upper Control Limit) for X

SPC 33

Calculate LCLX (Lower Control Limit) for X

= X + 3 0

= X - 3 0

= 2.545 + 3.822

= 6.367 Plot on graph

= 2.545 - 3.822 = -1.277

can’t have negative so take to be 0 Plot on graph

And that’s how you get one of these!

- a Control ChartUpper process limit

Mean

Lower process

limit

0

10

20

30

40

50

60

70

80


Things to remember

only need 20 data points to set up a control chart “standard deviation”

this is not the one used in formulae in Excel or on calculators.

d2 constant sample size of 2 refers to the sample size for moving range

(which is nearly always 2) - NOT the number of data points

20 data points produces 19 moving ranges

Remember the 2 ways to improve a process

If controlled variation process is stable variation is inherent to process therefore, process must be changed

If uncontrolled variation process is unstable variation is extrinsic to process cause should be identified and “treated”

CONTROLLED VARIATION

stable,consistent pattern of variation

“chance”/constant causes

0

10

20

30

40

50

60

70

80


Upper process

limit

Mean

Lower process

limit

Remember the 2 ways to improve a process

If controlled variation process is stable variation is inherent to process therefore, process must be changed

If uncontrolled variation process is unstable variation is extrinsic to process cause should be identified and “treated”

UNCONTROLLED VARIATION

•pattern changes over time

•“assignable”/special causes

0

20

40

60

80

100


DEFINING LACK OF CONTROL

A single point falls outside the 3-sigma control limits

2 out of 3 successive values fall on the same side of, and more than 2-sigma units from the central line

4 out of 5 successive values fall on the same side of, and more than 1-sigma unit from the central line

8 (or 7??) successive values fall on the same side of the central line, or all increasing or all decreasing

We live in a world filled with variation - and yet there is very little recognition or understanding of variation

WILLIAM SCHERKENBACH

Variation



SPC Spreadsheet Formulae

A B C D E F GDate Field

Data Average Moving Range

Average Moving Range

Lower Control Limit

Upper Control Limit

=AVERAGE(B2:B10)

=ABS(B3-B2) =AVERAGE(D3:D10)

=MAX(0,C2-

(3*(E2/1.128)))

=C2+(3*(E2/1.128))

Average of all the data list

The difference between

consecutive numbers

Average of the moving range

list

Average minus 3

multiplied by Average

moving range divided by

1.128

Average plus 3 multiplied by Average

moving range divided by

1.128

Example Data SetAdmissionsInpatients Average MR Average MRLower LimitUpper limit01-Feb-02 20 21.90 13.44444 0 57.656502-Feb-02 6 21.90 14 0 57.656503-Feb-02 14 21.90 8 0 57.656504-Feb-02 46 21.90 32 0 57.656505-Feb-02 41 21.90 5 0 57.656506-Feb-02 32 21.90 9 0 57.656507-Feb-02 40 21.90 8 0 57.656508-Feb-02 9 21.90 31 0 57.656509-Feb-02 2 21.90 7 0 57.656510-Feb-02 9 21.90 7 0 57.6565

Admissions Inpatients Average MR Average MR Lower Limit Upper limit20 =AVERAGE(B2:B11) =AVERAGE(D3:D11) =MAX(0,C2-(3*E2/1.128)) =C2+(3*(E2/1.128))6 21.9 =ABS(B2-B3) 0 57.656501182033114 21.9 =ABS(B3-B4) 0 57.656501182033146 21.9 =ABS(B4-B5) 0 57.656501182033141 21.9 =ABS(B5-B6) 0 57.656501182033132 21.9 =ABS(B6-B7) 0 57.656501182033140 21.9 =ABS(B7-B8) 0 57.65650118203319 21.9 =ABS(B8-B9) 0 57.65650118203312 21.9 =ABS(B9-B10) 0 57.65650118203319 21.9 =ABS(B10-B11) 0 57.6565011820331

Table 1. Shows what the data should look like.

Table 2. Shows how the formula should look.

Average, Lower limit and Upper limit should only have the formula in the first row and the value pasted for the entire dataset.

Example SPC Chart

010203040506070

I npatients

Average

Lower Limit

Upper limit

Within this process Trust x could expect to see between 0 and 58 admitted Inpatients per day, with and average of 22. Therefore, there needs to be 58 inpatient beds available everyday to match current demand.

Task

Split into equal groups around each laptop At least one analyst in each Let someone use the computer who is not

use to working with excel Others can coach them on how to use it You have a data file on your computers

called example.xls Compose a SPC chart and feedback

That’s all Folks !!!Any Last Questions?

Useful references Donald Wheeler. Understanding Variation. Knoxville: SPC

Press Inc, 1995 Walter A Shewhart. Economic control of quality of

manufactured product. New York: D Van Nostrand 1931. American Society for Quality

www.asq.org/about/history/shewhart.html WE Deming. Out of the crisis. Massachusetts: MIT 1986 Donald Wheeler. Advanced topics in statistical process control.

The power of Shewhart's charts. Knoxville: SPC Press Inc, 1995 Donald M Berwick. Controlling variation in health care: a

consultation from Walter Shewhart. Med Care 1991; 29: 1212-25.

Date post:	12-Jan-2016
Category:	Documents
Upload:	juan
View:	26 times
Download:	0 times

Australian Masterclass

Documents