1
Building a Measurement System That Works
Session B713 April 20161330 – 1500
Robert Lloyd, Ph.D.Vice President, Institute for Healthcare Improvement
Gary Sutton, BSc (Honors) Statistician
International Forum on Quality and Safety in Healthcare
Gothenburg Sweden
The presenters have
nothing to declare
IHI Faculty(bio sketches can be found at the end of this presentation)
Gary Sutton, BSc (Honors) Scottish Government, Statistician
Improvement Advisor; Scotland
@scotgov_ia
Robert Lloyd, PhDVice President, Institute for Healthcare Improvement
@rlloyd66
2
and the
3
1. What is your current level of knowledge about quality measurement?
2. What is your motivation for measuring?
3. Do you know the milestones in the Quality Measurement Journey (QMJ)?
4. Do you understand variation conceptually?
5. Do you understand variation statistically?
6. How well do you link measurement to improvement
Our Objectives for today…To answer Six Key Questions on Measurement
Question #1What is your current level of knowledge about
quality measurement?
This self-assessment is designed to help quality facilitators and improvement team
members gain a better understanding of where they personally stand with respect to
the milestones in the Quality Measurement Journey (QMJ). What would your
reaction be if you had to explain why is it preferable to plot data over time rather than
using aggregated statistics and tests of significance? Can you construct a run chart
or help a team decide which measure is more appropriate for their project?
You may not be asked to do all of the things listed below today or even next week.
But if you are facilitating a QI team or expect to be able to demonstrate
improvement, sooner or later these questions will be posed. How will you deal with
them?
The place to start is to be honest with yourself and see how much you know about
concepts and methods related to the QMJ. Once you have had this period of self-
reflection, you will be ready to develop a learning plan for yourself and those on your
improvement team.
Source: R. Lloyd, Quality Health Care: A Guide to Developing and Using
Indicators. Jones & Bartlett Publishers, 2004: 301-304.
3
ExerciseMeasurement Self-Assessment
Select the one response which best captures your opinion:
1. I'd definitely have to call in an outside expert to explain and apply
this topic.
2. I’ve heard of this topic but I would not feel comfortable applying it to
a team’s work.
3. I am familiar with this topic but would have to study it further before I
felt comfortable explaining it to a team.
4. I have knowledge about this topic and feel confident that I could help
a team apply it to their improvement efforts but I would not want to
stand up and teach this to a large group.
5. I consider myself an expert in this area and could apply easily to a
team’s work as well teach this topic to large groups.
Source: R. Lloyd, Quality Health Care: A Guide to Developing and Using
Indicators. Jones & Bartlett Publishers, 2004: 301-304.
Exercise: Measurement Self-AssessmentSource: R. Lloyd, Quality Health Care: A Guide to Developing and Using Indicators.
Jones & Bartlett Publishers, 2004: 301-304.
Measurement Topic or SkillResponse Scale
1 2 3 4 5
Help people in my organization determine why they are measuring (improvement, judgment or research)
Move teams from concepts to specific quantifiable measures
Building clear and unambiguous operational definitions for our measures
Develop data collection plans (including stratification and sampling strategies)
Explain why plotting data over time (dynamic display) is preferable to using aggregated data and summary
statistics (static display)
Explain the differences between random and non-random variation
Construct run charts (including locating the median)
Explain the reasoning behind the run chart rules
Interpret run charts by applying the run chart rules
Explain the statistical theory behind Shewhart control charts (e.g., sigma limits, zones, special cause rules)
Describe the basic 7 Shewhart charts and when to use each one
Help teams select the most appropriate Shewhart chart for their measures
Describe the rules for special cause variation on a Shewhart chart
Help teams link measurement to their improvement efforts
4
QualityBetter
Old WayQuality Assurance
(Data for judgment))
QualityBetter Worse
New Way(Quality Improvement)
Action taken on all occurrences
Reject
defectives
Question #2What is your motivation for measuring?
Source: Robert Lloyd, Ph.D., 2009.
Requirement,Specification or Target
No action
taken
here
Worse
Source: Provost, Murray & Britto (2010)
Example of Data for Judgement
5
Slide #9
Slide #9
How Is Error Rate Doing?
Source: Provost, Murray & Britto (2010)
Slide #10
Slide #10
How is Perfect Care Doing?
Source: Provost, Murray & Britto (2010)
6
20-20 Hindsight
“Managing a process on the basis of monthly (or quarterly) averages is like trying to drive
a car by looking in the rear view mirror.”
D. Wheeler Understanding Variation, 1993.
If you are serious about your quality improvement efforts, you should be collecting and analyzed data as close to the production of work as possible.
• What would it take to collect data on individual patients waiting to see the doctor?
• To track the number of patients being assessed for pressure ulcers each day?
• The percent of “did not attend” appoints for each week?
• Most measures can be collected more frequently than monthly!
Question #3Do you know the milestones in the Quality
Measurement Journey (QMJ)?
7
13
AIM (How good? By when?)
Concept
Measure
Operational Definitions
Data Collection Plan
Data Collection
Analysis ACTION
Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett Publishers, 2004.
Milestones in theQuality Measurement Journey
© 2015 Institute for Healthcare Improvement/R. Lloyd
AIM – reduce patient falls by 37% by the end of the year
Concept – reduce patient falls
Measures – Inpatient falls rate (falls per 1000 patient days)
Operational Definitions - # falls/inpatient days
Data Collection Plan – monthly; no sampling; all IP units
Data Collection – unit submits data to Quality
Improvement Dept. for analysis
Analysis – control chart ACTION
Milestones in theQuality Measurement Journey
8
15
AIM (How good? By when?)
Concept
Measure
Operational Definitions
Data Collection Plan
Data Collection
Analysis ACTION
Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett Publishers, 2004.
Milestones in theQuality Measurement Journey
Moving from a Concept to Measure
“Hmmmm…how do I move from a concept
to an actual measure?
Every concept can have MANY measures.
Which one is most appropriate?
9
Concept Potential MeasuresHand Hygiene Ounces of hand gel used each day
Ounces of gel used per staffPercent of staff washing their hands
(before & after visiting a patient)Percent of inpatients with C.Diff
Patient Falls Percent of patients who fell Fall rate per 1000 patient daysNumber of fallsDays between a fall
Employee Evaluations Percent of evaluations completed on time
Number of evaluations completed
Variance from completion due date
Every concept can have many measuresSource: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett, 2004.
© 2015 Institute for Healthcare Improvement/R. Lloyd18
Three Types of Measures� Outcome Measures
� Point to qualities that stakeholders value (voice of the customer)
� Is this system meeting the needs of those who care about its operation?
� Is our improvement work making a meaningful impact?
� Process Measures� Voice of the process.
� Are the parts/steps in the system performing as planned? Are processes reliable? Efficient? Patient-Centered?
� Are we on track to influence the Outcome measure(s)?
� Balancing Measures� Are we producing unintended consequences in our efforts to improve?
� What other factors may be affecting results?
� Looking at a system from different directions/dimensions.
� What happened to the system as we improved the outcome and process measures?
10
Provost, L.P. & Murray, S.K. (2011). The health care data guide: Learning from data for improvement. San Francisco: Jossey-Bass.
Types of Measures Description
The Surgical Sight Infection Family pf Measures
Outcome The voice of the customer or
patient. How is the system
performing? What is the result?
Surgical Sight Infection Rate
Process The voice of the workings of the
process. Are the parts or steps in
the system performing as planned.
Percentage of appropriate
prophylactic antibiotic selection.
Percentage of on time administration
of prophylactic antibiotics.
Percentage of a safety climate score
great than 4.
Balancing Looking at a system from different
directions or dimensions. What
happened to the system as we
improved the outcome and
improvement measures?
Patient satisfaction
Cost per case
Expectations for Improvement
% Compliance with process
% Compliance with process
Outcome measure - % reduction in?
When will my data start to move?
• Process measures will start to move first
• Outcome measures will most likely lag behind process measures
• Balancing measures – just monitoring – not looking for
movement (pay attention if there is movement)
11
Balancing Measures:Looking at the System from Different Dimensions
� Outcome (quality, time)
� Transaction (volume, no. of patients)
� Productivity (cycle time, efficiency, utilisation, flow,
capacity, demand)
� Financial (charges, staff hours, materials)
� Appropriateness (validity, usefulness)
� Patient satisfaction (surveys, customer complaints)
� Staff satisfaction
Balancing measures help keep you from sub-optimizing the system!
22Balancing Measures also help you be aware of Unintended Consequences
12
© 2015 R. C. Lloyd & Associates
1. A starting point for any QI project is to move from concepts to
measures that appropriately capture the concepts of interest.
2. Use the Organizing Your Measures Worksheet on the next
page to start this part of your journey.
3. List the concepts of interest in the far left column. Then identify
potential measures for these concepts in the second column.
Remember that a single concept might have more than one
potential measure.
4. Finally, indicate whether each potential measure is an
Outcome, Process or Balancing measure.
ExerciseOrganizing your Measures
© 2015 R. C. Lloyd & Associates
Concept Potential Measure(s) Outcome Process Balancing
Organizing Your Measures Worksheet
Topic for Improvement:
Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett, 2004.
13
Concept Potential Measure(s) Outcome Process Balancing
Patient Harm Inpatient falls rate
Patient Harm Number of falls
Compliance Percent of inpatients assessed for falls
Staff Education Percent of staff fully trained in falls assessment protocol
Assessment Time
The additional time it takes to conduct a proper falls assessment
ExampleOrganizing Your Measures Worksheet
Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett, 2004.
Topic for Improvement: Inpatient Falls
Conclusions:Moving from a Concept to a Measure
1. Moving from concept to measures requires focused work
to create agreement about adjectives such as recovery,
major, timely, complete, accurate or excellent.
2. A concept may need more than one measure and,
therefore, the development of more than one operational
definition.
3. The transition from concept to measure doesn’t just
happen, it requires both technical and clinical decision-
making to be blended with pragmatism and acceptance of
the imperfections of the measures.
14
27
AIM (How good? By when?)
Concept
Measure
Operational Definitions
Data Collection Plan
Data Collection
Analysis ACTION
Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett Publishers, 2004.
Milestones in theQuality Measurement Journey
28An Operational Definition...
… is a description, in quantifiable terms, of what to measure and the steps to follow to measure it consistently.
� It gives communicable meaning
to a concept
� Is clear and unambiguous
� Specifies measurement
methods and equipment
� Identifies detailed criteria for
inclusion and exclusion.
� Provides guidance on sampling
Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett Publishers, 2004.
15
© 2015 Institute for Healthcare Improvement/R. Lloyd29
From a food science perspective, it is difficult to define a food product that is 'natural' because
the food has probably been processed and is no longer the product of the earth. That said,
FDA has not developed a definition for use of the term natural or its derivatives. However, the
agency has not objected to the use of the term if the food does not contain added color,
artificial flavors, or synthetic substances.
What is the definition of 'natural' on the label of food?
September 23, 1999An expensive operational definition
problem!
NASA lost a $125 million Mars orbiter because one engineering team used metric units (newton-seconds) to guide the spacecraft while the builder (Lockheed Martin) used pounds-second to calibrate the maneuvering operations of the craft.
Information failed to transfer between the Mars Climate Orbiter spacecraft team at Lockheed Martin in Colorado and the mission navigation team in California. The confusion caused the orbiter to encounter Mars on a trajectory that brought it too close to the planet, causing it to pass through the upper atmosphere and disintegrate.
16
How do you define the following healthcare concepts?
• Medication error• Co-morbid conditions• A healthy life style• Cancer waiting times• Health inequalities• Asthma admissions• Childhood obesity• Patient education• Health and wellbeing• Adding life to years and years to life• Children's palliative care • Safe services• Smoking cessation• Urgent care• Complete history & physical• Surgical Screening
• Delayed discharges
• End of life care
• Falls (with/without injuries)
• Childhood immunizations
• Complete maternity service
• Patient engagement
• Moving services closer to home
• A well-baby visit
• Ambulatory care
• Access to health in deprived areas
• Diagnostics in the community
• Productive community services
• Vascular inequalities
• Breakthrough priorities
ExampleMedication Error Operational Definition
Measure Name: Percent of medication errors
Numerator: Number of outpatient medication orders with one or more errors. An error is defined as: wrong med, wrong dose, wrong route or wrong patient.
Denominator: Number of outpatient medication orders received by the family practice clinic pharmacy.
Data Collection:
• This measure applies to all patients seen at the clinic
• The data will be stratified by type of order (new versus refill) and patient age
• The data will be tracked daily and grouped by week
• The data will be pulled from the pharmacy computer and the CPOE systems
• Initially all medication orders will be reviewed. A stratified proportional random sample will be considered once the variation in the process is fully understood and the volume of orders is analyzed.
17
• Select one measure for your project (outcome or process) and write a clear and specific Operational Definition.
• Is the measure clearly defined? If you gave the definition of your measure to another person would they know precisely what you are attempting to measure?
• Are you clear about the name of the measure, what is to be included and the measurement steps required to obtain data?
• Use the Operational Definition Worksheet to record your responses.
ExerciseBuilding Your Operational Definitions
34
Measure Name: ________________________________________(Remember this should be specific and quantifiable, e.g., the time it takes to…,the number
of…, the percent of… or the rate of…)
Operational DefinitionDefine the specific components of this measure. Specify the numerator and denominator if
it is a percent or a rate. If it is an average, identify the calculation for deriving the average.
Include any special equipment needed to capture the data. If it is a score (such as a patient
satisfaction score) describe how the score is derived. When a measure reflects concepts
such as accuracy, complete, timely, or an error, describe the criteria to be used to determine
“accuracy.”
Operational Definition Worksheet
See Appendix D for a detailed Operational Definition worksheet
18
� Operational definitions are not universal truths!
� Operational definitions require agreement on terms, measurement methods and decision criteria.
� Operational definitions need to be reviewed periodically to make sure everyone is still using the same definitions and that the conditions surrounding each measure have not changed.
Conclusions:Developing Operational Definitions
36
AIM (How good? By when?)
Concept
Measure
Operational Definitions
Data Collection Plan
Data Collection
Analysis ACTION
Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett Publishers, 2004.
Milestones in theQuality Measurement Journey
19
Stratification
• Separation & classification of data according to predetermined categories
• Designed to discover patterns in the data
• For example, are there differences by shift, time of day, day of week, severity of patients, age, gender or type of procedure?
• Consider stratification BEFORE you collect the data
Common Stratification Levels
� Day of week
� Shift
� Severity of patients
� Gender
� Type of procedure
� Unit
� Age
What stratification
levels are appropriate
for your data?
20
Sampling
When you can’t capture data on the
entire population (an enumeration), you can estimate its characteristics
by sampling.
© Richard Scoville & I.H.I.
© 2015 Institute for Healthcare Improvement/R. Lloyd
Options for Sampling
Non-probability Sampling Methods
• Convenience sampling
• Quota sampling
• Judgment sampling
Probability Sampling Methods
• Simple random sampling
• Stratified random sampling
• Stratified proportional random sampling
• Systematic sampling
• Cluster sampling
Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett Publishers, 2004.
21
0
20
40
60
80
100
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
Sample Size = 1 per week
-10
10
30
50
70
90
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
Sample Size = 5 per week
-10
10
30
50
70
90
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
Sample Size = 10 per week
-10
10
30
50
70
90
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
Sample Size = 20 per week
-10
10
30
50
70
90
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
Sample Size = 50 per week
Sample Size will affect your ability to
detect a change
Provost. L. & Murray. (2008, November) The Date Guide: Learning from Data to Improve Health Care. Austin, TX: API.
Tips for building an effective measurement system
� Seek useful measures not perfection
� Think about stratification
� Use sampling (when appropriate)
� Integrate measurement into daily routine
� Collect qualitative and quantitative data
� Plot data over time
22
Integrate measures into daily routineand make it easy:
(example of attendance at a restaurant)
What measure could you collect in this way?
• Number of falls
• Number of pressure ulcers
• Number of cancelled appointments
• Number of patients coming to clinic
• Number of medication errors
• Number of staff evaluations not completed on time
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd
1. Sampling should produce representative and workable numbers for the unit of
interest.
2. Customers providing feedback about their service or the care they receive can
be very susceptible to sampling bias.
3. Sampling bias can be introduced if you always use the same place or time to
gather data and this location or time period (e.g., Tuesday afternoon in the
clinic at 1330) is not representative of the whole. This is a major problem when
single point in time audits are relied on as the sampling method.
4. When conducting surveys, recall bias occurs if the questions rely on the
individual’s to “think back and recall how they felt about their hospital stay.”
5. The worst case scenario occurs when you have no idea where the sample
came from or how representative it is of the population or organisation overall.
6. Clear guidance on data collection methods, in particular sampling and
stratification, is a critical for any successful QI project.
Conclusions:Data Collection
23
45
“If I had to reduce my message for
management to just a few words, I’d say it all had to do with reducing variation.”
W. Edwards Deming
Question #4Do you understand variation conceptually?
46
The Problem!
Aggregated data presented in tabular formats or with summary statistics, will
not help you measure the impact of process improvement efforts.
Aggregated data and summary statistics can only lead to judgment, not
to improvement.
24
47
Average Percent of Patients who FallBefore and After the Implementation of a New Protocol
Pe
rce
nt
of
Pa
tie
nts
w
ho
Fa
ll
Time 1 Time 2
3.8
5.2
5.0%
4.0%
WOW!
A “significant drop”
from 5% to 4%
Conclusion -The protocol was a success! A 20% drop in the average mortality!
48
24 Months
1.0
9.0
Now what do you conclude about the impact of the protocol?
5.0
UCL= 6.0
LCL = 2.0
CL = 4.0
Protocol implemented here
Average Percent of Patients who FallBefore and After the Implementation of a New Protocol
Pe
rce
nt
of
Pa
tie
nts
w
ho
Fa
ll
25
49
The average of a set of numbers can be created by many different distributions
Average
Measu
re
Time
Patients do NOT care about the average!
50
If you don’t understand the variation that lives in your data, you will be tempted to ...
• Deny the data (It doesn’t fit my view of reality!)
• See trends where there are no trends
• Try to explain natural variation as special events
• Blame and give credit to people for things over which they have no control
• Distort the process that produced the data
• Kill the messenger!
26
“A phenomenon will be said to be
controlled when, through the use of
past experience, we can predict, at least
within limits, how the phenomenon may be expected to vary in
the future”W. Shewhart. Economic Control of
Quality of Manufactured Product, 1931
Dr. Walter A Shewhart
“What is the variation in one system over time?” Walter A. Shewhart - early 1920’s, Bell Laboratories
52
time
UCL
Every process displays variation:• Controlled variation
stable, consistent pattern of variation
“chance”, constant causes
• Special cause variation“assignable”
pattern changes over time
LCL
Static View
Sta
tic V
iew
Dynamic View
27
53
Common Cause Variation• Is inherent in the design of the
process
• Is due to regular, natural or ordinary causes
• Affects all the outcomes of a process
• Results in a “stable” process that is predictable
• Also known as random or unassignable variation
Special Cause Variation• Is due to irregular or unnatural
causes that are not inherent in the
design of the process
• Affect some, but not necessarily all aspects of the process
• Results in an “unstable” process
that is not predictable
• Also known as non-random or
assignable variation
Types of Variation
Point …Variation exists!
Common Cause does not mean “Good Variation.” It only
means that the process is stable and predictable. For
example, if a patient’s systolic blood pressure averaged around
165 and was usually between 160 and 170 mmHg, this might be
stable and predictable but completely unacceptable.
Similarly, Special Cause variation should not be viewed as “Bad
Variation.” You could have a special cause that represents a
very good result (e.g., a downward trend in the turnaround time
for a particular med), which you would want to sustain. Special
cause variation merely means that the process is unstable and
unpredictable.
28
55
2 Questions …
1. Is the process stable? If so, it is predictable.
2. Is the process capable?
• The chart will tell you if the process is stable and predictable.
• You have to decide if the output of the process is capable of meeting
the target or goal you have set!
• Finally, note that you should only try to make improvements to
processes that exhibit common cause variation. Why? Because
they are stable and predictable. You have no idea where processes
with special causes will go next!
Attributes of a Leader WhoUnderstands Variation
� Leaders understand the different ways that variation is viewed.
� They explain changes in terms of common causes and special causes.
� They use graphical methods to learn from data and expect others to consider variation in their decisions and actions.
� They understand the concept of stable and unstable processes and the potential losses due to tampering.
� Capability of a process or system is understood before changes are attempted.
29
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd
• Select several measures your organization tracks on a regular basis.
• Do you and the leaders of your organization evaluate these measures according the criteria for common and special causes of variation?
• If not, what criteria do you use to determine if data are improving or getting worse?
12/9
5
2/9
6
4/9
6
6/9
6
8/9
6
10/9
6
12/9
6
2/9
7
4/9
7
6/9
7
8/9
7
10/9
7
12/9
7
2/9
8
4/9
8
6/9
8
8/9
8
10/9
8
12/9
8
2/9
9
4/9
9
6/9
9
month
Perc
ent
C-s
ections
0.0
5.0
10.0
15.0
20.0
25.0
30.0
35.0
UCL=27.7018
CL=18.0246
LCL=8.3473
Percent of Cesarean Sections Performed Dec 95 - Jun 99
Week
Num
ber
of
Medic
ations E
rrors
per
1000 P
atient
Days
0.0
2.5
5.0
7.5
10.0
12.5
15.0
17.5
20.0
22.5
UCL=13.39461
CL=4.42048
LCL=0.00000
Medication Error Rate
DialogueCommon and Special Causes of Variation
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd
1. The same data can show different patterns of variation
dependent on how much of it you present and how you
statistically analyse and display the data.
2. Data presented over time (i.e., plotting the data by day,
week or month) is the only way you will ever be able to
improve any aspect of quality or safety!
3. Avoid using aggregated data and enumerative statistics if
you are serious about improving quality and safety!
4. A leaders job is to understand patterns of variation and ask
why!
ConclusionsUnderstanding Variation
30
Question #5Quality Measurement Journey: Understanding Variation Statistically
60
How do we analyze variation for quality improvement?
Run and Control Charts are the best
tools to determine:
1. The variation that lives in the process
2. if our improvement strategies have had
the desired effect.
31
Process Improvement: Isolated Femur Fractures
0
200
400
600
800
1000
1200
1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64Sequential Patients
Min
ute
s E
D t
o O
R p
er
Patient
Holding the Gain: Isolated Femur Fractures
0
200
400
600
800
1000
1200
1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64Sequential Patients
Min
ute
s E
D t
o O
R p
er
Patient
3. Determine if we are holding the gains
Current Process Performance: Isolated Femur Fractures
0
200
400
600
800
1000
1200
1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64Sequential Patients
Min
ute
s E
D t
o O
R p
er
Patient
Three Uses of SPC Charts
2. Determine if a change is an improvement
1. Make process performance visible
62
How many data points do I need?
Ideally you should have between
10 – 15 data points before constructing
a run chart
10 – 15 patients
10 – 15 days
10 – 15 weeks
10 – 15 months
10 – 15 quarters (not useful for QI projects)
• If you are just starting to
measure, plot the dots and
make a line graph.
• Once you have 8-10 data
points make a run chart.
32
Me
as
ure
Time
X (CL)~
The centerline (CL) on a
Run Chart is the Median
Elements of a Run Chart
Four run rules are used to determine if non-random variation is present
How do we analyze a Run Chart
“How will I know what the Run Chart is trying
to tell me?”
It is actually quite easy:
1. Determine the number of runs.
2. Then apply the 4 basic run chart
rules decide if your data reflect
random or non-random variation.
33
What is a Run?• One or more consecutive data points
on the same side of the Median
• Do not include data points that fall on
the Median
First, you need to determine the Number of Runs
Me
as
ure
Time
X (CL)~
How many Runs on this chart?
Points on the Median (don’t count these when counting the number of runs)
Run = a series of consecutive data points above or below the median.
Ignore points on the median.
34
Me
as
ure
Time
X (CL)~
Draw circles around the individual runs
Did you identify 7 runs?
Now apply the Run Chart Rules to Identify any non-random patterns in the data
• Rule #1: A shift in the process, or too many data points in a run (6 or more consecutive points above or below the median)
• Rule #2: A trend (5 or more consecutive points all increasing or decreasing)
• Rule #3: Too many or too few runs (use a table to determine this one)
• Rule #4: An “astronomical” data point
68
35
Rule 1 – A Shift
A shift in the process is six or more consecutive points either all above or all below the median. Values that fall on the median do not add to nor break a shift. Skip values that fall on the median and continue counting.
0
5
10
15
20
25
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25Me
as
ure
or
Ch
ara
cte
ris
tic
Rule 1
Rule 2 – A Trend
Five or more consecutive points all going up or all going down. If
the value of two or more successive points is the same, ignore one of the
points when counting. Like values do not make or break a trend.
Rule 2
0
5
10
15
20
25
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
Measure
or
Chara
cte
ristic
Median 11
36
Rule 3 – Runs Test(Too few or too many runs)
Rule 3
0
5
10
15
20
25
1 2 3 4 5 6 7 8 9 10
Me
as
ure
or
Ch
ara
ce
ris
tic
Median
11.4
Data line crosses only onceToo few runs: total of only 2 runs
Rule #3: Requires a reference table
72
Use this table by first calculating the number of "useful observations" in your data set. This is done by subtracting the number of data points on the median from the total number of data points. Then, find this number in the first column. The lower number of runs is found in the second column. The upper number of runs can be found in the third column. If the number of runs in your data falls below the lower limit or above the upper limit then this is a signal of a special cause.
# of Useful Lower Number Upper Number Observations of Runs of Runs
14 4 1215 5 1216 5 1317 5 1318 6 1419 6 1520 6 1621 7 1622 7 1723 7 1724 8 1825 8 1826 9 1927 10 1928 10 2029 10 2030 11 21
Total data points
So, for 25 useful
observations we should observe
between 8 and 18 runs
Two data points on the median = 27 useful
observations
Source: Swed, F. and Eisenhart, C.
(1943) “Tables for Testing
Randomness of Grouping in a
Sequence of Alternatives.” Annals of
Mathematical Statistics. Vol. XIV, pp.
66-87, Tables II and III.
37
Rule 3 – RunsToo many runs – what is this data telling you?
Rule 3 Too Many Runs
0
1
2
3
4
5
6
7
Jan-0
8
Feb-0
8
Mar-08
Apr-08
May-0
8
Jun-0
8
Jul-08
Aug-0
8
Sep-0
8
Oct-08
Nov-0
8
Dec-0
8
Jan-0
9
Feb-0
9
Mar-09
Apr-09
May-0
9
Jun-0
9
Jul-09
Aug-0
9
Sep-0
9
Measure
of C
hara
cte
ristic
Rule 4: An Astronomical ValueFor detecting unusually large or small numbers:
• Data that is Blatantly Obvious different value
• Everyone studying the chart agrees that it is unusual
• Remember: Every data set will have a high and a low - this does not mean the high or low are astronomical
Rule 4
0
5
10
15
20
25
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
Measure
ment or C
hara
cte
ristic
Remember, this rule is detecting only ONE astronomically high or low data point.
38
The Run Chart Rules for identifying non-random patterns
Source: The Data Guide by L. Provost and S. Murray, Jossey-Bass Publishers, 2011.
A Shift: 6 or more
An astronomical data point
Too many or too few runs
A Trend
5 or more
Me
as
ure
Time
X (CL)~
Did you find a shift, a trend, too many or too few runs or an astronomical data point?
Apply the Run Chart Rules to the Chart
39
So, let’s practice…
60
65
70
75
80
85
90
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27
Percent Compliance
ExercisePercent Compliance with Proper Hand Hygiene
Median = 81
How many runs on this chart
Week
Perc
ent C
om
plia
nce
40
60
65
70
75
80
85
90
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27
Percent Compliance
Week
Perc
ent C
om
plia
nce
15 runs
NOTE: 27 data points with 3 on the median gives you 24 “useful observations.” For 24 useful observations you expect between 8 and 18 runs.
Percent Compliance with Proper Hand HygieneNow apply the Run Chart Rules
Week
Median = 81
Did you find a shift, a trend, too many or too few runs or an astronomical data point?
• What can you explain about Shewhart (control) charts?
• What do you need to think more about to be effective with applying Shewhart charts?
SO,
Let’s continue to understand
variation statistically
41
What are 4 reasons why Shewhart Charts are preferred over Run Charts?
Because Control Charts…
1. Are more sensitive than run charts:� A run chart cannot detect special causes that are due to point-to-point
variation (median versus the mean)
� Tests for detecting special causes can be used with control charts
2. Have the added feature of control limits, which allow us to
determine if the process is stable (common cause variation)
or not stable (special cause variation).
3. Can be used to define process capability.
4. Allow us to more accurately predict process behavior and
future performance.
Jan01 Mar01 May01 July01 Sept01 Nov01 Jan02 Mar02 May02 July02 Sept02 Nov02
Month
Nu
mb
er
of
Co
mp
lain
ts
5.0
10.0
15.0
20.0
25.0
30.0
35.0
40.0
45.0
50.0
A
B
C
C
B
A
UCL=44.855
CL=29.250
LCL=13.645
Elements of a Shewhart Control Chart
X (Mean)
Measure
Time
An indication of a
special cause
(Upper Control Limit)
(Lower Control Limit)
42
There Are 5 Basic Control Charts
Variables Charts Attributes Charts
• p-chart(the proportion or percent of defectives)
• c-chart(the number of defects)
• u-chart(defect rate, e.g., falls per 1000 patient days)
• X & S chart (Average & SD chart)
• XmR chart (Individual & moving range chart)
Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett, 2004, Chap.6
Deciding which chart is most appropriate starts with knowing what type of data you have collected
Variables Data
Attributes DataDefects
(occurrences only)Defectives
(occurrences plus non-occurrences)
Nonconforming UnitsNonconformities
43
The Control Chart Decision Tree
Variables Data Attributes Data
More than one
observation per
subgroup?
X bar & SXmR
Is there an equal area of opportunity?
Occurrences & Non-
occurrences?
p-chartu-chartc-chart
Decide on the type of data
Yes
Yes
Yes
No
No
The percent of
Defective Units
The number
of DefectsThe Defect
Rate
Individual
Measurement
Average and
Standard
Deviation
No
Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett, 2004.
86
Type of Chart Medication Production Analysis
X bar & S Chart TAT for a daily sample of 25 medication orders
Individuals Chart(XmR)
The number of medication orders processed each week
C-ChartUsing a sample of 100 medication orders each week, we count the errors (defects) on each order
U-ChartOut of all medication orders each week, we calculate the number of errors (defects) per 10k orders
P-ChartFor all medication orders each week, we calculate the percentage that have 1 or more errors (i.e., are defective)
The choice of a control chart depends on the measure you have defined!
44
©Copyright 2013 IHI/R. Lloyd
Your next move…
…to gain more knowledge about Shewhart Charts (a.k.a. control charts)
45
89
AIM* (How good? By when?)
Concept
Measure
Operational Definitions
Data Collection Plan
Data Collection
Analysis ACTION
Milestones in theQuality Measurement Journey
Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett Publishers, 2004.
90
But the Charts Don’t Tell You…
• The reasons(s) for a Special Cause.
• Whether or not a Common Cause process
should be improved (is the performance of
the process acceptable?)
• How the process should actually be
improved or redesigned.
46
You need a Framework forPerformance Improvement
• Establish appropriate measures.
• Set an aim and goal for each measure.
• Develop theories and predictions on how you plan on achieving the aim and an appropriate time frame for testing.
• Test your theory, implement the change concepts, follow the measures over time and analyze the results.
• Revise the strategy as needed.
311
Finally, remember that data is a necessary part of the Sequence of Improvement
Sustaining improvements and Spreading changes to other locations
Developing a change
Implementing a change
Testing a change
Theory and Prediction
Test under a variety of conditions
Make part of routine operations
47
and the
93
1. What is your current level of knowledge about quality measurement?
2. What is your motivation for measuring?
3. Do you know the milestones in the Quality Measurement Journey (QMJ)?
4. Do you understand variation conceptually?
5. Do you understand variation statistically?
6. How well do you link measurement to improvement
Our Objectives for today…To answer Six Key Questions on Measurement
94
Appendices
•Appendix A: General References on Quality
• Appendix BC: References on Measurement
• Appendix C: Operational Definition Worksheet
• Appendix D: Faculty Bios
48
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd
It must be remembered that there is nothing more difficult to plan, more doubtful of success, nor more dangerous to manage than the creation of a new system.
For the initiator has the enmity of all who would profit by the preservation of the old institution and merely lukewarm defenders in those who would gain by the new one.
A closing thought…
Machiavelli, The Prince, 1513Machiavelli, The Prince, 1513Machiavelli, The Prince, 1513Machiavelli, The Prince, 1513
96
Appendices
•Appendix A: General References on Quality
• Appendix BC: References on Measurement
• Appendix C: Operational Definition Worksheet
• Appendix D: Faculty Bios
49
97
Appendix AGeneral References on Quality
• The Improvement Guide: A Practical Approach to Enhancing Organizational Performance. G. Langley, K. Nolan, T. Nolan, C. Norman, L. Provost. Jossey-Bass Publishers., San Francisco, 1996.
• Quality Improvement Through Planned Experimentation. 2nd edition. R. Moen, T. Nolan, L. Provost, McGraw-Hill, NY, 1998.
• The Improvement Handbook. Associates in Process Improvement. Austin, TX, January, 2005.
• A Primer on Leading the Improvement of Systems,” Don M. Berwick, BMJ, 312: pp 619-622, 1996.
• “Accelerating the Pace of Improvement - An Interview with Thomas Nolan,”Journal of Quality Improvement, Volume 23, No. 4, The Joint Commission, April, 1997.
98
Appendix BReferences on Measurement
• Carey, R. and Lloyd, R. Measuring Quality Improvement in healthcare: A Guide to Statistical Process Control Applications. ASQ Press, Milwaukee, WI, 2001.
• Lloyd, R. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett Publishers, Sudbury, MA, 2004.
• Nelson, E. et al, “Report Cards or Instrument Panels: Who Needs What?Journal of Quality Improvement, Volume 21, Number 4, April, 1995.
• Provost, L. and Murray, S. The Health Care Data Guide. Jossey-Bass Publishers, 2011.
• Solberg. L. et. al. “The Three Faces of Performance Improvement: Improvement, Accountability and Research.” Journal of Quality Improvement23, no.3 (1997): 135-147.
50
Team name: _____________________________________________________________________________
Date: __________________ Contact person: ____________________________________
WHAT PROCESS DID YOU SELECT?
WHAT SPECIFIC MEASURE DID YOU SELECT FOR THIS PROCESS?
OPERATIONAL DEFINITIONDefine the specific components of this measure. Specify the numerator and denominator if it is a percent or a rate. If it is an average, identify the calculation for deriving the average. Include any special equipment needed to capture the data. If it is a score (such as a patient satisfaction score) describe how the score is derived. When a measure reflects concepts such as accuracy, complete, timely, or an error, describe the criteria to be used to determine “accuracy.”
Appendix COperational Definition Worksheet
Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett, 2004.
DATA COLLECTION PLANWho is responsible for actually collecting the data?How often will the data be collected? (e.g., hourly, daily, weekly or monthly?)What are the data sources (be specific)?What is to be included or excluded (e.g., only inpatients are to be included in this measure or only stat lab requests should be tracked).How will these data be collected?Manually ______ From a log ______ From an automated system
BASELINE MEASUREMENTWhat is the actual baseline number? ______________________________________________What time period was used to collect the baseline? ___________________________________
TARGET(S) OR GOAL(S) FOR THIS MEASUREDo you have target(s) or goal(s) for this measure?Yes ___ No ___
Specify the External target(s) or Goal(s) (specify the number, rate or volume, etc., as well as the source of the target/goal.)
Specify the Internal target(s) or Goal(s) (specify the number, rate or volume, etc., as well as the source of the target/goal.)
Appendix C: Operational Definition Worksheet (cont’d)
Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett, 2004.
51
Robert Lloyd, PhD is Vice President at the Institute for Healthcare Improvement (IHI).
Dr. Lloyd provides leadership in the areas of performance improvement strategies,
statistical process control methods, development of strategic dashboards and building
capacity and capability for quality improvement. He also serves as lead faculty for
various IHI initiatives and demonstration projects in the US, the UK, Sweden, Denmark,
Norway, New Zealand, Australia and Africa.
Before joining the IHI, Dr. Lloyd served as the Corporate Director of Quality Resource
Services for Advocate Health Care (Oak Brook, IL). He also served as Senior Director of
Quality Measurement for Lutheran General Health System (Park Ridge, IL), directed the
American Hospital Association's Quality Measurement and Management Project
(QMMP) and served in various leadership roles at the Hospital Association of
Pennsylvania. The Pennsylvania State University awarded all three of Dr. Lloyd’s
degrees. His doctorate is in agricultural economics and rural sociology.
Appendix D: Robert Lloyd, PhD
Dr. Lloyd has written many articles and chapters in books. He is also the co-author of the internationally
acclaimed book, Measuring Quality Improvement in Healthcare: A Guide to Statistical Process Control
Applications (American Society for Quality Press, 2001, 5th printing) and the author of Quality Health Care: A
Guide to Developing and Using Indicators, 2004 by Jones and Bartlett (Sudbury, MA).
Dr. Lloyd lives in Chicago with his wife Gwenn and amusing dog Cricket. The Lloyds have a 22 year old
daughter Devon who is in her final year of university majoring in performance dance and choreography.
@rlloyd66
Appendix D: Gary Sutton, BSc (Honors) P102
Gary Sutton, is a Statistician and Improvement Advisor for
the Scottish Government. He holds a BSc (Honors) in Applied
Statistics. He has been employed as a statistician in the UK
and Scottish Government for over 20 years where he has
been involved in the management of a number of government
statistical surveys. Between July 2012 and December 2015,
Gary has specialized in applying Quality Improvement (QI) to
the Early Years Collaborative (EYC). He is a graduate of the
IHI Improvement Advisor Development Programme (2013)
and has also been an IA Grad for the IHI IA Programme in
Scotland (2015). Gary has presented QI topics to both large
and audiences and QI teams. He has also provided support
to IHI Faculty in the training and development of Scottish
faculty.