Measuring & Understanding Quality Improvement in Healthcare

Steve Meurer, MBA / MHS, PhDVice President - Operations

St Mary Medical CenterLanghorne, PA

Continuing the Journey1) Research Questions

• Experience in Healthcare Operations

2) Theory / Models•Search and Study

•Develop Questions and Hypotheses

3) Develop / Test•Develop 2 into something that can

help answer 1

4) Examine Results• How does 3 answer 1

Oral Exam

Defense

Focus

Initial Research QuestionsDeveloped from 8 years of frustration in healthcare

management knowing that I wasn’t equipped to provide appropriate support to clinicians

What I Could Provide What I Needed to ProvideLeadershipUnderstanding of HealthcareFinancial DirectionStrategic DirectionManagement Capabilities

A More Balanced Approach to Managing AssessmentData Management Study Design

Initial Research Questions How do I know something works?

Continued requests for equipment, supplies and instruments

Do patients get better? The End of Medicine

Is there one way to do a procedure that is better than another? Significant variation in preference cards

How do I measure quality?

Defining Quality

IOM – The degree to which health services for individuals and populations increase the likelihood of desired health outcomes and are consistent with current professional knowledge.

Donabedian - The systematic measurement and evaluation of the predetermined outcomes of a process, and the subsequent use of information to improve the process based on expectations of the customer.

Theories / Models – Oral Exam Ernest Amory Codman

end results idea – 1920s Florence Nightingale (late 1800s) and Walter Shewhart (1920s)

Statistical Process Control Edwards Deming, Joseph Juran and Kauru Ishikawa

Continuous Quality Improvement (CQI) Avedis Donabedian

structure – process – outcome Implicit vs. explicit criteria

Everett Rogers Dissemination of Innovation

Don Berwick, Paul Batalden, Brent James and Steve Shortell Recent literature

Structure – Process - Outcome

Structure Definition Something arranged in a definite pattern of organization Organization of parts as dominated by the general

character of the whole

Quality of healthcare can be assessed on the basis of structure, process (how care is delivered), and outcome (mortality, functional status, quality of life, and patient satisfaction) good measures of the first two are those that have a clear relationship to the thirdstructure must proceed process which must proceed outcome

Implementing CQI is Largely Structure

Organization Division of labor HR / Training Specialty Mix Coordination Customer Focus Resources Training/Experience Planning Work Load Power Process Access Resources Innovation Buildings Supplier Partner Information Information Leadership Leadership Policies/Procedures Tasks

Medical Scott CQI Outcomes

Structural Dimensions of CQI Implementation

Strategic Cultural Technical Organiz. Result

No Yes Yes Yes No significant results on anything important

Yes No Yes Yes Small, temporary effects

Yes Yes No Yes Frustration & False starts

Yes Yes Yes No Inability to capture the learning & diffusion

Yes Yes Yes Yes Lasting process change

Adapted from Shortell et al. 1996

Translating Theory To Research – Putting Shape to my Frustrations

1. The healthcare system is broken The IOM reports

3. CQI, if implemented properly, can be the appropriate structure Managerial Philosophy

4. Healthcare providers are are finding it difficult to implement CQI

Problem Solving Methodology

2. Appropriate structure – “the forgotten, but important component of the quality triad” – is wrongly assumed as present

JCAHO

Research Question Why haven’t healthcare organizations

been able to use CQI to differentiate themselves in terms of quality?

Answer: Accountability & Assessment

Accountability

Current Motivators that may force change Patient Safety – Medical Errors

“report cards” - PA, NY and CA Increase in Costs and Premiums Increase in the use of Alternative

Medicine Variation in processes Increase in litigation

Individual motivation has not been successful Midnight at the Waldorf-Astoria Rhetoric, not Reality (The Halothane Study)

Large scale environmental change is needed Environment, Organization, Micro-System, Pt

Assessment Issues & Research Questions1. Low CQI knowledge level of senior leadership

Do step by step instructions exist that assists researchers in studying CQI, and healthcare leaders in implementing CQI?

2. Implementing only a subset of the CQI domainsDoes a comprehensive survey and scale of CQI implementation exist?

3. Poor measurement strategiesAre psychometrics examined appropriately in measuring CQI implementation?

4. Rhetoric does not equal the reality

Does a measure exist that can be used to develop a quick snapshot of CQI implementation efforts in a hospital and is there a corresponding descriptive scale?

Step 1: Search the Literature

Methods - Used Ovid databases with a focus on business, psychology,

sociology and healthcare Keywords: measurement quality, continuous quality

improvement, total quality management, implementation Scrolled through quickly at least 10,000 references

most focused on implementing only a few domains of CQI Articles not deleted included: Measuring Quality; Domains

and Implementation of Quality

Does a comprehensive, valid and easy to administer measurement tool exist that provides organization leaders with a descriptive scale and instructions for next steps?

Quality OverloadUsing Medline through PubMed

from 1995 to the present / English only ‘quality improvement’ = 8,848 ‘continuous quality improvement’ = 1,100 ‘quality’ in title = 17,466 ‘quality improvement measure’ in title = 3

from 2000 to present / English only ‘quality improvement’ in title = 350

50 usable, 30 ‘easily findable’, 15 good, 1 measurement

Worldwide Measurement for QA/QI StructureAWARDS - too time intensive, gold standard domains

The Malcolm Baldrige AwardLeadership, HR/Training, Process, Business Results, Customer Focus, Information Systems, Planning, Partnership

EFQM US State-Based Awards

ACCREDITATION & CERTIFICATION - questions on validity and too time intensive

JCAHOIS0 9002 - 2000

Inspection, Contract, Public Responsibility, Innovation, Product Control, Servicing

SURVEY INSTRUMENTS

Survey InstrumentsEleven worldwide surveys examining CQI as a managerial philosophy were analyzed (Tables - pges 1 & 2) 8 from the US, 1 from Canada, the Netherlands, and Australia5 were specific to healthcare, including the ‘gold standard’ from Shortell et al.Most examined psychometrics while very few provided a scaleThe shortest (22 questions) was also the least comprehensiveThe Baldrige Domains dominated

Survey Instruments

Weaknesses of the current surveys included:Relatively few domains other than the Baldrige were even mentioned The most comprehensive surveys are much too long (depth vs. breadth)Strong potential for respondent bias if survey is only given to one level employee in an organizationVery few surveys provided a scale

Published in the International Journal for Quality in Health Care 2001: Volume 13, Number 3: pp 197-207

Step 2: Develop and test a comprehensive and concise measure of CQI implementation

Initial Survey 14 domains (Baldrige, EFQM, ISO) and 70 items, with

each domain containing at least 4 items All but a very few items were from the 11 surveys

analyzed in Step 1 Items were chosen by the researchers using a

subjective analysis and whether or not the question could be answered using a 5 point Likert scale

Content Validity The benefits of a content validity study for this study

True experts in the field of CQIPast measures have gone through psychometric testingExcellent method of data reduction

MethodsStatistical method described in Grant & Davis (1997) and Lynn (1986)1. Panel of ExpertsAll either attend an invite only CQI symposium sponsored by Dartmouth, have recently taught CQI at a Masters level or are positional leaders of QI efforts in a healthcare organization Best to have between 7 and 10 - this study had 7 from the US and 1 from England

Content Validity Methods (cont.)

2. Scoring Grid (See Sample Grid - pge 3) Each expert was emailed the scoring grid with

definitions and instructions. Is the item clear and understandable?

4 point scale

Does the item represent CQI? 4 point scale

Match the item with a domain. 1 through 14 representing each domain & 15 representing unable to

classify

Content Validity Methods (cont.)

3. IndicesInter-rater agreement (IR) = # of raters who scored an

item as high / total # of raters high defined as a 1 or 2 on both 4 point scales acceptable IR > / = .70

Content Validity Index (CVI) = # of items where all experts rated high / # of items

acceptable CVI > / = .80Domain congruence = % of time where experts chose

the same domain as the investigators

Content Validity Results After 4 analysis iterations where poorly rated items

were deleted, the questionnaire included: 22 items 8 domains Clarity IR of .91 (range of .85 - 1) Representativeness IR of .93 (range of .87 - 1) Clarity CVI of .73 using Lynn’s (1986) method Representativeness CVI of .91 using Lynn’s (1986)

method Overall, the experts chose the same domain as the

investigators in the original measure 76% of the time

Content Validity Results Investigators added 6 items to ensure that every domain except for Supplier Partnership contained 3 items Investigators changed some wording to increase clarity Final survey was sent back to experts for comments

Other ResultsThe scoring grid took a lot longer to complete than originally thoughtThe leadership domain had the highest rate of agreementQI must be differentiated from QABaldrige criteria dominate

Step 3: Develop a corresponding scale of CQI implementation

A 5 level corresponding scale (pge 4) was developed by the investigators from: Roger’s Diffusion of Innovations

agenda setting, matching, redefining / restructuring, confirmation, clarifying, routinizing

Deming’s Continuous Quality Improvement match domains with appropriate level

Samsa & Matchar CQI as a problem solving methodology vs. a managerial philosophy

Characteristics: CQI is a developmental process Time is important Scale provides focus for future quality initiatives

Published and Voted Best Student-Led Paper in the 2002 Business and Health Administration

Proceedings,pges 198-204

Will be further published in Either Quality in Health Care or Hospital Topics

Steps 2 & 3 - Content Validity and Scale

Step 4: Is the survey and scale easy to use and are the results practical?

Pilot Study Worked with the Missouri Hospital Association for contacts.

83 Missouri hospitals eligible (above 40 beds), 40 participated 5 responses from each hospital: CEO/COO, Director of

Quality, a non-salaried MD, and 2 managers Survey and results disseminated via email

1. There will be measurable differences between and within hospitals.2. The survey will have high known-groups validity.3. The items and domains will differentiate between levels as hypothesized by the conceptual scale.

Hypotheses based on Paper 1 Weaknesses

Pilot Study Methods Known Groups Validity

Examined the relationship between the survey and: state quality team winners >/= Level 2 state quality organization winners >/= Level 3 national quality award finalists >/= Level 3 subjective quality assessment at 10 of the 40 hospitals question 1 (pge 5) asking the participants to categorize their

quality structure

Reliability Cronbach’s Alpha for each domain, each title, and the overall

measure

Pilot Study Methods Between Hospital Variation

One-way ANOVA & Bonferroni by size, region and ownership model

Within Hospital Variation Repeated Measures ANOVA & Bonferroni

by title

Item and Domain Analysis ANOVA & Bonferroni

determine which items and which domains discriminate well between different levels of the scale

Pilot Study Hospital Total LevelN = 40 (min of 90, max of 130)

0

2

4

6

8

10

12

14

16

Level 1(</= 100)

Level 2(101-110)

Level 3(111-120)

Level 4(121-130)

Level 5(131-140)

# of hospitals

Pilot Study Results Sample Characteristics (pge 6)

40 hospitals are significantly larger and more likely to be for profit and part of a system

Of the 200 returned surveys, there was less than 5% missing values and ‘I don’t know / NA’

Known Groups Validity 2 of the 12 did not score as hypothesized

hospitals were not significantly different than others 4 of 9 (44%) similar for the subjective assessment 40% agreement for question 1 assessment

Pilot Study Results Reliability

Cronbach’s Alpha ranged from .54 (HR/Training) to .84 (Innovation) for the domains

Information .69; Process, Planning .77; Customer .78; Leadership .83

Cronbach’s Alpha ranged from .88 (Director /Manager and QI Director) to .92 (MDs) for different groups

Cronbach’s Alpha was .94 for the overall measure

Pilot Study Results

Between Hospital Variation Region was only attribute that was significant

Within Hospital Variation Senior Executives significantly lower than QI Directors QI Directors significantly higher than Managers /

Directors MDs significantly higher than Managers/ Directors

supports surveying more than one level employee

Pilot Study Results Item and Domain Analysis (pge 7)

Of the 28 items, 4 did not show good differentiation between any of the levels

these should be either reworded or changed Leadership showed significant differentiation between

all levels studied Planning showed significant differentiation between 2 of

the 5 levels

Pilot Study ResultsQuality Improvement Scale

Level Components after Domain Analysis1. Quality Assurance

2. QI Low High Focus – Leadership (visibility); Customer FocusMedium Focus – Innovation

3. QI Medium High Focus – Process; HR / Training; PlanningMedium Focus – Leadership (support)

4. QI High Medium Focus – Information; Supplier PartnershipLow Focus – Leadership (consistency); Planning

5.QI - Absorbed

Findings The survey is easy to administer The survey provides a reliable and valid snapshot of

CQI implementation in a healthcare organization No known group exists The scale is a practical method of providing hospital

leaders with a roadmap for CQI implementation Leadership is the most important component of

implementing CQI

Submitted to Health Services Research

A Likely Future Scenario1. Patient Safety provides accountability to analyze

quality and outcomes 2. Healthcare leaders see CQI as a methodology to

improve patient outcomes3. Hospitals use the survey and scale to help

assess & implement CQI appropriately, which in turn eliminates structure issues discussed

4. Because of this, hospitals can effectively assess their processes and improve their outcomes

Future Research Questions Can a clearer snapshot of CQI implementation emerge using

line worker responses, and senior leadership interviews? Administered the survey to a 40 random line workers at 10

hospitals Initial results include:

50% return rate high number of ‘I don’t know / NA’ responses all hospitals overall employee score < 100 (Quality Assurance)

Senior hospital leadership meetings to discuss quality structure Initial results include:

low knowledge of ‘quality’ among the senior leaders structures developed with little statistical or facilitation resources

Future Questions Does a higher level of CQI implementation lead to better

financial, quality and safety outcomes? If not, why?

Develop and Find Financial, Operational and HR Effectiveness and Efficiency Measures Counte & Glandon, 1995

Build one, clean database with CQI implementation scores and measures

Analyze to assess relationships

Future QuestionsIs CQI, as its described in the literature, an effective

method for improving quality outcomes? Change the four items that did not differentiate well

1. How many multi-disciplinary teams currently work to improve the processes of care in your organization? (Process)

more statistics, less teams• Human Factors Research• Toyota• Six Sigma

2. Do people in your organization know who their customers are? (Customer Focus) not clear

3. Are employees in this organization encouraged to try new and better ways of doing things? (Innovation)

4. Is creativity actively encouraged in this organization? (Innovation) healthcare has typically not been innovative and so these questions may need to be

more specific

Acknowledgements and Questions

Mentor – Dr. Counte

Committee – Drs. Arrington, Rubio & Burroughs

Dr. Dunagan

Gretchen