Date post: | 14-Jun-2015 |
Category: |
Health & Medicine |
Upload: | mahboob-khan-mhacphq-hqm-harvard-university |
View: | 376 times |
Download: | 0 times |
Measuring & Understanding Quality Improvement in Healthcare
Mahboob ali khan, MHA / CPHQ,
Consultant Quality - Operations
Continental Hospitals
Hyderabad, India
Continuing the Journey1) Research Questions
• Experience in Healthcare Operations
2) Theory / Models• Search and Study
• Develop Questions and Hypotheses
3) Develop / Test• Develop 2 into something that can
help answer 1
4) Examine Results• How does 3 answer 1
Oral Exam
Defense
Focus
Initial Research QuestionsDeveloped from 8 years of frustration in healthcare
management knowing that I wasn’t equipped to provide appropriate support to clinicians
What I Could Provide What I Needed to Provide
LeadershipUnderstanding of HealthcareFinancial DirectionStrategic DirectionManagement Capabilities
A More Balanced Approach to Managing AssessmentData Management Study Design
Initial Research Questions How do I know something works?
Continued requests for equipment, supplies and instruments
Do patients get better? The End of Medicine
Is there one way to do a procedure that is better than another? Significant variation in preference cards
How do I measure quality?
Defining Quality
IOM – The degree to which health services for individuals and populations increase the likelihood of desired health outcomes and are consistent with current professional knowledge.
Donabedian - The systematic measurement and evaluation of the predetermined outcomes of a process, and the subsequent use of information to improve the process based on expectations of the customer.
Theories / Models – Oral Exam Ernest Amory Codman
end results idea – 1920s Florence Nightingale (late 1800s) and Walter Shewhart
(1920s) Statistical Process Control
Edwards Deming, Joseph Juran and Kauru Ishikawa Continuous Quality Improvement (CQI)
Avedis Donabedian structure – process – outcome Implicit vs. explicit criteria
Everett Rogers Dissemination of Innovation
Don Berwick, Paul Batalden, Brent James and Steve Shortell Recent literature
Structure – Process - Outcome
Structure Definition Something arranged in a definite pattern of organization Organization of parts as dominated by the general
character of the whole
Quality of healthcare can be assessed on the basis of structure, process (how care is delivered), and outcome (mortality, functional status, quality of life, and patient satisfaction) good measures of the first two are those that have a clear relationship to the thirdstructure must proceed process which must proceed outcome
Implementing CQI is Largely Structure
Organization Division of labor HR / Training Specialty Mix Coordination Customer Focus Resources Training/Experience Planning Work Load Power Process Access Resources Innovation Buildings Supplier Partner Information Information Leadership Leadership Policies/Procedures Tasks
Medical Scott CQI Outcomes
Structural Dimensions of CQI Implementation
Strategic Cultural Technical Organiz. Result
No Yes Yes Yes No significant results on anything important
Yes No Yes Yes Small, temporary effects
Yes Yes No Yes Frustration & False starts
Yes Yes Yes No Inability to capture the learning & diffusion
Yes Yes Yes Yes Lasting process change
Adapted from Shortell et al. 1996
Translating Theory To Research – Putting Shape to my Frustrations
1. The healthcare system is broken The IOM reports
3. CQI, if implemented properly, can be the appropriate structure Managerial Philosophy
4. Healthcare providers are are finding it difficult to implement CQI
Problem Solving Methodology
2. Appropriate structure – “the forgotten, but important component of the quality triad” – is wrongly assumed as present
JCAHO
Research Question
Why haven’t healthcare organizations been able to use CQI to differentiate themselves in terms of quality?
Answer: Accountability & Assessment
Accountability
Current Motivators that may force change Patient Safety – Medical Errors “report cards” - PA, NY and CA Increase in Costs and Premiums Increase in the use of Alternative Medicine Variation in processes Increase in litigation
Individual motivation has not been successful Midnight at the Waldorf-Astoria Rhetoric, not Reality (The Halothane Study)
Large scale environmental change is needed Environment, Organization, Micro-System, Pt
Assessment Issues & Research Questions1. Low CQI knowledge level of senior leadership
Do step by step instructions exist that assists researchers in studying CQI, and healthcare leaders in implementing CQI?
2. Implementing only a subset of the CQI domainsDoes a comprehensive survey and scale of CQI implementation exist?
3. Poor measurement strategiesAre psychometrics examined appropriately in measuring CQI implementation?
4. Rhetoric does not equal the realityDoes a measure exist that can be used to develop a quick snapshot of CQI implementation efforts in a hospital and is there a corresponding descriptive scale?
Step 1: Search the Literature
Methods - Used Ovid databases with a focus on business,
psychology, sociology and healthcare Keywords: measurement quality, continuous quality
improvement, total quality management, implementation
Scrolled through quickly at least 10,000 references most focused on implementing only a few domains of CQI
Articles not deleted included: Measuring Quality; Domains and Implementation of Quality
Does a comprehensive, valid and easy to administer measurement tool exist that provides organization leaders with a descriptive scale and instructions for next steps?
Quality OverloadUsing Medline through PubMed
from 1995 to the present / English only ‘quality improvement’ = 8,848 ‘continuous quality improvement’ = 1,100 ‘quality’ in title = 17,466 ‘quality improvement measure’ in title = 3
from 2000 to present / English only ‘quality improvement’ in title = 350
50 usable, 30 ‘easily findable’, 15 good, 1 measurement
Worldwide Measurement for QA/QI StructureAWARDS - too time intensive, gold standard domains
The Malcolm Baldrige AwardLeadership, HR/Training, Process, Business Results, Customer Focus, Information Systems, Planning, Partnership
EFQM US State-Based Awards
ACCREDITATION & CERTIFICATION - questions on validity and too time intensive
JCAHOIS0 9002 - 2000
Inspection, Contract, Public Responsibility, Innovation, Product Control, Servicing
SURVEY INSTRUMENTS
Survey InstrumentsEleven worldwide surveys examining CQI as a managerial philosophy were analyzed (Tables - pges 1 & 2) 8 from the US, 1 from Canada, the Netherlands, and Australia5 were specific to healthcare, including the ‘gold standard’ from Shortell et al.Most examined psychometrics while very few provided a scaleThe shortest (22 questions) was also the least comprehensiveThe Baldrige Domains dominated
Survey Instruments
Weaknesses of the current surveys included:Relatively few domains other than the Baldrige were even mentioned The most comprehensive surveys are much too long (depth vs. breadth)Strong potential for respondent bias if survey is only given to one level employee in an organizationVery few surveys provided a scale
Published in the International Journal for Quality in Health Care 2001: Volume 13, Number 3: pp 197-207
Step 2: Develop and test a comprehensive and concise measure of CQI implementation
Initial Survey 14 domains (Baldrige, EFQM, ISO) and 70 items, with
each domain containing at least 4 items All but a very few items were from the 11 surveys
analyzed in Step 1 Items were chosen by the researchers using a
subjective analysis and whether or not the question could be answered using a 5 point Likert scale
Content Validity The benefits of a content validity study for this study
True experts in the field of CQI
Past measures have gone through psychometric testing
Excellent method of data reduction
MethodsStatistical method described in Grant & Davis (1997) and Lynn
(1986)1. Panel of ExpertsAll either attend an invite only CQI symposium sponsored by Dartmouth, have recently taught CQI at a Masters level or are positional leaders of QI efforts in a healthcare organization Best to have between 7 and 10 - this study had 7 from the US and 1 from England
Content Validity Methods (cont.)
2. Scoring Grid (See Sample Grid - pge 3) Each expert was emailed the scoring grid with
definitions and instructions. Is the item clear and understandable?
4 point scale
Does the item represent CQI? 4 point scale
Match the item with a domain. 1 through 14 representing each domain & 15 representing
unable to classify
Content Validity Methods (cont.)
3. IndicesInter-rater agreement (IR) = # of raters who scored an
item as high / total # of raters high defined as a 1 or 2 on both 4 point scales acceptable IR > / = .70
Content Validity Index (CVI) = # of items where all experts rated high / # of items
acceptable CVI > / = .80
Domain congruence = % of time where experts chose the same domain as the investigators
Content Validity Results After 4 analysis iterations where poorly rated items
were deleted, the questionnaire included: 22 items 8 domains Clarity IR of .91 (range of .85 - 1) Representativeness IR of .93 (range of .87 - 1) Clarity CVI of .73 using Lynn’s (1986) method Representativeness CVI of .91 using Lynn’s (1986)
method Overall, the experts chose the same domain as the
investigators in the original measure 76% of the time
Content Validity Results
Investigators added 6 items to ensure that every domain except for Supplier Partnership contained 3 items
Investigators changed some wording to increase clarity
Final survey was sent back to experts for comments
Other ResultsThe scoring grid took a lot longer to complete than originally thoughtThe leadership domain had the highest rate of agreementQI must be differentiated from QABaldrige criteria dominate
Step 3: Develop a corresponding scale of CQI implementation
A 5 level corresponding scale (pge 4) was developed by the investigators from: Roger’s Diffusion of Innovations
agenda setting, matching, redefining / restructuring, confirmation, clarifying, routinizing
Deming’s Continuous Quality Improvement match domains with appropriate level
Samsa & Matchar CQI as a problem solving methodology vs. a managerial
philosophy Characteristics:
CQI is a developmental process Time is important Scale provides focus for future quality initiatives
Published and Voted Best Student-Led Paper in the 2002 Business and Health Administration
Proceedings,pges 198-204
Will be further published in Either Quality in Health Care or Hospital Topics
Steps 2 & 3 - Content Validity and Scale
Step 4: Is the survey and scale easy to use and are the results practical?
Pilot Study Worked with the Missouri Hospital Association for contacts.
83 Missouri hospitals eligible (above 40 beds), 40 participated 5 responses from each hospital: CEO/COO, Director of
Quality, a non-salaried MD, and 2 managers Survey and results disseminated via email
1. There will be measurable differences between and within hospitals.2. The survey will have high known-groups validity.3. The items and domains will differentiate between levels as hypothesized by the conceptual scale.
Hypotheses based on Paper 1 Weaknesses
Pilot Study Methods Known Groups Validity
Examined the relationship between the survey and: state quality team winners >/= Level 2 state quality organization winners >/= Level 3 national quality award finalists >/= Level 3 subjective quality assessment at 10 of the 40 hospitals question 1 (pge 5) asking the participants to categorize
their quality structure
Reliability Cronbach’s Alpha for each domain, each title, and the
overall measure
Pilot Study Methods Between Hospital Variation
One-way ANOVA & Bonferroni by size, region and ownership model
Within Hospital Variation Repeated Measures ANOVA & Bonferroni
by title
Item and Domain Analysis ANOVA & Bonferroni
determine which items and which domains discriminate well between different levels of the scale
Pilot Study Hospital Total LevelN = 40 (min of 90, max of 130)
0
2
4
6
8
10
12
14
16
Level 1(</= 100)
Level 2(101-110)
Level 3(111-120)
Level 4(121-130)
Level 5(131-140)
# of hospitals
Pilot Study Results Sample Characteristics (pge 6)
40 hospitals are significantly larger and more likely to be for profit and part of a system
Of the 200 returned surveys, there was less than 5% missing values and ‘I don’t know / NA’
Known Groups Validity 2 of the 12 did not score as hypothesized
hospitals were not significantly different than others 4 of 9 (44%) similar for the subjective assessment 40% agreement for question 1 assessment
Pilot Study Results Reliability
Cronbach’s Alpha ranged from .54 (HR/Training) to .84 (Innovation) for the domains
Information .69; Process, Planning .77; Customer .78; Leadership .83
Cronbach’s Alpha ranged from .88 (Director /Manager and QI Director) to .92 (MDs) for different groups
Cronbach’s Alpha was .94 for the overall measure
Pilot Study Results
Between Hospital Variation Region was only attribute that was significant
Within Hospital Variation Senior Executives significantly lower than QI Directors QI Directors significantly higher than Managers /
Directors MDs significantly higher than Managers/ Directors
supports surveying more than one level employee
Pilot Study Results
Item and Domain Analysis (pge 7) Of the 28 items, 4 did not show good differentiation
between any of the levels these should be either reworded or changed
Leadership showed significant differentiation between all levels studied
Planning showed significant differentiation between 2 of the 5 levels
Pilot Study ResultsQuality Improvement Scale
Level Components after Domain Analysis1. Quality Assurance
2. QI Low High Focus – Leadership (visibility); Customer FocusMedium Focus – Innovation
3. QI Medium High Focus – Process; HR / Training; PlanningMedium Focus – Leadership (support)
4. QI High Medium Focus – Information; Supplier PartnershipLow Focus – Leadership (consistency); Planning
5.QI - Absorbed
Findings
The survey is easy to administer The survey provides a reliable and valid snapshot of
CQI implementation in a healthcare organization No known group exists The scale is a practical method of providing hospital
leaders with a roadmap for CQI implementation Leadership is the most important component of
implementing CQI
Submitted to Health Services Research
A Likely Future Scenario
1. Patient Safety provides accountability to analyze quality and outcomes
2. Healthcare leaders see CQI as a methodology to improve patient outcomes
3. Hospitals use the survey and scale to help assess & implement CQI appropriately, which in turn eliminates structure issues discussed
4. Because of this, hospitals can effectively assess their processes and improve their outcomes
Future Research Questions Can a clearer snapshot of CQI implementation emerge using
line worker responses, and senior leadership interviews? Administered the survey to a 40 random line workers at 10
hospitals Initial results include:
50% return rate high number of ‘I don’t know / NA’ responses all hospitals overall employee score < 100 (Quality Assurance)
Senior hospital leadership meetings to discuss quality structure Initial results include:
low knowledge of ‘quality’ among the senior leaders structures developed with little statistical or facilitation resources
Future Questions Does a higher level of CQI implementation lead to better
financial, quality and safety outcomes? If not, why?
Develop and Find Financial, Operational and HR Effectiveness and Efficiency MeasuresCounte & Glandon, 1995
Build one, clean database with CQI implementation scores and measures
Analyze to assess relationships
Future QuestionsIs CQI, as its described in the literature, an effective
method for improving quality outcomes? Change the four items that did not differentiate well
1. How many multi-disciplinary teams currently work to improve the processes of care in your organization? (Process)
more statistics, less teams• Human Factors Research• Toyota• Six Sigma
2. Do people in your organization know who their customers are? (Customer Focus)
not clear3. Are employees in this organization encouraged to try new and
better ways of doing things? (Innovation)4. Is creativity actively encouraged in this organization? (Innovation)
healthcare has typically not been innovative and so these questions may need to be more specific
THANK YOU