Date post: | 05-Dec-2014 |
Category: |
Health & Medicine |
Upload: | thl |
View: | 180 times |
Download: | 1 times |
Dr. Brian Rush
Centre for Addiction and Mental Health, Toronto, OntarioandDepartments of Psychiatry and School of Public Health, University of Toronto
EVALUATION AND QUALITY IMPROVEMENT IN SUBSTANCE USE TREATMENT AND SUPPORT: GOING FROM “SIMPLE” TO “COMPLEX” INTERVENTIONS.
SOME BASICS ABOUT EVALUATION
WHAT DO WE MEAN BY “EVALUATION”?
Evaluation involves the systematic collection of feedback about a program or intervention. The feedback is used in making decisions.
need assessment
participant satisfaction
outcome evaluation
economic evaluation
Different Types of Evaluation
Process
Need
Satisfaction
Outcome
Economic
WHY DO WE DO IT?
Accountability (service or system) - promises we make with public funds
Quality Improvement (service or system) – using feedback to improve what we do
Research/testing theory(service or system) – finding out things that others will be interested in and sharing it with them
Precontemplation – never think about it; actively avoid; only if pressured; don’t like to ask too many questions
Contemplation – do the minimum stats; ok but how do I start? Interested but afraid of the answers; went to a workshop once but no time to do it; bought a book about evaluation
Action – have a logic model and evaluation plan for one of my services; been talking to a researcher to help out; found a good way to start and made a proposal to the administration
Maintenance – wake up every morning thinking about how to evaluate my service; have a plan for each service; routine
DOES ANY ONE CARE ABOUT IT?
need assessment
participant satisfaction
outcome evaluation
economic evaluation
Establishing a Healthy Culture for Evaluation
The Importance of a Healthy Evaluation Culture
Process
Need
Satisfaction
Outcome
Economic
Treatment modality (CBT, motivational interviewing, naltrexone, methadone)
Service/program level (e.g., inpatient unit, day or evening drop-in service; family group, opiate substitution service)
Organization level (e.g., hospital, NGO, therapeutic community)
System/network level (e.g., substance use and mental health services, primary and urgent care, hospital and community networks, justice, education)
EVALUATION CAN BE DONE AT DIFFERENT LEVELS
PURPOSES AND LEVELS OF EVALUATION
Accountability/ Monitoring
Quality Improvement
Research/ Development
Service/ Intervention
Program/ Organization
System/Network
THE IMPORTANCE OF “EVIDENCE”
GOING FROM THEORY TO PRACTICE IN THE COMMUNITY
THE IMPORTANCE OF “EVIDENCE” IN SUBSTANCE USE TREATMENT AND
SUPPORT
Increase chances of really helping people (and at reasonable cost to society)
Research evidence does play a critical role in treated safely and respectfully
Research evidence is needed to ensure one’s “approach” gains the acceptance of professionals, funders and the community as a whole
Current challenge – there are many kinds of research and many ways of “knowing” something
THE “EVIDENCE PYRAMID” BEHIND CURRENT MEDICAL
PRACTICE
Meta-Analysis
Systematic Reviews
RCT”s
Cohort Studies
Case Control Studies
Case Reports
Animal Research
ADAPTATION OF THE PYRAMID FOR
QUALITATIVE EVIDENCE
I - Generalizable conceptual studies
II- Descriptive Studies
III- Single Case Studies
BUT THIS IS THE PYRAMID THAT”RULES”
Meta-Analysis
Systematic Reviews
RCT”s
Cohort Studies
Case Control Studies
Case Reports
Animal Research
EVOLVING MODELS OF RESEARCH EVIDENCE
More recognition of the limitations of RCT’s Who is actually represented in the research populations?
Does this research design really work for highly complex and system-level interventions?
More recognition now of “practice-based evidence” to complement evidence-based practice
More recognition of the challenges with this model in working with indigenous populations and studies of traditional medicine - a different view on the nature of “knowing”
CHALLENGE(S) MEASURING “SYSTEMS CHANGE”
CHALLENGE(S) MEASURING “SYSTEMS CHANGE”, “INTEGRATION”,
“COLLABORATION”
in most intervention research OUTCOME = intervention MINUS “noise” internal validity trumps external validity
in systems or complexity–based research OUTCOME = intervention PLUS context nothing can be understood outside of the context in which it exists
CHALLENGE #1
The goals or anticipated benefits of a systems change are not always shared or explicit among the key actors
HE SAYS:
“THIS IS THE PART I ALWAYS HATE!”
Substance Abuse
Treatment and Support
System
Mental Health
Treatment and Support
System
A process is underway in many countries to improve this problem
The personal and clinicalexperience of co-occurring disorders
Mental Disorders
Overlap in Canadian General Population
Substance Use
Disorders15-20%
Mental Disorders
Substance Use Services (Ontario)
Substance Use
Disorders70- 80%
Mental Disorders
Overall Mental Health System (Ontario)
Substance Use
Disorders15-20%
Mental Disorders
Young males in inpatient MH settings
Substance Use
Disorders55%
Mental Disorders
To help keep things in perspective…
Where are we with the justice system?
Substance Use
Disorders80-90%
WITH RESPECT TO THE INTEGRATION OF MENTAL HEALTH AND ADDICTIONS IN CANADA AND
ELSEWHERE …...
The train has clearly left the station!
ANTICIPATED BENEFITS OF “INTEGRATION”
To reduce stigma and discriminationTo improve access to servicesTo get better retentionTo improve continuity of careTo improve outcomes
??? ….. improved prevention of SU (e.g. addressing mental health and determinants of MH at young age)
DOES THE INTEGRATION OF HEALTH CARE SYSTEMS HAVE PREVENTION
IMPLICATIONS?
Prevention
Resources
Treatment
Resources
OTHER POSSIBLE GOALS???
Increased efficiency/less managementMore competitive positioning for resourcesInter-disciplinary issues of world
view/power/control (e.g., medical – non-medical)
Leveling the playing field (wages, credentialing, workforce mobility)
Expectations and perceived benefits are essentially “values based”
CHALLENGE # 2
“Systems Integration” does not mean the same thing to the various actors
MANY FIND IT HELPFUL TO CONSIDER “INTEGRATION” ALONG A CONTINUUM…
Full segregation
Co-operation
SO MANY WAYS TO CONSIDER “INTEGRATION”
Services vs Systems integration
Functional vs Horizontal
Normative integration (values based)
Information continuity: the use of information on past events and personal circumstances to make current care appropriate for each individual
Management continuity: a consistent and coherent approach to the management of a health condition that is responsive to a person’s changing needs
Relational continuity: an ongoing therapeutic relationship between a person and one or more providers
DIFFERENT ASPECTS OF INTEGRATION OR “CONTINUITY OF CARE”
SO MANY WAYS TO PUT “INTEGRATION” INTO PRACTICE
integrated clinical teams
centralized access and intake
shared electronic records
linkage managers/case management
the devil is in the details
every situation is unique
CHALLENGE # 3
“It is not clear how we should go about achieving “Systems Integration”
THE MECHANISMS/PROCESSES TO ACHIEVE IMPROVED “INTEGRATION” ARE NOT CLEAR OR EVIDENCE-BASED
May have a model in mind and its about “best practice” implementation – implementation science calls for active implementation supports
May be driven by a defined quality improvement process (Plan, Do, Study, Act) or change management process
Top down (Big world) vs bottom up (small world)Muddling along
Whatever the initial vision or model one typically works with the existing services and sub-systems as starting point(s) – leads to a basic strategy of “incrementalism” and ”contextual adaptation”
ESSENTIALLY A MIRACLE OCCURS….
THE MECHANISMS/PROCESSES TO ACHIEVE IMPROVED “INTEGRATION” ARE NOT CLEAR OR EVIDENCE-BASED
Whatever the initial vision or model, one typically works with the existing services and sub-systems as starting point(s) – leads to a basic process of “incrementalism” and ”contextual adaptation”
CHALLENGE # 4
“Health outcomes are often distal not proximal to the change efforts”
INTERMEDIARY OUTCOMES
(Process)Outcomes
- Access- Continuity of care- Information moves
Integration activity or
intervention
(Health)Outcomes
- SU- MH- QOL
CHALLENGE # 5
“Systems are never static – always responding to external context – can you really isolate the intervention from its dynamic context”
SYSTEMS ARE ALWAYS RESPONDING TO THE
ENVIRONMENT (TO SURVIVE)
SYSTEMS ARE ALWAYS RESPONDING TO THE
ENVIRONMENT (TO SURVIVE)
Aging population
Resource
AvailabilityNew needs
Political
reform
Health Determinants
CHALLENGE # 6
“Systems are never what they appear to be”
ARCHEOLOGICAL SYSTEMS AND ORGANIZATIONS
SecretsMemories
Myths
Old warriorsLies Truths
NEED TO DIG DEEP TO UNDERSTAND
SecretsMemories
Myths
Old warriorsLies Truths
CHALLENGE # 7
“Implementation of any systems change is never smooth – rarely do you end up with what you set out to achieve”
CHALLENGE # 8
“Routine information systems will always present challenges for evaluation of system change”
CHALLENGES WITH ROUTINE INFORMATION SYSTEMS
health data rarely collected for research and evaluation purposes
challenges with data linkage
mental health and addictions information can be lost
regionalization challenges commonality in data elements and quality
IS THERE ANY ROOM FOR OPTIMISM IN THE EVALUATION OF
INTEGRATION?
EMERGENT REALIST EVALUATION MODEL
Intervention + context = outcome
Systems analysis and complexity science
Mixed methods - qualitative and quantitative
Realist research synthesis and contribution analysis
Working on it and sharing the experiences
Thank you and good luck in your personal and professional work!
BUT WHAT DO WE DO?