Phyllis C. Panzano, Ph.D., PI Dee Roth, M.A., Co-PI Bev Seffrin, Ph.D, Senior Consultant Dushka...

transcript

Phyllis C. Panzano, Ph.D. , PI Dee Roth, M.A., Co-PI

Bev Seffrin, Ph.D, Senior Consultant Dushka Crane-Ross, Ph.D., Project Manager

Decision Support Services, Inc. Ohio Dept of Mental Health, OPER

The Innovation Diffusion and Adoption

Research Project (IDARP)

ODMH RESEARCH RESULTS BRIEFING 2003

Funded by the ODMH & the Mac Arthur Foundation

Ohio’s Quality Agenda

Outcomes

Salience

Coordinating Centers of Excellence (CCOEs)

Salience

Advance Directives

MH/Schools

MH/Criminal Justice OMAP

Family Psychoeducation

Cluster-Based Planning

MST SAMI-IDDT

o University or local partnership

o One Best Practice per CCOE

o Statewide service area

Structure of CCOEs

o Promotion of Best Practices

o Education & training

o Capacity development

o Fidelity measurement

o Cross-system sharing

Role of CCOEs

What factors and processes influence

the adoption, assimilation, and

impact of evidence-based practices by

mental health provider organizations?

Research Question

o Characteristics of the Best Practice

o Adoption Decision & Implementation Process

o Adopting Organization

o Adopting Organization – CCOE Relationship

Independent Variables

Research Team

Decision Support Services, Inc.

Phyllis Panzano, Ph.D

Beverly Seffrin, Ph.D.

Sheri Chaney, M.A.

Vandana Vadyanathan, M.A.

Sheau-yuen Yeo, M.A.

Ohio Dept of Mental Health

Dee Roth, M.A.

Dushka Crane-Ross, Ph.D.

Rick Massatti, M.A.

Carol Carstens, Ph.D.

Ohio State University: Fisher College of Business Department of Psychology

Theoretical Background

o Numerous literatures are relevantNumerous literatures are relevant

o Resulting Assumptions:Resulting Assumptions:

o EBPs are innovationsEBPs are innovations

o Scientific evidence necessary but not Scientific evidence necessary but not

sufficientsufficient

o Upper Echelon Theory relevantUpper Echelon Theory relevant

o Implementation effectiveness Implementation effectiveness Innovation Innovation

effectivenesseffectiveness

o Factors at many “levels” impact outcomesFactors at many “levels” impact outcomes

o 3 phases: initiation; decision; 3 phases: initiation; decision;

implementationimplementation

100-PIECE JIGSAW PUZZLE

COMPLEXITY

IDARP Models

Model 1:

Adoption Decision –Decision making under risk

Phase 1: Decision Under Risk

• IMPLEMENT

• ADOPTER

• WAIT & SEE

• NEVER

Perceived

Risk of

Adopting

Capacity to

Manage or

Absorb RiskRisk-

taking

Propensity

ANTECEDENTS

Likely

Less Likely

LIKELIHOOD OF

IMPLEMENTING

Model 2: Multi-level Influences on Implementation Success

Interested in Two Classes of Outcomes

o Measures of Measures of InnovationInnovation effectiveness: effectiveness:

Benefits that accrue to an organization and its Benefits that accrue to an organization and its

stakeholders as a result of implementing an innovative stakeholders as a result of implementing an innovative

practice (positive consequences for clients, staff, etc.)practice (positive consequences for clients, staff, etc.)

o Measures of Measures of ImplementationImplementation effectiveness: effectiveness:

Accurate, committed and consistent use of Accurate, committed and consistent use of

practice practice

by targeted employees (assimilation, fidelity, by targeted employees (assimilation, fidelity,

etc.)etc.)

Expected Link Between Two Classes of Outcomes

IMPLEMENTATION EFFECTIVENESS

INNOVATION EFFECTIVENESS

For example:

FIDELITY POSITIVEOUTCOMES

Variables at multiple levels are expected to impact these

two classes of outcomes

Examples of Variables by Level

Level ExampleENVIRONMENTENVIRONMENT System and professional norms System and professional norms

IOR (Org with CCOE)IOR (Org with CCOE) Quality of communicationQuality of communication

ORGANIZATIONORGANIZATION Learning cultureLearning culture

PROJECT PROJECT

Re: OrganizationRe: Organization

Re: DecisionRe: Decision

Re: Re: ImplementationImplementation

Availability of dedicated resourcesAvailability of dedicated resources

Commitment to decision to adoptCommitment to decision to adopt

Access to technical assistanceAccess to technical assistance

INNOVATIONINNOVATION Scientific supportScientific support

Experiential evidenceExperiential evidence

Level 4: Inter-organizational

Level 3: Adopting organization

Level 2: Project level

Level 1: Innovation level

Dependent Variables:

• Implementation effectiveness

• Innovation effectiveness

Level 5: EnvironmentModel 2

Model 3:

Cross-Phase Effects on Cross-Phase Effects on Implementation OutcomesImplementation Outcomes

INITIATION Decision

IMPLEMENTATION Outcomes

Model 3: Cross-phase effects

Experiential Evidence

Objective Process

Access to Technical

Assistance

Positive Consequen

Model 3: Examples of Cross-phase Effects

Initiation Decision

Implementation

Model 4:

Effects of Implementation Effects of Implementation Variables on Outcomes Variables on Outcomes

Over TimeOver Time

PRESENT OUTCOMES

Model 4: Effects of Implementation Variables Over Time

Implementation

PRESENT

Implementation

PRESENT OUTCOMES

Model 4: Examples of Effects of Implementation Variables Over

Access to Technical

Assistance

PRESENTDedicated Resources

Methods & Progress to Date

Four CCOEs Participating

Selection criteria maximize generalizabilitySelection criteria maximize generalizability

1)1) Cluster-Based Planning Alliance Cluster-Based Planning Alliance

2)2) Multi-systemic Therapy (CIP)Multi-systemic Therapy (CIP)

3)3) Ohio Medication Algorithm Project Ohio Medication Algorithm Project

4)4) Integrated Dual Disorder Treatment (IDDT) –Integrated Dual Disorder Treatment (IDDT) –New Hampshire - Dartmouth model New Hampshire - Dartmouth model

Research Design

o Longitudinal studyLongitudinal study

o Organizations at different stages of Organizations at different stages of adoptionadoption

o Multiple key informants at each Multiple key informants at each organizationorganization

o Quantitative and qualitative dataQuantitative and qualitative data

o Interviews, surveys & archival dataInterviews, surveys & archival data

Participating Projects*by Type of Innovation

*18 organizations involved in multiple projects; Total of 74 organizations with 91 projects under study.

Participating Projectsby Stage of Adoption at Time One

12 179

101520253035404550

N = 91

Participating Projectsby Stage of Adoption at Time Two

101520253035404550

N = 50

Key Informants by Level at Time One

aker C

rative

N = 369

Key Informants by Level at Time Two

rative

N = 135

Findings

Strongly disagree

Very satisfied

$ 37,500

Do the data support our four models?

THE TIP….OF THE TIP

___________________________________

POSITIVE CORRELATION

As the value of one variable increases,

the value of a second variable also increases

Median Income

Years of Formal Education

Higher

Less More

+ correlation (r = +1.00)

___________________________________

NEGATIVE CORRELATION

As the value of one variable increases,

the value of a second variable decreases

Unemployment

Higher

Less More

- correlation (r = -1.00)

___________________________________

ZERO ‘0’ CORRELATION

The relationship between the value of one variable and the value of a second variable is

random

Shorter

Taller

Less More

Zero Correlation

Height

(r = 0.00)

___________________________________

CORRELATION CANNOT BE DETERMINED

because the value of one (or both) variable(s) is constant or almost constant

Higher

Years of Formal Education = BA, BS

Unemployment

The Adoption Decision (Model 1)

Time 1/First contact data

Phase 1: A Decision Under Risk

Likelihood of implementing as

indicated by Stage

Implementer Adopter Wait & See Never

Perceived

Risk of

Adopting

Capacity to

Manage or

Absorb Risk

Risk-taking

Propensity

Antecedents to Risk Perceptions

Likelihood of implementing as

indicated by Stage

Implementer Adopter Wait & See Never

Perceived

Risk of

Adopting

Capacity to

Manage

Risk-taking

Propensity

ANT E C E D ENT S

Antecedents to Perceived Risk

Perceived

Innovation Level Factors

• Relative Advantage

• Scientific Evidence

• Experiential Evidence

Org-Level Factors

• Knowledge Set

Environmental Factors

• Norms for Adoption

Antecedents to Risk Management

Capacity to

Manage Risk

EBP–Level Factors

Ease of Use

Org–Level Factors

Top Mgmt. Support

Environmental Factors

Environmental uncertainty

Craft Skills+ .2

Dedicated Resources

Antecedents to Risk Propensity

Organization-Level Factors

Learning Encouragement

Managerial Attitude

About Change

Propensity

Summing Up: Model 1

3. Antecedents have implications for action3. Antecedents have implications for action

1. Adoption decision is a decision 1. Adoption decision is a decision involving riskinvolving risk2. Organizations are more likely to adopt 2. Organizations are more likely to adopt if:if:

Perceived risk of adopting is lowPerceived risk of adopting is low

Capacity to manage risk is highCapacity to manage risk is high

Propensity to take risks is highPropensity to take risks is high

Model 2: Implementation Phase

Understanding Outcomes of Understanding Outcomes of Implementation Implementation

for Adopters and Implementersfor Adopters and Implementers

Time 2/Second Contact Data

Two classes of outcomes

1. Implementation Effectiveness

(e.g., fidelity, assimilation)

2. Innovation/practice Effectiveness

(e.g., positive outcomes)

Is implementation effectiveness related to innovation effectiveness?

Reinvention1

Positive outcomes

Assimilation Positive outcomes

1 Self report; reflects extent to which practice was modified

Level 4: Inter-organizational

Level 3: Adopting organization

Level 2: Project level

Level 1: Innovation level

Dependent Variables:

• Implementation effectiveness

• Innovation effectiveness

Level 5: Environmental Model 2

Assimilation: One measure of

implementation effectiveness

Is Assimilation Explained by Variables at Multiple Levels?

DYAD: Communication quality +.45

ORG: Learning culture +.30

Centralization +.43

PROJECT: Dedicated resources +.52

Ease of use +.40

INNOV: Fit w/Tx philosophy +.45

Dependent Variable: Assimilation*

* Extent practice seen as part of permanent operations

Some Examples

Variables at multiple levels are related to reported

assimilation.

Are views about positive outcomes explained by

variables at multiple levels?

Positive outcomes

Overall positive consequencesOverall positive consequences Positive outcomes for consumersPositive outcomes for consumers Positive impact on organization’s imagePositive impact on organization’s image Positive impact on organization Positive impact on organization

functioningfunctioning Overall positive impactOverall positive impact Extent expectations realizedExtent expectations realized

DYAD: Identification +.40 to +.60

ORG: Risk mgmt. +.27 to +.40

PROJECT: Perf. monitoring +.52 to +.74

Access to TA +.48 to +.66

Reinvention -.24 to -.49

INNOV: Scientific evidence +.30 to +.60

Dependent Variable: Positive Outcomes

Some Examples

Variables at multiple levels are related to perceived positive outcomes.

Model 3: Cross-Phase Effects on Implementation Outcomes

Understanding Effects of Initiation-Phase Understanding Effects of Initiation-Phase and Decision-Phase Variables on and Decision-Phase Variables on

Implementation Outcomes Implementation Outcomes

INITIATION Decision

IMPLEMENTATION Outcomes

Outcomes

TimeTimeTimeTime

Model 3: Cross-Phase Effects Cross-Phase Effects on Implementation Outcomeson Implementation Outcomes

Time 1Time 1 Time 1Time 1 Time 2Time 2

INITIATION(Time 1)(Time 1)

Decision

IMPLEMENTATION OutcomeOutcomess

(Time 2)

OutcomeOutcomess

(Time 2)

TimeTime

Model 3: Initiation-Phase Initiation-Phase EffectsEffects

Expected Benefits +.44Relative advantage +.63Trust CCOE +.46Results demonstrability +.49

Assimilation

Initiation-Phase Effects

Expected Benefits +.58 to +.69Relative advantage +.59 to +.74 Trust CCOE +.38 to +.57Results demonstrability +.26 to +.51

Positive Outcomes

Initiation-Phase Effects

INITIATION Decision

(Time (Time 1)1)

IMPLEMENTATION

TimeTime

Model 3: Decision-Phase EffectsDecision-Phase Effects

Outcomes

(Time 2)

Outcomes

(Time 2)

Objective decision +.37Information access +.34Internal influence +.28Organizational commitment +.37

Assimilation

Decision-Phase Effects

Objective decision +.46 to +.71Information access +.42 to +.61Internal influence +.31 to +.46Organizational commitment +.42 to +.52

Positive Outcomes

Decision-Phase Effects

Variables in earlier phases can have enduring effects on

implementation outcomes.

Model 4: Understanding Effects of Implementation-Phase Variables Over Time

PRESENT OUTCOMES

Implementation

PRESENT

Implementation

PRESENT

Implementation

Top Support

PRESENT

Top Support

PRESENT

Top Support

PRESENT OUTCOMES

Freedom to Express Doubt

PASTFreedom to

Express Doubt

PRESENTFreedom to

Express Doubt

PRESENTFreedom to

Express Doubt

PRESENT OUTCOMES

Access to Technical

Assistance

PASTAccess to Technical

Assistance

PRESENT OUTCOMES

PRESENTAccess to Technical

Assistance

PRESENTAccess to Technical

Assistance

Dedicated Resources

PASTDedicated Resources

PRESENTDedicated Resources

PRESENT OUTCOMES

Implementation strategies need to be sustained in order to have positive impacts on long-term

outcomes.

Shifting Gears:Comparing Different EBPs

at Time One

Do adopting organizations hold similar views about the four

practices?

Clustering Clustering (n = 23)(n = 23) MST MST (n = 16)(n = 16) OMAP OMAP (n = 15)(n = 15) IDDT/SAMI IDDT/SAMI (n = 16)(n = 16) IDDT/SAMI with Initial Funding IDDT/SAMI with Initial Funding (n = (n = 1212**))

* 9 funded demonstrations; 12 projects

Organization: Organizational Commitment

MST, IDDT & IDDT-FUNDED > OMAP

5.4 6.15 5.7 5.9

Adopter

Strongly Agree

StronglyDisagree

Innovation: Experiential Evidence MST, IDDT-FUNDED & IDDT > CBP > OMAP

4.15.5 5.8

Adopter

Strongly Agree

StronglyDisagree

Innovation: Scientific Evidence MST, IDDT-FUNDED & IDDT > CBP & OMAP

4.76.4

4.55.9 6.3

Adopter

Strongly Agree

StronglyDisagree

Innovation: Magnitude of Change Required to Implement

MST, IDDT-FUNDED & IDDT > CBP & OMAP

2.66.4 6.5

Adopter

Innovation: Fidelity Seen as Crucial to

Implementing the Practice MST > OMAP, CBP, IDDT & IDDT-FUNDED

3.95.4

4.4 3.8 3.5

Adopter

Strongly Agree

StronglyDisagree

Implementation: Resources* for Initial Implementation

No differences

4.2 4.7 5 4.75.5

Adopter

Strongly Agree

StronglyDisagree

* money, personnel & time

Implementation: Resources* for Ongoing Implementation OMAP > CBP, MST & IDDT > IDDT-FUNDED

4.2 4.25 4.4

Adopter

Strongly Agree

StronglyDisagree

* money, personnel & time

Implementation: Problems Recruiting Staff

MST > CBP, OMAP, IDDT & IDDT-FUNDED

3.6 4 4.9

Adopter

GreatExtent

No Extent

Implementation: Reinvention No differences

2.8 2.3 2.8 3.3 3.5

Adopter

Very GreatExtent

No Extent

Outcome: Assimilation CBP, OMAP & IDDT > IDDT-FUNDED

5.85 5.7 6

Adopter

Very GreatExtent

No Extent

A peek at

interview data

Approach

Interview team produces transcripts from Interview team produces transcripts from interviews with multiple agency sourcesinterviews with multiple agency sources

Qualitative Qualitative “codes”“codes” attached to text in Atlas Ti attached to text in Atlas Ti Software program such as:Software program such as:

Issue diagnosis & decision processIssue diagnosis & decision process Planning process for EBP implementationPlanning process for EBP implementation Facilitators & BarriersFacilitators & Barriers Expected/unexpected, +/- outcomesExpected/unexpected, +/- outcomes

Unit of Analysis =Unit of Analysis = “mentions” “mentions” or coded phrasesor coded phrases

Focus of today’s look at qualitative data

Data collected during time one/first contact Data collected during time one/first contact with 36 projects (~ 3 interviews per)with 36 projects (~ 3 interviews per)

Projects X StageProjects X Stage 18 Implementers18 Implementers 7 Adopters7 Adopters 7 Wait & See/Never 7 Wait & See/Never 4 De-adopters4 De-adopters

Projects X EBPProjects X EBP 13 IDDT13 IDDT 9 Cluster-Based Planning9 Cluster-Based Planning 7 OMAP7 OMAP 7 MST7 MST

Glossary of Categories

CCOECCOE - relating to the CCOE, its staff and - relating to the CCOE, its staff and services it provides.services it provides.

EBPEBP – perceptions relating to the innovation. – perceptions relating to the innovation.

MoneyMoney - expenses (actual or anticipated), - expenses (actual or anticipated), funding of the EBP and financial issues that funding of the EBP and financial issues that impact the agency.impact the agency.

Staff Staff - reactions, recruitment, retention and - reactions, recruitment, retention and qualifications of staff. qualifications of staff.

System System - coordination, collaboration, and - coordination, collaboration, and interest in Mental Health and other related interest in Mental Health and other related systems.systems.

CCOE: Major Themes

Facilitators (n = 119)Facilitators (n = 119)

Attended CCOE Attended CCOE presentation/became presentation/became aware of CCOE/had aware of CCOE/had previous experience previous experience with (n = 55)with (n = 55)

CCOE provides CCOE provides instrumental help instrumental help (n = 50)(n = 50)

Positive reaction to Positive reaction to CCOE (n = 11)CCOE (n = 11)

Barriers (n = 36)Barriers (n = 36)

CCOE doesn’t CCOE doesn’t understand the understand the agency’s issues or agency’s issues or constraints (n = 9)constraints (n = 9)

CCOE: Mentions by Stage

Wait & See/

Adopter Implementer Deadopter

Facilitators Barriers

EBP: Major Themes

Received training or Received training or information (n = 53)information (n = 53)

EBP is good match with EBP is good match with culture, systems, or culture, systems, or similar to what we similar to what we already do (n = 24)already do (n = 24)

EBP might be useful EBP might be useful (n = 19)(n = 19)

Don’t know how to Don’t know how to proceed – in the dark proceed – in the dark (n = 24)(n = 24)

EBP isn’t a good fit to EBP isn’t a good fit to this organization this organization (n = 13)(n = 13)

EBP might NOT be EBP might NOT be useful (n = 13)useful (n = 13)

EBP: Mentions by Stage

Wait & See/

Money: Major Themes

Received funds (n = Received funds (n = 31)31)

Identified potential Identified potential funds (n = 24)funds (n = 24)

There is a potential There is a potential savings from the EBP savings from the EBP (not necessarily for the (not necessarily for the agency) (n = 6)agency) (n = 6)

Agency has financial Agency has financial issues/EBP costs issues/EBP costs money (n = 80)money (n = 80)

Funding for the EBP is Funding for the EBP is not sustainable (n = not sustainable (n = 29)29)

Our funding Our funding shrunk/was lost/ended shrunk/was lost/ended (n = 11)(n = 11)

Money: Mentions by Stage

Wait & See/

Staff: Major Themes Facilitators (n = 122)Facilitators (n = 122)

Staff is interested/ Staff is interested/ supportive/likes the supportive/likes the EBP (n = 25)EBP (n = 25)

Staff hired for program Staff hired for program (n = 18) (n = 18)

Staff thinks the EBP Staff thinks the EBP makes sense (n = 8)makes sense (n = 8)

Recruitment & Recruitment & turnover issues (n = turnover issues (n = 67)67)

Resistance to EBP, Resistance to EBP, skepticism, lack of skepticism, lack of interest (n = 44)interest (n = 44)

Competing priorities Competing priorities (e.g. innovation vs. (e.g. innovation vs. productivity) (n = 15)productivity) (n = 15)

Staff: Mentions by Stage

Wait & See/

System: Major Themes

Support and interest in Support and interest in the system (n = 34)the system (n = 34)

Collaboration/ Collaboration/ cooperation/ cooperation/ communication & communication & integration in the integration in the system (n = 33)system (n = 33)

Lack of support/no Lack of support/no interest (n = 29)interest (n = 29)

Conflict, lack of Conflict, lack of collaboration between collaboration between important entities, no important entities, no communication (n = 29)communication (n = 29)

Competing priorities & Competing priorities & turmoil in system (n = turmoil in system (n = 9)9)

System: Mentions by Stage

Wait & See/

Summing Up: Facilitator/Barrier Analysis - Category

Overall, facilitators Overall, facilitators were mentioned were mentioned more often than more often than barriers (641:557).barriers (641:557).

EBP: The category EBP: The category with the most with the most mentions of mentions of facilitators (225);facilitators (225);

Staff: The category Staff: The category with the most with the most mentions of mentions of barriers (171).barriers (171).

System

StaffSystem

Facilitator & Barrier Analysis -Phase

Facilitators and barriers can usually be Facilitators and barriers can usually be identified as occurring during specific identified as occurring during specific phasesphases of the process. of the process.

The next analysis separates most of the The next analysis separates most of the same “mentions” in terms of phase in same “mentions” in terms of phase in which they occurred.which they occurred.

Facilitator & Barrier Analysis – Initiation Phase

Initiation PhaseInitiation Phase – A facilitator or barrier – A facilitator or barrier that is anticipated or experienced PRIOR that is anticipated or experienced PRIOR to the adoption decision. to the adoption decision.

Initiation PhaseInitiation Phase Facilitators = 229 Facilitators = 229 Initiation PhaseInitiation Phase Barriers = 91 Barriers = 91

Mentions during Initiation Phase

Wait & See/

Early in the Implementation Phase

Early ImplementationEarly Implementation – A facilitator or – A facilitator or barrier that is anticipated or barrier that is anticipated or experienced AFTER the adoption experienced AFTER the adoption decision, but before full implementation. decision, but before full implementation.

Early ImplementationEarly Implementation Facilitators = 166 Facilitators = 166 Early Implementation Early Implementation Barriers = 122Barriers = 122

Mentions during Early Implementation

Wait & See/

Implementation Phase

ImplementationImplementation – A facilitator or barrier – A facilitator or barrier that is anticipated or experienced that is anticipated or experienced AFTER the agency begins to implement AFTER the agency begins to implement the EBP.the EBP.

ImplementationImplementation Facilitators = 155 Facilitators = 155 ImplementationImplementation Barriers = 250 Barriers = 250

Mentions during Implementation Phase

Wait & See/

Summing Up: Facilitator/Barrier Analysis - Phase

Initiation:Initiation: Facilitators are mentioned more Facilitators are mentioned more than TWICE AS FREQUENTLY as barriers.than TWICE AS FREQUENTLY as barriers.

Early Implementation:Early Implementation: Adopters & Adopters & Implementers mention 50% more facilitators Implementers mention 50% more facilitators than barriers (trend not seen in Wait & than barriers (trend not seen in Wait & See/Never or Deadopters).See/Never or Deadopters).

Implementation:Implementation: While there are more While there are more barriers than facilitators mentioned throughout barriers than facilitators mentioned throughout the Implementation phase, Deadopters the Implementation phase, Deadopters mention nearly FOUR TIMES more barriers mention nearly FOUR TIMES more barriers than facilitators.than facilitators.

So what does all this mean?

Major Messages

Adoption decision is a risky decision Adoption decision is a risky decision

Implementation effectiveness related to but Implementation effectiveness related to but not equal to innovation effectivenessnot equal to innovation effectiveness

Factors at many levels contribute to successFactors at many levels contribute to success

What happens early (e.g., during initiation) can What happens early (e.g., during initiation) can have enduring effects have enduring effects

Present implementation climate explains Present implementation climate explains present outcomespresent outcomes

SOME IMPLICATIONS

Phyllis C. Panzano, Ph.D., PI Dee Roth, M.A., Co-PI Bev Seffrin, Ph.D, Senior Consultant Dushka...

Documents