Date post: | 18-Dec-2015 |
Category: |
Documents |
Upload: | darleen-marshall |
View: | 212 times |
Download: | 0 times |
Implementing Evidence-Based Practices
Our Obligation to Program Fidelity
Kimberly Gentry Sperber, Ph.D.
Efforts To Date“What Works” Literature
Principles of Effective InterventionsGrowing evidence based on individual program evaluations and meta-analyses
Continuing Gap Between Science and Practice
Few programs score as satisfactory on CPAI
CPAI Data
Hoge, Leschied, and Andrews(1993) reviewed 135 programs assessed by CPAI
35% received failing score; only 10% received score of satisfactory or better.
Holsinger and Latessa (1999) reviewed 51 programs assessed by CPAI
60% scored as satisfactory but needs improvement or unsatisfactory; only 12% scored as very satisfactory.
CPAI Data Continued
Gendreau and Goggin (2000) reviewed 101 programs assessed by CPAI
Mean score of 25%; only 10% scored received satisfactory score
Matthews, Hubbard, and Latessa (2001) reviewed 86 programs assessed by CPAI
54% scored as satisfactory or satisfactory but needs improvement; only 10% scored as very satisfactory.
Fidelity Research
Landenberger and Lipsey (2005)Brand of CBT didn’t matter but quality of implementation did.Implementation defined as low dropout rate, close monitoring of quality and fidelity, and adequate training for providers.
Schoenwald et al. (2003) Therapist adherence to the model predicted post-treatment reductions in problem behaviors of the clients.
Henggeler et al. (2002)Supervisors’ expertise in the model predicted therapist adherence to the model.
Sexton (2001)Direct linear relationship between staff competence
and recidivism reductions.
More Fidelity Research
Schoenwald and Chapman (2007)A 1-unit increase in therapist adherence score predicted 38% lower rate of criminal charges 2 years post-treatmentA 1-unit increase in supervisor adherence score predicted 53% lower rate of criminal charges 2 years post-treatment.
Schoenwald et al. (2007)When therapist adherence was low, criminal outcomes for substance abusing youth were worse relative to the outcomes of the non-substance abusing youth.
Washington State Example(Barnowski, 2004)
• For each program (FFT and ART), an equivalent comparison/control group was created
• Felony recidivism rates were calculated for each of three groups, for each of the programs
• Youth who received services from therapists deemed ‘competent’
• Youth who received services from therapists deemed ‘not competent’
• Youth who did not receive any services (control group)
Family Functional Therapy Results: % New Felony
139 6
2519
11
3227
17
0
10
20
30
40
50
60
70
6 Months 12 Months 18 Months
FFT Not Competent
Control group
FFT Competent
Results calculated using multivariate models in order to control for potential differences between groups
UC Halfway House/CBCF Study in Ohio: A Look at Fidelity Statewide
Average Treatment Effect was 4% reduction in recidivismLowest was a 41% Increase in recidivismHighest was a 43% reduction in recidivism
Programs that had acceptable termination rates, had been in operation for 3 years or more, had a cognitive behavioral program, targeted criminogenic needs, used role playing in almost every session, and varied treatment and length of supervision by risk had a 39% reduction in recidivism
What Do We Know About Fidelity?
Fidelity is related to successful outcomes (i.e., reductions in recidivism, relapse, and MH instability).Poor fidelity can lead to null effects or even iatrogenic effects.Fidelity can be measured and monitored.Fidelity cannot be assumed.
Why Isn’t It Working?Latessa, Cullen, and Gendreau (2002)
Article notes 4 common failures of correctional programs:
Failure to use research in designing programsFailure to follow appropriate assessment and classification practicesFailure to use effective treatment modelsFailure to evaluate what we do
Ways to Monitor Fidelity
Training post-testsStructured staff supervision for use of evidence-based techniquesSelf-assessment of adherence to evidence-based practicesProgram audits for adherence to specific models/curriculaFocus review of assessment instrumentsFormalized CQI process
Ensuring Training Transfer
Use of knowledge-based pre/post-testsUse of knowledge-based proficiency testsUse of skill-based rating upon completion of trainingMechanism for use of data
Staff must meet certain criteria or score to be deemed competent.Failure to meet criteria results in consequent training, supervision, etc.
Staff Supervision
Staff supervision is a “formal process of professional support and learning which enables individual practitioners to develop knowledge and competence, assume responsibility for their own practice and enhance [client]… care in complex … situations.”
Modified from Department of Health, 1993
Performance Measurement for Staff
Standardized measurementConsistencyEveryone measured on same items the same way each time
Consistent meaning of what is being measured
Everyone has same understanding, speaks the same language
Sample Measures
Uses CBT language during encounters with clients.Models appropriate language and behaviors to clients.Avoids power struggles with clients.Consistently applies appropriate consequences for behaviors.Identifies thinking errors in clients in value-neutral way.
Sample Employee Observation Evaluation
Employee Name: Supervisor Name: Site: Observation Time (amount): Start Time: Stop Time: Place of Observation: Activity Observed: Behavior Below
Expectations Needs Improvement
Meets Expectations
Exceeds Expectations
Comments on Direct Observations
Uses corrective thinking language during encounters with clients.
Models appropriate language and behavior to clients. Includes: Speaking positively about program, law, courts, etc. Does not use derogatory language/jokes op sarcasm.
Avoids power struggles with clients (e.g., does not argue with clients, raise voice at clients, antagonize clients)
Consistently applies appropriate consequences for behaviors (both positive and negative)
Identifies thinking barriers in clients in value-neutral way
Overall Score
Agency Self-Assessment:Assessing Best Practices at 17
Sites
Use of ICCA Treatment Survey to establish baselineComplete again based on best practicePerform Gap AnalysisAction Plan
ICCA Treatment Survey
CQI Manager and Clinical Director met with key staff from each program to conduct self assessment of current practices.
Evaluated performance in 6 key areasStaffAssessment/ClassificationProgrammingAftercareOrganizational ResponsivityEvaluation
Agency Response:Strategic Plan
FY2006Required to submit at least 1 action plan to “fix” an identified gap.Gaps in the areas of risk and need to be given priority.
FY2007Required to submit 2 action plans.One on use of role-plays and one on appropriate use of reinforcements.
FY2008Proposed focus on fidelity measurement at all sites.Creation of checklists and thresholds.
Program Audits:CBIT Site Assessments
Cognitive Behavioral Implementation TeamSite visits for observation and ratingStandardized assessment processStandardized reports back to sitesCombination of quantitative data and qualitative data
Individual LSI ReviewsSchedule of videotaped interviewsSubmitted for reviewUse of standardized audit sheetFeedback loop for staff developmentAggregate results to inform training efforts
LSI Audit Form Reviewer: Date: Interviewer:
1 = Poor 2 = Fair/Needs Improvement 3 = Good 4 = Excellent 1 2 3 4
1. Explanation of the purpose of the interview. ❑ ❑ ❑ ❑
2. Established structure for the interview. ❑ ❑ ❑ ❑
3. Adequate use of open-ended questions. ❑ ❑ ❑ ❑
4. Avoidance of double-barreled questions. ❑ ❑ ❑ ❑
5. Avoidance of biased/leading questions. ❑ ❑ ❑ ❑
6. Adequate use of follow-up questions. ❑ ❑ ❑ ❑ 7. Avoided barriers to listening (such as moralizing, disagreeing,
Blaming, shaming, reinforcing). ❑ ❑ ❑ ❑
8. Interview overcame problems such as silence or excessive talking. ❑ ❑ ❑ ❑
9. Interviewer used the interview guide. ❑ ❑ ❑ ❑
10. Notes were made indicating why items were or were not scored. ❑ ❑ ❑ ❑
11. Adequate documentation in the case of an override. ❑ ❑ ❑ ❑
12. Treatment plan clearly relates to information captured in the LSI. ❑ ❑ ❑ ❑
Total score: ______ divided by _______ = Reviewer Comments:
Formal CQI Model
Data Collection/Review Requirements:Peer Review (documentation)MUI’s/IncidentsComplaints/GrievancesEnvironmental ReviewClient SatisfactionProcess IndicatorsOutcome Indicators
Formal CQI Model
Programs required to review data monthly and to action plan accordingly.Each program’s data and action plans reviewed once per quarter by agency’s Executive CQI Committee
CQI Committee Infrastructure
Oversight CQI Committee
Risk Management Committee
Safety Committee Human Subjects Committee
Diversity Committee Corporate Compliance Committee
Cluster CQI Committees
Program Peer Review Committees
Morbidity & Mortality Conference
The Talbert House Strategic PlanFocus on Fidelity
FY2008 – FY2010 Objective:Improve Quality of Client Services
FY2008 – FY2010 Goal:Exceed 90% of quality improvement measures annually
FY2008 – FY2010 Strategy:Talbert House programs demonstrate fidelity to best practice service/treatment models as demonstrated by site specific best practice fidelity check sheet.
100% of programs create a Fidelity measurement tool by 12/31/07. 100% of programs establish and measure its site-specific fidelity threshold by 1/30/08. Programs will be expected to meet/exceed established fidelity thresholds by 6/30/09.Programs will be expected to meet/exceed established fidelity thresholds by 6/30/10.
Getting Started
What services do you say/promise that you deliver?
What does your contract say?What do referral sources expect?
List all programming componentsWhat is the model (e.g., CBT, MI, IDDT, IMR, TFM, etc.)?What curricula are in use?
Identify which is most importantMake selection for measurement
Creating a Tool for Measurement
Scale should adequately sample the critical ingredients of the EBP.Need to be able to differentiate between programs/staff that follow the model versus those that do not.Scale should be sensitive enough to detect progress over time.Need to investigate what measurement tools may already exist.
Sample Measures
CBT Group Observation FormTFM Fidelity Review Sheets and DatabaseIMR Fidelity Rating ScaleIDDT Fidelity ScaleMotivational Interviewing Treatment Integrity (MITI) Code
Sample Project - TFM
4 residential adolescent programs implemented Teaching Family Model.Required to record all teaching interactions with all clients.Required to record data on standardized form and to enter into Fidelity database.CQI Indicator = percentage of staff achieving 4:1 ratio.
Sample Project – CBT Groups
Several programs conducting group observations using standardized rating form.Needed to operationalize who would do observations and how frequently.Needed to operationalize how data would be collected, stored, analyzed, and reported.CQI Indicator = percentage of staff achieving a rating of 3.0. (on scale of 0-3).
Measuring CBT in GroupsYear One
Chose 5 items from observation tool:Use of role plays/or other rehearsal techniquesAbility of the group leader to keep participants on task Use of peer interaction to promote prosocial behaviorUse of modelingUse of behavioral reinforcements
Measuring CBT in GroupsYear Two
Refinement of role-play indicators:Percentage of groups observed where staff modeled the skill prior to having clients engage in role-playPercentage of role-plays containing practice of the correctivesPercentage of role-plays that required observers to identify skill steps and report back to the group
Sample Project – Dosage by Risk and Need
Program created dosage grid by LSI-R risk category and criminogenic need domains.
Requires prescribed set of treatment hours by risk
Program created dosage report out of automated clinical documentation system.Review monthly to insure clients are receiving prescribed dosage.Also review individual client data at monthly staffings.CQI Indicator = percentage of successful completers receiving prescribed dosage (measured monthly).
Sample Dosage ProtocolCommunity Correctional Center Risk Level Structure Guide
MediumHigh High Medium Low/ Moderate Low
LSI Score Range 34+ 31-33 24-30 19-23 0-18Length of Stay Target (days) 147 133 119 105 60
Corrective Thinking 200 180 132 92 52AOD 62 54 46 38 28Individualized Relapse Prevention 21Anger Management 24 24 24 24 if neededDomestic Violence 24 15 15 15 if neededVocational* 15 15 15 15 8Life Skills* 16 16 16 16 8Personal Development* 10 10 10 10 if needed
*not counted in dosage total
Total hours available: 351 314 258 210 117
Sample Dosage Protocol
CT Groups
Total progam hours LSI range
CT--1 8 19 and below CT--2 12 20--26
CT--3 16 26 and above
CD Groups
Total program
hours Diagnosis
Level 1 12 Abuse Level 2 18 Dependence
Relationship Between Evaluation and
Treatment Effect (based on UC Halfway House and CBCF study)
6
1
0
2
4
6
8
Internal QA No Internal QA
% C
hang
e in
Rec
idiv
ism
NPC Research on Drug Courts
Drug Court Uses Evaluation Feedback to Make Modifications
11%
44%
0%
10%
20%
30%
40%
50%
YesN=4
NoN=6
* "Percent improvement in outcome costs" refers to the percent savings for drug court compared to business-as-usual
Per
cen
t Im
pro
vem
ent
in O
utc
om
e C
ost
s*
Conclusions
Many agencies are allocating resources to selection/implementation of EBP with no evidence that staff are adhering to the model.There is evidence that fidelity directly affects client outcomes.There is evidence that internal CQI processes directly affect client outcomes.Therefore, agencies have an obligation to routinely assess and assure fidelity to EBP’s.Requires a formal infrastructure to routinely monitor fidelity performance.