Post on 21-Jan-2016
description
transcript
1
Process Evaluation of the Process Evaluation of the King County Family King County Family
Treatment CourtTreatment CourtPreliminary ResultsPreliminary Results
Eric J. Bruns, Tracy JonesEric J. Bruns, Tracy JonesJustin D. Smith, Eric TrupinJustin D. Smith, Eric Trupin
University of WashingtonUniversity of WashingtonDivision of Public Behavioral Health and Division of Public Behavioral Health and
Justice PolicyJustice Policy
March 30, 2006March 30, 2006
2
Components of the Components of the EvaluationEvaluation
Data element and data collection Data element and data collection assessmentassessment
Process evaluationProcess evaluation Interviews with team membersInterviews with team members Comparison study of short-term outcomes Comparison study of short-term outcomes
for KCFTC participants vs. matched for KCFTC participants vs. matched comparison groupcomparison group
Parent interviewsParent interviews Record reviews (KCFTC, TARGET, and CAMIS)Record reviews (KCFTC, TARGET, and CAMIS)
Outcomes evaluation designOutcomes evaluation design Cost-benefit analysis designCost-benefit analysis design
3
Process EvaluationProcess EvaluationKCFTC PROCESSES AND FUNCTIONS•Comprehensive SB Assessment•High quality, appropriate CD Services•Timely/effective MH and other services•Effective care planning and management•Expanded and more frequent visitation•Consistent, timely incentives & sanctions•Random UA Screens•Effective pre-hearing case conferences•Effective judicial interaction
SHORT-TERM OUTCOMES•Eligibility/enrollment completed quickly•Enrollment in appropriate CD services•Parents compliant with/complete treatment•Parents ultimately able to remain sober•Parents/children more fully engaged in svcs•Parents/children receive needed services•Decreased placement disruptions•Parents compliant with court orders•Less negative effect on child well-being•Less disruption of child-parent bonds•Increased family reunification rates•Earlier determination. of alternate placement options
NATIONAL BEST PRACTICES•Communication bw Court and providers•Judge plays active role in Tx process•Judge responds to positive & noncompliant beh.•Mechanisms for shared decision making•Accountability for Tx services•Strategy for responding to noncompliance•MIS allow data to be assembled/reviewed•Enhancement of due process•Team members provided adeq resources
4
MethodMethod Interviews with KCFTC staff and key Interviews with KCFTC staff and key
informants (e.g., advisory group members)informants (e.g., advisory group members) N=35 initially identified; 5 deemed not N=35 initially identified; 5 deemed not
appropriateappropriate N=9 added through staff hires and identification N=9 added through staff hires and identification
by Evaluation Advisory Committeeby Evaluation Advisory Committee Total N=39; 34 completed, 5 scheduled or Total N=39; 34 completed, 5 scheduled or
pendingpending Average interview time = 63 minutes Average interview time = 63 minutes
(range: 40-80 min)(range: 40-80 min)
5
Staff and key informant Staff and key informant interviewinterview
Respondent informationRespondent information General questions about KCFTC goals, General questions about KCFTC goals,
target population, and overall successtarget population, and overall success Eligibility and referral processEligibility and referral process KCFTC process and functionsKCFTC process and functions Adherence to best practices (child welfare Adherence to best practices (child welfare
and treatment courts)and treatment courts) Short-term outcomesShort-term outcomes KCFTC teamwork and collaborationKCFTC teamwork and collaboration Ratings of individual team members’ Ratings of individual team members’
effectivenesseffectiveness Open-ended questions on strengths, Open-ended questions on strengths,
weaknesses, and areas for improvementweaknesses, and areas for improvement
6
RespondentsRespondents
7
Respondents Respondents (total N=34)(total N=34)
Respondent typeRespondent type NN %%
Advisory Group memberAdvisory Group member 99 27%27%
DSHS Social WorkerDSHS Social Worker 44 12%12%
Court Appointed Special Advocate (CASA)Court Appointed Special Advocate (CASA) 44 12%12%
Attorney (Parent or Child)Attorney (Parent or Child) 55 15%15%
JudgeJudge 22 6%6%
CASA Manager or SupervisorCASA Manager or Supervisor 22 6%6%
Social Work Supervisor, Program Social Work Supervisor, Program Coordinator, Family Treatment Court Coordinator, Family Treatment Court Specialist, Wraparound Coordinator, Specialist, Wraparound Coordinator, Treatment Provider, Treatment Provider Treatment Provider, Treatment Provider Supervisor, AAG, OtherSupervisor, AAG, Other
1 Each1 Each 22%22%
Team member interviews conducted 19 (56%)Key informant interviews conducted 15 (44%)
8
Time Spent Directly with Time Spent Directly with KCFTC Court or Families KCFTC Court or Families
(n=33)(n=33)
0%
5%
10%
15%
20%
25%
30%
35%
Percent 30% 21% 18% 3% 27%
<10% 11-25% 26%-50% 51%-99% 100%
22 respondents
21 respondents
Survey Survey RespondentsRespondents
9
Results, part 1:Results, part 1:Major QuestionsMajor Questions
10
Overall SuccessOverall Success“How successful do you feel the KCFTC has been in “How successful do you feel the KCFTC has been in
accomplishing its goals overall?”accomplishing its goals overall?”
0%
10%
20%
30%
40%
50%
All Staff 3% 3% 47% 35% 9%
Not at all successful
A little bit successful
Somewhat Successful
Moderately sucessful
Extremely successful
11
Overall Success Overall Success (n=33)(n=33)“How successful do you feel the KCFTC has been in “How successful do you feel the KCFTC has been in
accomplishing its goals overall?”accomplishing its goals overall?”
0%
10%
20%
30%
40%
50%
60%
70%
80%
Time <25% 0% 6% 29% 53% 12%
Time >25% 6% 0% 69% 19% 6%
Not at all successful
A little bit successful
Somewhat successful
Moderately sucessful
Extremely successful
Time <25%: Mean 3.71 Std Dev. .772 Time >25%: Mean 3.19 Std. Dev. .834
Between Group Comparison: Front-line (>25% with Between Group Comparison: Front-line (>25% with KCFTC) versus Other (<25%) Team MembersKCFTC) versus Other (<25%) Team Members
12
Overall Success Overall Success (compared to (compared to regular court)regular court)
“Compared to regular dependency court process, how “Compared to regular dependency court process, how successful do you feel the KCFTC is in accomplishing successful do you feel the KCFTC is in accomplishing its overall goals for participating families?” (n=33)its overall goals for participating families?” (n=33)
0%
5%
10%
15%
20%
25%
30%
All Staff 6% 6% 21% 27% 27% 12%
Somewhat less
About the same
A little bit more
Somewhat more
A good deal more
Much more succesful
13
Overall Success Overall Success (compared to (compared to regular court)regular court)
“Compared to regular dependency court process, how “Compared to regular dependency court process, how successful do you feel the KCFTC is in accomplishing successful do you feel the KCFTC is in accomplishing its overall goals for participating families?” (n=33) its overall goals for participating families?” (n=33)
0%
10%
20%
30%
40%
50%
60%
Time <25% 6% 6% 12% 35% 29% 12%
Time >25% 6% 6% 29% 53% 12%
Somewhat less
About the same
A little bit more
Somewhat more
A good deal more
Much more successful
Time <25%: Mean 5.12 Std Dev. 1.317 Time >25%: Mean 4.73 Std. Dev. 1.335
Between Group Between Group ComparisonComparison
14
Success Serving Target Success Serving Target PopulationPopulation
“How successful do you believe the KCFTC has been in “How successful do you believe the KCFTC has been in serving its target population?” (n=33)serving its target population?” (n=33)
0%5%
10%15%20%
25%30%35%40%45%
All Staff 3% 12% 36% 39% 9%
Not at all successful
A little bit successful
Somewhat Successful
Moderately sucessful
Extremely successful
15
Success Serving Target Success Serving Target PopulationPopulation
“How successful do you believe the KCFTC has been in “How successful do you believe the KCFTC has been in serving its target population?” (n=32)serving its target population?” (n=32)
0%
10%
20%
30%
40%
50%
60%
Time <25% 0% 6% 31% 50% 12%
Time >25% 6% 13% 44% 31% 6%
Not at all successful
A little bit successful
Somewhat successful
Moderately sucessful
Extremely successful
Time <25%: Mean 3.69 Std Dev. .793 Time >25%: Mean 3.19 Std. Dev. .981
Between Group ComparisonBetween Group Comparison
16
Success Serving Representative Success Serving Representative PopulationPopulation
“How would you describe the KCFTC’s success in serving “How would you describe the KCFTC’s success in serving a population that reflects the race, ethnicity, and gender a population that reflects the race, ethnicity, and gender of the general population of parents involved in the DCFS of the general population of parents involved in the DCFS
dependency system?” (n=30)dependency system?” (n=30)
0%
5%
10%
15%
20%
25%
30%
35%
All Staff 23% 23% 30% 17% 7%
Not at all successful
A little bit successful
Somewhat Successful
Moderately sucessful
Extremely successful
17
Success Serving Representative Success Serving Representative PopulationPopulation
“How would you describe the KCFTC’s success in serving a “How would you describe the KCFTC’s success in serving a population that reflects the race, ethnicity, and gender of population that reflects the race, ethnicity, and gender of the general population of parents involved in the DCFS the general population of parents involved in the DCFS
dependency system?” (n=29)dependency system?” (n=29)
0%
5%
10%
15%
20%
25%
30%
35%
40%
Time <25% 36% 21% 36% 7% 0%
Time >25% 13% 20% 27% 27% 13%
Not at all successful
A little bit successful
Somewhat successful
Moderately sucessful
Extremely successful
Time <25%: Mean 2.14 Std Dev. 1.027 Time >25%: Mean 3.07 Std. Dev. 1.280
Between Group ComparisonBetween Group Comparison
18
Summary of Major Summary of Major QuestionsQuestions
00.5
11.52
2.53
3.54
<25% 3.71 3.69 2.14
Front line 3.19 3.19 3.07
All respondents 3.45 3.39 2.6
Overall SuccessServing target population
Serving representative population
19
KCFTC CapacityKCFTC Capacity“In your opinion is the KCFTC currently serving too “In your opinion is the KCFTC currently serving too many families for its capacity, too few, or just about many families for its capacity, too few, or just about
right?” (n=32)right?” (n=32)
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
All Staff 13% 81% 6%
Too Few Just About Right Too Many
20
Success of Referral and Success of Referral and Eligibility ProcessEligibility Process
“Overall, how successful do you think the KCFTC “Overall, how successful do you think the KCFTC referral and eligibility process has been?” (n=30)referral and eligibility process has been?” (n=30)
0%
5%
10%
15%
20%
25%
30%
35%
All Staff 7% 30% 29% 26% 3% 11%
Not at all successful
A little bit successful
Somewhat Successful
Moderately sucessful
Extremely successful
Don't Know
21
FTC CollaborationFTC Collaboration“A primary component of the KCFTC is “non-adversarial “A primary component of the KCFTC is “non-adversarial teamwork, in which team members know each other as teamwork, in which team members know each other as
individuals.” How successful have the KCFTC team members individuals.” How successful have the KCFTC team members and agencies been in achieving this ideal?” (n=34)and agencies been in achieving this ideal?” (n=34)
0%
5%
10%
15%
20%
25%
30%
35%
40%
All Staff 0% 15% 35% 38% 12%
Not at all successful
A little bit successful
Somewhat Successful
Moderately sucessful
Extremely successful
22
FTC Shared VisionFTC Shared Vision“Please rate the extent to which there is a shared vision “Please rate the extent to which there is a shared vision
among the different FTC team members. (n=34)among the different FTC team members. (n=34)
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
All Staff 15% 27% 41% 18% 12%
No shared vision
A little bit of a shared vision
Some shared vision
A good amount of shared vision
Substantial shared vision
23
Results, part 2:Results, part 2:KCFTC Processes KCFTC Processes
and Functionsand Functions
24
3.55
3.82
3.09
3.39
4.13
3.03
3.88
3.61
3.77
3.09
3.29
3
2 2.5 3 3.5 4 4.5 5
Comprehensive SB Assessment
High quality, appropriate CD Services
Timely/effective other services (parents)
Timely/effective other services (children)
Care planning and management
Expanded and more freq visitation
Consistent, timely incentives & sanctions
Random UA Screens
Effective pre-hearing case conferences
Training and education for FTC staff
Effective judicial interaction
FTC staff collab with other agencies
Summary of Process and Function Questions
A little bit successful
Moderately
Somewhat
Extremely
25
Results, part 3:Results, part 3:Achievement of Achievement of Best PracticesBest Practices
26
4.75
4.7
5.1
4.94
3.75
4.79
5.71
4.23
4.67
5.47
5.48
5.53
3 3.5 4 4.5 5 5.5 6
Court+providers maintain communication
Judge plays active role in Tx process
Judge responds to positive & neg behavior
Mechanisms for shared decision making
Accountability for Tx services
AOD testing frequent in first months
Strategy for responding to noncompliance
MIS allow data to be assembled/reviewed
Children protected from abuse and neglect
Children kept safely in their homes possible
Enhancement of due process
Team members provided adeq resources
Summary of Best Practices Questions
About the same Somewhat more
Little more successful
Good deal more
27
Results, part 4:Results, part 4:Proposed KCFTC Proposed KCFTC
OutcomesOutcomes
28
4.73
4.87
4.97
5.19
5.09
4.97
4.9
4.7
5.36
4.65
5.13
5.3
3 3.5 4 4.5 5 5.5 6
Eligibility/enrollment completed quickly
Enrollment in appropriate CD services
Parents compliant with/complete Tx
Ultimately able to be and remain sober
Parents/childr more fully engaged in svcs
Parents/ch receive services they need
Decreased plcement disruptions
Parents compliant with court orders
Less negative effect on child well-being
Less disruption of child-parent bonds
Increased family reunification rates
Earlier determ. of alternate plcmt options
Summary of Outcomes Questions
About the same Somewhat more
Little more successful
Good deal more
29
Results, part 5:Results, part 5:Open ended Open ended questionsquestions
30
Gaps in Resources, Barriers, Gaps in Resources, Barriers, and other issues that need to and other issues that need to
be addressed.be addressed.THEME Responses
Percent of Total (n=231)
Client-Related Needs 39 18% Resources and Services 28 12%
Housing 12
Visitations 10
Transportation 4
Day Care 2
Focus on the Child 3
Selection 2
Outreach 2
Other 2
Support 1
Outcomes 1
Treatment Issues 38 16% Choice of Providers 16
Other/Unspecified 14
Wraparound 4
Inpatient Treatment 2
Strength-Based Approach 2
31
THEME ResponsesPercent of Total
(n=231)
Communication and Collaboration 29 13% Team Building 17
Staffings 5
Wraparound 4
Uniformity 2
Other 1
Funding Needs 27 12% General Funding Needs 19
Expansion 3
Incentives for Clients/Families 3
Donations 2
Training Needs 27 12% No Time or Money for Training 12
About the Program Specifically 10
To Facilitate Cooperation 5
FTC Procedures 22 9% Established Policy 11
Organization 6
Other 5
Gaps continued…
32
THEME ResponsesPercent of Total
(n=231)
Technology 18 8%
MIS 12
Computers 6
Administration 12 5%
Support to the Team 6
General 6
Time 8 3%
Better Use of Time 3
Time Constraints 5
No Response/Don't Know 11 5%
Gaps continued…
33
THEME ResponsesPercent of Total
n=137
KCFTC Model/Program Strengths 38 28% Model Itself Facilitates Success 20 15%
Strength-Based Approach and Vision 9
General 6
Wraparound 5
Day Care 2
Scope and Intensity of the Program 10
Accountability for Everyone 8 6%
Low Case Loads 5
General 3
Team Members 31 23% Investment/Involvement/Commitment/Dedication 18
Quality of Providers and Treatments 13
General Strengths of the KCFTC
34
THEME ResponsesPercent of Total
n=137
Court 23 17% Frequency of Hearings/Proceedings 10
Court Atmosphere - Positive 7
Accessibility 5
Presence of Team Member 1
Collaboration 18 13% Between Team Members 12
In Decision-Making 4
Team Members and Clients 2
Participation 17 12% General Participation of Families in FTC 12
Establishes Relationship 4
Promotes Reunification 1
Treatment Providers 10 7% Treatment Providers in General 5
Individualized/Tailored Treatment 5
Strengths continued…
35
Greatest Challenges or Greatest Challenges or Weaknesses of the Weaknesses of the
KCFTCKCFTCTHEME Responses
Percent of Total (n=112)
Team Members Issues 41 37%
Team Concept 14 13%
Team Roles 9
Team Stability 3
Team Member Involvement 2
Cohesiveness 11
Communication 10 9%
Between Team Members 9
With Treatment Providers 1
Organization/Management 2
Education and Training 2
Outcomes 2
36
THEME ResponsesPercent of Total
(n=112)
Treatment 24 21% Choice of Providers 10 9%
General Choice 8
Individualized 2
Quality of Treatment 5
Inpatient Treatment 3
Unclassified Remarks 2
Instability of Current Treatment 1
Equality for All Clients 1
Timeliness of Service Not Good 1
Approach/Vision of Program 17 15% Clarity of Vision/Mission of Program 7
Focus Needs to be on Child 6
Strength-Based 4
Challenges continued…
37
THEME ResponsesPercent of Total
(n=112)
Client's/Families 12 11%
Selection/Referral Process 5
Level of Engagement 4
Educating Clients 3
Visitations 1
Court 9 8%
Rewards and Sanctions 6
Court in General Needs Work 3
Accountability 5 5%
Financial Considerations 4 4%
Challenges continued…
38
Additional Results:Additional Results:Ratings of Ratings of
Individual Team Individual Team Members’ Members’
EffectivenessEffectivenessEffectiveness at being a non-Effectiveness at being a non-adversarial team memberadversarial team member
Overall effectiveness at achieving Overall effectiveness at achieving positive outcomes for enrolled positive outcomes for enrolled familiesfamilies
39
JudgeJudgeNon-Adversarial Team Member Overall Non-Adversarial Team Member Overall
Effectiveness (n=19)Effectiveness (n=19)
0%
10%
20%
30%
40%
50%
60%
70%
All Staff 0% 0% 5% 37% 58%
No at all successful
A little bit successful
Somewhat successful
Moderately successful
Extremely successful
0%
10%
20%
30%
40%
50%
60%
70%
All Staff 0% 0% 21% 58% 21%
Not at all successful
A little bit successful
Somewhat successful
Moderately successful
Extremely successful
40
DSHS Social WorkerDSHS Social WorkerNon-Adversarial Team Member Overall Non-Adversarial Team Member Overall
Effectiveness (n=19)Effectiveness (n=19)
0%
10%
20%
30%
40%
50%
60%
70%
All Staff 0% 5% 0% 37% 58%
No at all successful
A little bit successful
Somewhat successful
Moderately successful
Extremely successful
0%5%
10%15%20%25%30%35%40%45%50%
All Staff 0% 0% 11% 47% 42%
Not at all successful
A little bit successful
Somewhat successful
Moderately successful
Extremely successful
41
Team members ratedTeam members rated
JudgeJudge DSHS Social DSHS Social
WorkersWorkers Social Work Social Work
SupervisorSupervisor CASACASA CASA CASA
Supervisor/MgrSupervisor/Mgr KCFTC Program MgrKCFTC Program Mgr KCFTC SpecialistKCFTC Specialist
PCAP PCAP RepresentativesRepresentatives
Treatment providerTreatment provider Tx provider Tx provider
supervisorsupervisor Child’s attorneyChild’s attorney Parents’ attorneyParents’ attorney Wraparound Wraparound
facilitatorfacilitator
42
SUMMARY OF RESULTSSUMMARY OF RESULTS 88% believe the KCFTC is at least “a little 88% believe the KCFTC is at least “a little
bit more successful” than the regular bit more successful” than the regular dependency court at achieving outcomesdependency court at achieving outcomes Overall, 94% of respondents believe the KCFTC Overall, 94% of respondents believe the KCFTC
has been at least “somewhat successful” overallhas been at least “somewhat successful” overall Advisors and those with less overall contact with Advisors and those with less overall contact with
KCFTC participants gave higher ratingsKCFTC participants gave higher ratings Respondents less confident that goal of Respondents less confident that goal of
serving a representative population is being serving a representative population is being metmet
Respondents overwhelmingly believe Respondents overwhelmingly believe current client load is about right for capacitycurrent client load is about right for capacity
Mixed opinions on the amount of shared Mixed opinions on the amount of shared vision among team membersvision among team members
43
Relative strengthsRelative strengths
Processes, functions, and best practicesProcesses, functions, and best practices VisitationVisitation Random UA screens and AOD testingRandom UA screens and AOD testing Judicial interactionJudicial interaction Communication between court and providersCommunication between court and providers
Outcomes (compared to regular Outcomes (compared to regular dependency court)dependency court) Parents’ ability to remain soberParents’ ability to remain sober Reduction of negative effects on childReduction of negative effects on child Parents’ compliance with court ordersParents’ compliance with court orders Enrollment in appropriate CD servicesEnrollment in appropriate CD services
44
Relative weaknessesRelative weaknesses Processes, functions, and best practicesProcesses, functions, and best practices
Strengths-based assessmentsStrengths-based assessments Timely and effective MH and other services for Timely and effective MH and other services for
parentsparents Consistent and timely sanctions and incentivesConsistent and timely sanctions and incentives Staff training and educationStaff training and education MISMIS CD Treatment accountabilityCD Treatment accountability
Outcomes (compared to regular dependency Outcomes (compared to regular dependency court)court) Compliance with and completion of treatmentCompliance with and completion of treatment Full engagement in servicesFull engagement in services Earlier determination of alternative placement optionsEarlier determination of alternative placement options
45
Areas that need Areas that need improvementimprovement
(From open-ended questions)(From open-ended questions)• Resources and Services for clients
• Housing• Visitations• Transportation• Day Care
• Treatment services• Greater individualization and choice for services
• Improved Communication and Collaboration• Team Building• Staffings
• Increased Funding and Sustainability• Training Needs – Especially to ensure understanding of the model
and to facilitate shared vision
• More fully established policies and procedures• MIS and data
46
Next stepsNext steps
Complete surveying and analysisComplete surveying and analysis Complete analysis of Open-ended Complete analysis of Open-ended
questionsquestions Develop reports and presentations Develop reports and presentations
based on needsbased on needs Move ahead with Participant-level Move ahead with Participant-level
component of evaluationcomponent of evaluation