1 Overview of Conducting an FDC Evaluation Grantee Kick-Off Meeting January 12 2011 This project is supported by Award No. 2009-DC-BX-K069 awarded by the Office of Juvenile Justice and Delinquency Prevention, Office of Justice Programs January 12, 2011 1 Agenda • National Perspective and Lessons Learned • National Prevalence Data • Sharing and Accessing the Data in FDC • Key Data Elements for FDC Evaluation • FDC Case Study: Sacramento County, California • Resources and Next Steps • Questions and Discussion 2 Introductions • Nancy K. Young, Ph.D. Executive Director, Children and Family Futures • Sharon Boles, Ph.D. Research and Evaluation Director, Children and F il Ft Family Futures 3 National Perspective and Lessons Learned 4
Transcript
1
Overview of Conducting an
FDC Evaluation
Grantee Kick-Off MeetingJanuary 12 2011
This project is supported by Award No. 2009-DC-BX-K069 awarded by the Office of Juvenile Justice and Delinquency
Prevention, Office of Justice Programs
January 12, 2011
1
Agenda
• National Perspective and Lessons Learned
• National Prevalence Data
• Sharing and Accessing the Data in FDC
• Key Data Elements for FDC Evaluationy
• FDC Case Study: Sacramento County, California
• Resources and Next Steps
• Questions and Discussion
2
Introductions
• Nancy K. Young, Ph.D.
Executive Director, Children and Family Futures
• Sharon Boles, Ph.D.
Research and Evaluation Director, Children and F il F tFamily Futures
3
National Perspective and Lessons Learned
4
2
Why Evaluation is Needed
• How do we know FDCs are effective and if they work? Measured by what goals?
• How can the efforts and resources needed to operate the FDC be sustained?operate the FDC be sustained?
- Does the FDC save money?
- Is there an ability to conduct a cost analysis?
• If the FDC is effective, should it be expanded?
- What is the scale of the FDC?
5
FDC Evaluation Lessons
What do you want to know? Formulate the three most important questions the evaluation is intended to answer—and who wants to know. Then be sure the information system is collecting the data needed to answer the questions.
You need the wheels and engine!The outcomes are the wheels, but the cost savings are the engine.
Compared to what?• It’s important to have a comparison group
• Know your baselines
• Compare apples to apples
6
FDC Evaluation Lessons
Client outcomes matter, BUT so do system changes!
Look for institutional changes that will last beyond the project
Consider definition of success! C f “ fA recent FDC evaluation defined success as “removal of
children without termination of parental rights, enabling birth parents to have contact with children.” Success—but not the same as permanency for the child through reunification
The end matters!• Court, child welfare, and treatment outcomes are different
• Know what audiences care most about
7
FDC Evaluation Lessons
What about the children?Be sure adequate data is collected on child outcomes, because that is a critical difference between adult and family drug courts and most likely to generate political and community support
What is missing?Missing data is sometimes more important than the available data (e.g. prevalence rates for substance abuse among child welfare clients)
Where did they go?Know where you are losing clients: a drop-off analysis enables tracking clients from identification of the problem to referral to enrollment to positive treatment outcomes
8
3
National Prevalence Data
9
Treatment Admissions by Primary Substance & Child Maltreatment
for Sample State
sions /Reports
50000
60000
70000
80000
Total Tx Admissions
10
Office of Applied Studies, SAMHSA Quick Statistics from the Drug and Alcohol Services Information System, Accessed 4.20.10U.S. Department of Health and Human Services, Administration on Children, Youth and Families. Child Maltreatment 2008. Washington, DC: U.S. Government Printing Office, 2009.* Indicates CFSR Round 1 On-Site Review** Indicates CFSR Round Two On-Site Review
1) U.S. Department of Health and Human Services, Administration on Children, Youth and Families. Child Maltreatment 2008(Washington, DC: U.S. Government Printing Office, 2009). Tables 3.1, 3.32) Substance Abuse and Mental Health Services Administration, Office of Applied Studies. Treatment Episode Data Set (TEDS) Highlights - - 2009 National Admissions to Substance Abuse Treatment Services. Tables 2a, 6a. OAS Series #S-40, DHHS Publication No. (SMA) 08-4313, Rockville, MD, 2009.
*Includes those with disposition of substantiated, indicated or alternative response victim. Percentage is number of maltreatment cases out of total number of children who received a CPS investigation.
11
Child Maltreatment By Age Group, 2006
United States Sample State
Age Group of Victims Number Percent Number Percent
Age 0-3 273,082 30.9% 3,358 28.8%
Age 4-7 213,194 24.1% 2,823 24.2%
Age 8-11 170,944 19.3% 2,333 20.0%
Age 12-15 170,635 19.3% 2,295 19.7%
Age 16-17 54,029 6.1% 842 7.3%
Unknown 3,110 .3%
Total 884,994 11,651
Children’s Bureau. Administration of Children and Families (2006) Child Maltreatment12
4
Sharing and Accessing Data in FDC
13
Building Cross-System Collaboration: Developing the Structure to Create and
Resource: Introduction to Cross-System Data Sources in Child Welfare, Alcohol and Other Drug Services, and Courts (aka Data Primer)
Connecting Data Systems
ConfidentialityRelease of Information
CWS
18
Memorandum of Understanding
Court AOD
Accessing Data:Key Considerations
Memorandum of Understanding (MOU)• Specify the types of information that may be shared
• And the process to be used to ensure that current confidentiality requirements are met
CWS
19
y q
• Remember that 42 CFR includes an evaluation exemption
COURT AOD Release Forms• Mutually agreed release forms can simplify exchange of information
• FDC Team should perfect the timely execution, utilization, and incorporation of releases in FDC contracts and consents
Confidentiality
• Each agency operates within strict federal, state and jurisdictional guidelines
• Policies that allow agencies for information sharing
• Strategy: consent form signed by the
AOD
20
• Strategy: consent form signed by the parent that allows specific, limited information to be shared with designated entities
• Jurisdictions can develop a mutually agreed upon Release of Information and Informed Consent Form that will comply with Federal and State confidentiality laws
• Federally-approved consent form
COURTCWS
6
Joint Responsibility
• Each system shares the responsibility
• Each system has a unique role
• Each system establish joint
CWS
• Each system establish joint agreements
21
COURT AOD
Child Welfare Services Checklist
Collects information regarding client progress in treatment
Collects or receives judicial orders regarding the case
Shares data regarding Shares data regarding placement, case management activities, case outcomes; costs of foster care and other services
Uses data for case planning, case management, case tracking, policy decisions
22
CWS
Drug Treatment Checklist
Collects information about children of parents in treatment
Shares progress, treatment p g ,episodes, completion
Uses information about the child to address issues related to abuse or neglect in treatment
Uses data for case planning, case tracking, policy decisions
23
AOD
Court Checklist
Collects information regarding client progress in treatment and services
Collects information regarding CWS case management
Court
activities
Shares data regarding status of case, court dates, timelines; judicial orders; cost of processing cases
Uses data for resource management, case processing, funding decisions
24
Court
7
Cross-Systems Checklist
Establish agreed upon data components and procedures for tracking clients and service impact
Establish common outcome measures
Formulate and utilize formal
Court
Formulate and utilize formal communication tools to facilitate info gathering, facilitate reviews, and reduce duplicative reporting (Example: standardized report forms)
Monitor the outcomes of information sharing
Uses data reports to modify policies and protocols
25
CWS AOD
Key Data Elements for FDC Evaluation
26
Key FDC Outcomes
Safety (CWS)
• Re-entry into f t
Permanency (Court)
• Time to ifi ti
Recovery (AOD)
• Engagement and retention in
foster care• Recurrence of
abuse/neglect
reunification• Time to
permanency• Days in care
treatment• Number of
negative UA’s• Number of
graduates• Reduction in use• Employment• Criminal behavior
27
Savings
Foster Care• Shorter Stays in Foster Care
• Less Re-entries to Foster Care
• Lower Levels of Care
• Less recurrence of maltreatment
Treatment• Lower Levels of Care
• Successful completion
28
8
Key Data Elements for FDC Evaluation
• Parent and Child Characteristics
• Child Welfare Data
• Substance Abuse Treatment DataSubstance Abuse Treatment Data
Percent of Children Living with a Parent at 36 Months
50 9 53.560
80
100
nt
42.4 41.5 44.350.9
0
20
40
Year One Year Two Year Three Year Four Year Five
Pe
rce
Comp n=173; DDC n=2477 Source: CWS/CMS41
Time to Reunification Among Children Reunifying by 36 Months
20
24
28
32
36
ths
10.4 9.8
0
4
8
12
16
20
Comparison DDC
Mo
nt
Not SignificantComp n=173; DDC n=2814 Source: CWS/CMS42
DDC Graduation Criteria
For 180 consecutive days, parent must:
• Produce negative drug tests
• Attend all required group and individual treatment
sessionssessions
• Attend all scheduled Recovery Specialist (STARS)
meetings
• Attend at least 3 support / 12-step meetings weekly
• Attend all required DDC appearances
• Complete all requirements of the court43
Parents DDC Graduation Status
60
80
100
ent
33.6
24.7
41.8
0
20
40
Graduated (n=964) 90 Day Certificate (n=709)
Neither Landmark (n=1200)
Per
ce
Source: STARS44
12
70.3
60
80
100
nt
Reasons for Not Completing DDC (n=868)
3.15.3 4.7
8.4
2.9 0.94.4
0
20
40
Failure to Appear (n=610)
Mental Health (n=27)
Incarcerated (n=46)
Out of County (n=41)
Services Terminated
(n=73)
Prop 36 (n=25)
Adult Drug Court (n=8)
Timed Out of Level 2
(n=38)
Percen
45
62.2
74.3
52 560
80
100
nt
Child Reunification Rates by DDC Graduation Status Over Time
19.1
26.0
45.6
52.5
1925.1
0
20
40
12 Months 36 Months
Per
cen
Comparison Graduated 90 Day Certificate Neither Landmark
46
40
60
rcent
Re-Entry to Foster Care Rates
15.012.1
19.7
5.3
14.7
0
20
Federal (median)
California Sacramento Comparison DDC
Pe
Comp n=173; DDC n=2086 Source: CFSR, CWS/CMS47
Recurrence of MaltreatmentRates Within 6 Months
40
60
ent
5.48.2 9.4
5.2 3.6
0
20
Federal California Sacramento Comparison DDC
Perce
Comp n=173; DDC n=2814 Source: CFSR, CWS/CMS48
13
24 Month Cost Savings Due To Increased Reunification
What would have happened regarding out of home care costs in the absence of DDC?
27.2% - Reunification rate for comparison children49.6% - Reunification rate for DDC children= 766 fewer DDC children would have reunified
33.1 - Average months in out-of-home care for comparison children8.98 - Average months to reunification for DDC children= 24 months that DDC kids would have spent in out of home care (OHC)
$1,828.92 – Out of home care cost per month766 x 24.12 x 1,828.92=$33,790,979 Total Savings in OHC Costs
49
Questions for the Grantees
• What would you want to know about your FDC?
• What would you want others to know about your FDC?
• What do you think makes a good evaluation?• What do you think makes a good evaluation?
• What are the challenges or barriers to evaluation?
• What would you use results of the evaluation for?
50
Next Steps & Resources
51
Next Steps
• Take an inventory – what exists
• Identify what/who is missing
• Develop or enhance what is
52
• Develop or enhance what is needed
• Prioritize and implement strategically
14
FDC Technical Assistance Products for Evaluation
• Sacramento County Dependency Drug Court Year Seven Outcome and Process Evaluation Findings
• Sacramento Dependency Drug Court: Development and Outcomes• Family Drug Treatment Courts: Process Documentation and
Retrospective Outcome Evaluation• Family Treatment Drug Court Evaluation: Final Report by NPC
53
• Family Treatment Drug Court Evaluation: Final Report by NPC• Jackson County Community Court Process, Outcome, and Cost
Evaluation Final Report• Running the Marathon: Sacramento County’s Ten-Year Journey
Towards Enhanced Services to Parents with Substance Abuse Disorders in the Child Welfare System
Visit Children and Family Futures – www.cffutures.org
