Washington State IV-E Waiver Demonstration Project
Family Assessment Response Interim Evaluation Report
Submitted:
December 2016
4450 Arapahoe Avenue, Suite 100, Boulder, CO 80303
Family Assessment Response: Interim Evaluation Report 1
Introduction and Background Thisreportdescribesthefirsttwoyears(January2014throughDecember2015)ofWashingtonState’simplementationoftheFamilyAssessmentResponse(FAR)asaTitleIV-EWaiverdemonstrationproject.Inaddition,preliminaryoutcomeandcostdataarepresentedanddiscussed.WashingtonState’sTitleIV-EWaiverDemonstrationProjectfocusesontheimplementationofFamilyAssessmentResponse(FAR),adifferentialresponsepathwayforscreened-inallegationsofabuseandneglectasanalternativetotraditionalChildProtectiveServices(CPS)investigations.TheoriginalFARframeworkoutlinedspecificstepstobetakenbytheDepartmentofSocialandHealthServices(DSHS)tofocuschildwelfareresourcesonfourareastoimproveoutcomesforsafety,permanency,andwell-being:
1. Increasedconnectionswithextendedfamily,naturalsupports,andcommunitytoenhancechildsafetybyengagingfamiliesoutsideofthetraditionalinvestigativeprocess.Byofferingservicesandsupportwithoutaformal“finding”regardingchildabuseorneglect,thestatehopesfamilieswillbemoreopentoacceptingservices.
2. Provisionofconcretegoodsandservicestosupportfamilies,safelypreventplacementinout-of-homecare,safelyreunifychildrenwiththeirfamilies,andimprovechildandfamilywell-being.
3. Expandeduseofevidence-basedpracticestoprovidetargetedinterventionsthateffectivelyaddresstheneedsofchildrenandtheirfamilies,improvechildsafetyinthehome,preventout-of-homeplacement,andincreasechildandfamilywell-being.
4. ExpansionofWashingtonState’spracticemodels,specifically,SolutionBasedCasework1andtheSafetyFramework.
TargetPopulation:FARfocusesonchildrenandtheirfamilieswhoarereported(screenedin)toCPSforneglectandlow-to-moderatephysicalabusewithanon-emergent,72-hourresponsetime.TheFARimplementationandevaluationhasbenefitedfromthedevelopmentandimplementationoftwodistinctStructuredDecisionMaking(SDM)tools:anIntakeToolandaRiskAssessmentTool.Intake&RiskAssessmentTool:TheWashingtonStateChildren’sAdministration(CA)workedwiththeChildren’sResearchCenter(CRC)todevelopanSDMIntakeTooltodeterminewhichfamiliesareeligibleforFAR.Thistoolguidesintakeworkersthroughaseriesofquestionsaimedtodeterminewhetherthereisanallegationofchildabuseorneglectasdefinedinstatestatute.OnceacasescreensinforaCPSresponse,theSDMIntakeToolhelpsintakestaffdetermine
1Children’sAdministrationmadechangestopracticemodelsduringtheFARimplementation.Thisisdiscussedintheimplementationsectionofthisreport.
Family Assessment Response: Interim Evaluation Report 2
whetheraninvestigativeorFARresponseisappropriateforthefamily.AnexistingSDMOverallRiskAssessmentToolhasalsobeenutilizedintheFARandinvestigativepathwaystohelpdeterminefamilyriskfactorsandneedsforservices.InOctoberof2013,theCAtrainedintakestaffintheimplementationoftheFARpathway.TheSDMIntakeToolwasfullyimplementedacrossthestateatthattime.ThismeansthatFAReligibilitywasdeterminedforallscreened-inintakesregardlessofwhetheranofficehadbegunFARimplementation.2ThisstatewideintakecreatedtheopportunitytocarefullymatchcomparisongroupsfortheFARevaluation.TheFARpathwayisoptional.Familieschoosetoparticipate,and,unlikemanyotherstatesimplementinganalternativeresponse,participantsmustsignanagreementofparticipation(thisagreementisalsosignedbythecaseworker).TheFARagreementispartoftheenablinglegislationfortheprogram’simplementation.FamilieswhodeclinetoparticipateinFARaretypicallytransferredtotheinvestigativepathway.3Implementationofalternativeresponse(AR)modelsinotherstatesinformedthedevelopmentoftheWashingtonFARmodel.ToprovidecontextforevaluationfindingsregardingtheimplementationandpreliminaryoutcomesofFAR,weattheTriWestGroup(TriWest)reviewedevaluationsofdifferentialresponseeffortsinsixotherstates:Colorado,Illinois,Minnesota,Missouri,Nevada,andNewYork.Thesestateswerechosenfortheirrespectiveprograms’similaritiestotheWashingtonFARmodelandfortheavailabilityofsimilarprocessandoutcomemeasures.WeusedfindingsfromtheseprogramstoinformourevaluationworkandtodiscussfindingswithWashingtonFARstakeholders.Ourreviewrelieddirectlyonformalevaluationsofalternativeresponse(AR)demonstrations(sometimesalsocalled“differentialresponse”).Whilemanystateshaveimplemented—orareintheprocessofimplementing—ARdemonstrationsforchildabuseandneglectcases,evaluationresultswerenotavailableforallstates,eitherbecausesomestateshavenotcompletedformalevaluationscontainingdetailedoutcomeanalysisorbecausewewereunabletoobtainpublishedevaluationresults.Thus,thereviewwasnotintendedtobeacompleteinventoryofoutcomeresultsfromallARdemonstrationsintheUnitedStates.Additionally,whileotherorganizations(suchasCaseyFamilyProgramsandtheQualityImprovementCenter
2ThephasedrolloutofFARinofficesacrossthestateisdiscussedlaterinthisreport.3Insomecases,familiesparticipateintheassessmentprocessundertheFARpathwaybutfailtosigntheFARagreement.Ifthecaseworkerbelievesnofurtherservicesoractionsarenecessary,thecasemaybeclosedwithoutbeingtransferredtotheinvestigativepathway.
Family Assessment Response: Interim Evaluation Report 3
onDifferentialResponse)offerabbreviatedoutcomesummariesofselectedARprograms,wechosetorelyontheoriginalevaluationdocumentsforthepurposesofthisreport.Researchfocusedonaspectsofprogramstructure(includingscope,jurisdiction,intakes,programeligibility,andthestructureoftheintervention),theevaluation(includingsamplingmethodologyandevaluationdesign),anddemonstrationoutcomes(includingre-referralrates,removalrates,caseloadandcaselengthdata,serviceprovision,andcostsofthedemonstration.)Thisreportmostlyomitsqualitativefindingssuchassurveyandinterviewresultsfromfamily,caseworker,administrator,andcommunitymembers,thoughtheresourcescitedinthereportoftencontainadditionaldataconcerningtopicssuchaschangeincaseworkerattitudes,familyengagement,andfamilysatisfactionwithAR.StateswithoutcomespresentedinthisreportincludeColorado,Illinois,Minnesota,Missouri,Nevada,NewYork,NorthCarolina,andOhio.AdditionaleffortsweremadetofindprimarysourcesforprogramsinArizona,Connecticut,Florida,Hawaii,Iowa,Kentucky,Louisiana,Maine,Maryland,Oklahoma,Tennessee,Texas,Vermont,andWyoming.Evaluationsorotherlessformalprimarysourcesofprogramdataforthislattergroupofstatesoftendidnotcontainenoughdetaileddataonprogramoutcomes(e.g.,removalandre-referralrates)towarrantinclusionhere.Additionally,evaluationsforsomeofthesestatesarecurrentlystillinprogress.OverallfindingsfromtheseevaluationswereconsistentwiththeexperiencesofWashingtonState.Findingsrelatedtoparticularoutcomequestionsarecitedineachrelevantsection.StagedRollOutofFARinWashingtonState
TheimplementationofFARinWashingtonStatewasplannedtooccurinmultiplephases.This“phased”approachbecameacentralfeatureoftheFARevaluation.BecauseonlysomeofficesimplementedFARatspecifictimes,familiesreceivingCPSservicesinnon-FARofficesservedasasourceforacomparisongroup.Additionally,thephasedimplementationallowedtheCAtoassesssuccessesandchallengestoimplementationinofficesfromearlyphasesandtomakemid-coursecorrectionstoensurebetterimplementationinlaterphases.Initially,FARwasimplementedinthree“pilot”sites(pleaseseemaponthefollowingpage)inJanuaryof2014.Thesethreesiteswereselectedbasedontheirgeographicallocationsandtheirreadinesstoimplementthenewpathway.ThemaponthefollowingpageshowsthelocationofofficesinwhichFARwasimplemented(markedwithastar)andindicatesthedegreetowhichFARwasavailableinthecounty.CountieswithfullFARavailabilityareindicatedindarkgreen,whilecountieswithsomeFARimplementation(butwheretheentirecountywasnotcovered)areshowninlightgreen.GrayshadingindicatesthatFARwasnotavailableatthetimeofthatspecificrolloutphase.
Family Assessment Response: Interim Evaluation Report 4
Followingthesix-monthpilotsiteimplementation,theCAbeganaddingFARintonewofficeseachquarter.TheofficesidentifiedinthemapbelowbeganimplementingtheFARpathwayinJulyof2014.
Family Assessment Response: Interim Evaluation Report 5
InOctoberof2014,anadditionalfiveofficeswereaddedacrossthestate(pleaseseemapbelow).
Thefinalmap(below)showstheextentofstatewideFARimplementationattheendofthesecondprojectyear(2015).AllremainingofficeswillbeimplementingFARbymid-2017.
EvaluationMethods
ThecomprehensiveevaluationoftheTitleIV-EWaiverProjectincludesanexaminationofprojectprocesses,outcomes,andcostsintheimplementationoftheFARmodel.Themodelisbeingimplementedonarollingbasis,allowingformatchingbetweenlocalofficesimplementingthewaivertonon-FARofficesscheduledtorolloutinlaterquarters.Inadditiontomatchesat
Family Assessment Response: Interim Evaluation Report 6
thelocalofficelevel,wealsomatchedindividualsparticipatinginFARtothosewhowereservedviatraditionalservicesinnon-FARoffices.Specificresearchquestionsaddressedbytheprocessandoutcomeevaluation,aswellthecostanalysis,aredetailedintheappropriatesectionsbelow.Theevaluationisdesignedtoanswerthefollowingquestions:
• HowwastheFARmodelimplemented(descriptive)?• WasthestateabletousethewaivertoimplementFARwithfidelity?• Whatwerethebiggestchallengestoimplementation?• HowdidimplementationchangechildwelfarepracticeinthestateofWashington?• DidtheFARimplementationresultingreaterorlesserdisproportionalityinservices
offeredtofamilies?• DidtheFARimplementationreducechildmaltreatmentinparticipatingfamilies?• DidtheFARimplementationreduceout-of-homeplacement?• DidtheFARimplementationresultinimprovedchildandfamilyfunctioning?• WastheimplementationofFARunderthewaivercost-neutral?
Thetablebelowoutlinesthedatasourcesutilizedforthisevaluation.DataCollectionTool Population ProgramPurpose
FAMLINK
Washington’sSACWISsystem. Alladministrativedata,includingintakesintoFARorInvestigations.
SDMIntakeTool(administeredbyintake)
AllreferralstotheChildren’sAdministration
DetermineeligibilityforFARpathway.
SDMRiskAssessmentTool(afterintakeandFAReligibilitydetermination)(administeredbyallCPScaseworkers)
FARpathwayfamiliesInvestigativepathwayfamilies
Assessfamilyriskfactorsandneedforservices.
FamilySurvey(administeredbyParentAllies)
FARpathwayfamilies
Assessfamilyperspectivearoundkeyprocessandoutcomevariables.
Sitevisitsandkeyinformantinterviews
Caseworkers(FARandinvestigative),supervisorsandadministratorsinallFARimplementingoffices
Collectdataregardingprogramimplementationandfidelity.
Washington’sStateAutomatedChildWelfareInformationSystem(SCWIS)isFAMLINK.TheFAMLINKdatasystemextractsprovideinformationonallreferralstoCPSinthestate.The
Family Assessment Response: Interim Evaluation Report 7
systemwasusedtoidentifyunduplicated4familieswithanintakeduringthestudyperiod(n=91,433).IntakedatainFAMLINKarethenusedtoseparatefamiliesintostudycohorts(treatment,control,excluded)basedonwhether1)theintakeisscreened-inandnota“risk-only”case,5and2)whethertheintakeisFAR-eligible.Thediagramonpage10ofthisreportshowstheflowofthoseintakesintospecifictreatmentandcontrolgroups.InadditiontoadministrativedatafromFAMLINK,TriWestcollectedFARimplementationdatathroughsitevisitsandKeyInformantInterviews,withallcaseworkers(bothFARandInvestigativeworkers),supervisors,andadministrators.Thevisitsandinterviewswereconducted3–4monthsfollowingtheimplementationofFARintherespectiveoffice.Duringthefirsttwoyearsofimplementation,TriWestconducted29sitevisitsand399KeyInformantInterviews.Dataarealsocollectedfromparents/guardianswhoparticipateinFARthroughaFamilySurvey.Atcaseclosure,parents/guardiansaresentacaseclosureletterthatremindsthemthatanevaluationteammembermaycontactthemtocompleteatelephonesurvey.Theletteralsoprovidesinformationforcompletingaweb-basedorautomatedtelephonesurveyifthatmethodispreferred.Eachmonth,CAcompilesalistofallclosuresandsendsTriWestrecentphonenumbersofFARparticipantswhoindicated,intheFARagreement,thattheywerewillingtobecontactedregardingthesurvey.Parentalliescalltheseparents/guardianstoconductthefulltelephonesurveys.FARparentsorguardianswhoparticipateinthefulllivetelephonesurveyareoffereda$10Walmartgiftcardasatokenofappreciation.Thosecompletingtheshorterweb-basedortelephonesurveysareoffereda$5giftcard.Atotalof240surveyswerecompletedduringthefirsttwoyearsofFARimplementation.AfulldescriptionofsurveyresponseratescanbefoundintheDecember2016FamilySurveySummaryreport.FurtherinformationregardingevaluationdatacollectionisprovidedintheFAREvaluationPlanandinthetechnicalappendixtothisdocument(AppendixA).Theevaluationutilizesanintenttotreat(eligibility)design,whichmeansthat,inofficesthathaveimplementedFAR,allfamiliesthatareassignedtotheFARpathwaybytheSDMIntake
4Thestudyidentifiedfamiliesbyfirstintakewithinaspecificstudyperiod(cohort).Whilethecountofintakesisunduplicatedforeachcohort,onefamilymaybecountedagaininasubsequentcohort.5Risk-onlycasesarethosecasesinwhichachildisatimminentriskofharmbutthereisnotChildAbuseorNeglect(CA/N)tobeinvestigated.ThesecaseswouldnotbeassignedtoaCPSInvestigationand,therefore,arenoteligibleforthealternativeFARresponse.Seehttps://www.dshs.wa.gov/ca/2000-child-protective-services/2200-intake-process-and-responseforafulllistofdefinitions.
Family Assessment Response: Interim Evaluation Report 8
Tool,excludingsupervisoroverrides,areincludedintheFARtreatmentgroup.Iffamiliesdeclinetoparticipateorarelatertransferredtotheinvestigativepathwayduetosafetyconcerns,theyarestillincludedinthetreatmentgroup.Becauseofthephasedimplementationandthestatewideuseoftheintaketool,apoolofFAR-eligiblefamiliesbeingservedinofficesthatarenotyetimplementingFARareavailableforinclusioninamatchedcomparisongroup.Utilizingpropensityscorematching,TriWestcreatedacomparisongroupoffamiliesmatchedtoFARfamilieson26demographic,CPS,andriskassessmentvariables.
Thediagrambelowshowstheevaluationdesign.
Offices&with&Family&Assessment&
Response&(FAR)&
Offices&with&Inves9ga9ve&Pathway&Only&
Intent&to&Treat&(Eligible&for&FAR)&
Not&Eligible&for&FAR&
Assessment&Intent&to&Treat&B&Received'FAR'
Intent&to&Treat&B&Opted&Out&of&FAR&
Inves9ga9ve&BMatched&
Comparison&
Inves9ga9ve&B&&&&&Not&Eligible&
Eligibility&Design&
Assessment&Would&have&been&
FAR&eligible&
Would¬&have&been&FAR&eligible&
Family Assessment Response: Interim Evaluation Report 9
FAR(treatment)familiesaregroupedintosix-monthstudycohortsbasedonthedateoftheirfirstFAR-eligibleintakeduringtheperiod.6EachcohortincludesfamiliesservedinalloftheofficesimplementingFARduringtheperiod.Forexample,thefirstcohortincludesallfamiliesservedinthefirstsixmonthsoftheproject(January1,2014throughJune30,2014),whichonlyincludesthefirstthreepilotsites.However,thenextevaluationcohortincludesthefirstthreepilotsites,aswellasthenexttwophasesofoffices(rolledoutJuly2014throughDecember2014).
StudyCohortNumberof
FamilieswithaFARIntake
NumberofSampled7FARGroupFamilies
NumberofMatchedComparisonGroup
Families
Cohort1(Jan–Jun2014)Phase1Offices(pilot)
664 664 664
Cohort2(Jul–Dec2014)Phase1-3Offices
2,630 2,630 2,630
Cohort3(Jan–Jun2015)Phase1-5Offices
5,593 2,000 2,000
Cohort4(Jul–Dec2015)Phase1-5Offices8
5,432 1,000 1,000
Thediagramonthefollowingpageshowstheflowfromintaketoinclusionintoeachofthestudygroups.
6Familieswereonlyincluded/countedoncepercohort,throughaspecificfamilycouldbeincludedinmultiplecohortsduetonewintakes.7BeginningwithCohort3,arandomsampleofFARfamilieswasusedforcomparativeanalysis.AsmoreofficesimplementedFAR,thecomparisonpooloffamiliesinnon-FARofficesbecametoosmalltodrawacomparisongroupthesamesizeasthefullFARgroup.8DuetoadelayinFARimplementation,noadditionalofficesbeganFARimplementationduringtheCohort5studyperiod.
Family Assessment Response: Interim Evaluation Report 10
Far Case Disposition(of 8,897)
Cohort 1 Cohort 2 Cohort 3 Cohort 4 Total
0=Missing 0 0 8 80 881=Remained FAR 597 2,350 4,974 4,786 12,7072=Declined FAR (no investigation)
39 (5.9%)
171 (6.5%)
317 (5.7%)
288 (5.3%)
815
3=Transferred(including investigation)
27 (4.1%)
81 (3.1%)
126 (2.3%)
125 (2.3%)
359
Disposition 15(excluded from analysis)
1 28 168 153 350
Cases Screened Out42,103
(Intake type=0)Cohort 1: 12,470Cohort 2: 10,572Cohort 3: 10,413Cohort 4: 8,648
FAR Cases14,319
(Intake type=1)Cohort 1: 664Cohort 2: 2,630Cohort 3: 5,593Cohort 4: 5,432
Investigative Cases30,102
(Intake type=2)Cohort 1: 12,336Cohort 2: 7,700Cohort 3: 5,673Cohort 4: 4,393
Risk-Only Cases4,197
(Intake type=3)Cohort 1: 1,111Cohort 2: 1,031Cohort 3: 950Cohort 4: 1,105
Missing Values712
(Intake type=NA)Cohort 1: 6Cohort 2: 83Cohort 3: 295Cohort 4: 328
Total Intakes91,433
Cohort 1: 26,587Cohort 2: 22,016Cohort 3: 22,924Cohort 4: 19,906
Cohort Sample PeriodsCohort 1: Jan–Jun, 2014 Cohort 3: Jan–Jun, 2015Cohort 2: Jul–Dec, 2014 Cohort 4: Jul–Dec, 2015
Cases that Would Have Been Eligible for FAR
18,655(Potential Control Observations)Cohort 1: 8,515Cohort 2: 4,953Cohort 3: 3,192Cohort 4: 1,995
Cases Not Eligible for FAR Even If Available
9,065Cohort 1: 2,663Cohort 2: 2,002Cohort 3: 2,163Cohort 4: 2,237
Other Investigative Cases2,382
Cohort 1: 1,158Cohort 2: 745Cohort 3: 318Cohort 4: 161
Family Assessment Response: Interim Evaluation Report 11
Evaluation of Family Assessment Response (FAR) Implementation in Washington State
Asmentionedpreviously,thisreportaddressesFARimplementationandpreliminaryoutcomesforthefirsttwoprogramyears(January2014throughDecember2015).Duringthosetwoyears,TriWestvisitedeachofficeseveralmonthsafterFARimplementationtodiscusssuccesses,challenges,andstaffperceptionsofchangescausedbytheadditionofthenewCPSpathway.Keyinformantinterviews(KIIs)wereconductedwithcaseworkersfrombothFARandinvestigativepathways,9supervisors,administrators,andcommunityserviceproviders.Basedonfindingsfromthesesitevisits,aswellascaseconsultationsandmoreinformaldiscussionswithcaseworkersinthefield,theChildren’sAdministration(CA)madeseveralimportantprogramchangestotheFARimplementation.Thesechangesarediscussedattheendofthissection.CaseworkerReportsofPreparednessforFARImplementation(KII)
OnerecurringthemeininterviewswithbothFARandinvestigativecaseworkersisthatFARseemstobeabetterfitforsomecaseworkersthanothers.TheCAallowedvoluntarytransfersfrominvestigativecaseworktoFARcasework,thus,mostcaseworkersprovidingservicestofamiliesintheFARpathwayhadchosentobeincludedinthatprogram.Thisvoluntaryassignmentlikelybenefittedimplementationascaseworker“buy-in”totheFARmodelwasanimportantfeatureofsuccess.Overallratingsofpreparednessforimplementationwerefairlyhigh,fallingbetween“somewhatprepared”and“mostlyprepared”(or2.7onafour-pointscale).ThesescoreswerethesameforYearOneandYearTwoandwerevirtuallyidenticalforFARcaseworkersandinvestigativeworkers.
9Interviewswithinvestigativecaseworkerswereaddedaftersitevisitstoeachofthethreepilotsites.
2.7 2.72.5 2.6
1.0
2.0
3.0
4.0
FARYr1AverageRa4ng FARYr2AverageRa4ng InvYr1AverageRa4ng InvYr2AverageRa4ng
Ra#ngScale:4-VeryPrepared3-MostlyPrepared2-SomewhatPrepared1-NotPrepared
PreparednessforFARatImplementa#on
Family Assessment Response: Interim Evaluation Report 12
ImplementationSuccessesandChallenges
OfficestaffingpatternsatthetimeoftheFARrolloutseemedtomoststronglyinfluenceimplementation,withfullystaffedofficesreportingsmootherimplementation.Staffvacancies(duetovacations,leave,andordinaryturnover)thatoccurredatthetimeofimplementationcreatedachallengeforstaff.Additionally,trainingwasoriginallyratedsomewhatpoorlybycaseworkers.However,significantchangestothetrainingcurriculumandlanguageusedtodescribeFARweremadeoverthecourseofthefirstprojectyear.Caseworkers’perspectivesofFARtrainingimprovedinYearTwoofimplementation.TwofeaturesoftheFAR-enablinglegislationwerecitedasbarrierstoimplementingFARsuccessfully:therequirementthatfamiliessigntheFARagreementandthe45-daytimelimitformostFARcases.CaseworkersobservedthatsomefamiliesseemedparticularlyreluctanttosigntheFARagreement,eitherbecausetheydidnottrust“thestate”andwereworriedthatwereadmittingtowrongdoing,becauseofadviceofcounsel,orbecauseofanactivechildcustodycaseinwhichaformalfindingwasdesired.WhileitispossibleunderFARtoextendthetimeperiodupto90days,mostcaseworkerstriedtoworkwithintheinitial45-daytimelimit.Someseemedunawareofthepossibilityofextendingthecaseto90days.Caseworkersconsistentlyreportedthatthe45-daytimeperiodwastooshortformostservicesneededbyfamiliesand,inparticular,thatitlimitedtheirabilitytouseevidence-basedpractices(EBPs)becausebythetimeafamilywasreferredandbeganservices,therewasnotenoughtimetocompletetheservice.Asaresult,caseworkersreportedusingfewEBPswithfamilies.Somespecificprovidersdidattempttomodifyprogramstoaccommodateashortenedtimeframe,butthisdidnotsignificantlyresolvetheissue.Overall,caseworkersinYearOnereportedthatthebarriersdescribedabovecauseda“noticeablebarrier”toFARimplementation.However,astrainingforandcommunicationaboutFARimproved,thoseratingsimprovedsomewhatforFARworkers.Investigativeworkerstendedtoratebarriersaslower(“somewhat”comparedto“noticeable”).Theirperspectivesdidnotchangeacrossthetwoyears.
Family Assessment Response: Interim Evaluation Report 13
Despiteimplementationchallengesduringthefirsttwoprogramyears,mostrespondentsacrossofficesfeltthatFARhadledtoarelativelyhighdegreeofpositivechange.ThesechangesweretypicallyrelatedtotheexperiencesofFARfamiliesandFARcaseworkers’abilitytoprovidecommunityservicestomeetfamilies’needs.FARfamiliesweremuchmoreengagedwithsocialworkersoncetheyunderstoodthattheseworkerswerenotseekingafinding.FamiliesalsoappreciatedtheincreasedtransparencyandhonestyinherentintheFARmodel.FamilieswhohadpreviousexperienceswithCPSlikedtheFARpathwaymoreandfelttheyhadbetterexperiencesthroughFAR.RespondentsalsoreportedmorecommunitysupportandcommentedthatcommunitiesarebeginningtoseeCPSmorepositively.Caseworkers,onaverage,aremorefamiliarwithcommunityservicesandarebetterabletoworkwithfamiliestohelpthemmeettheirneedsafterFARimplementation.Asshowninthefigurebelow,bothFARandinvestigativeworkersreported,onaverage,“noticeable”positivechangesintheofficeasaresultofFARimplementation.ThesepositiveratingswerealittlelowerforinvestigativeworkersinYearTwo.
3.02.7
2.32.4
1.0
2.0
3.0
4.0
FARYr1AverageRa2ng FARYr2AverageRa2ng InvYr1AverageRa2ng InvYr2AverageRa2ng
Ra#ngScale:4-VeryMuchaBarrier3-No2ceableBarrier2-SomewhataBarrier1-NotaBarrier
ImpactofBarrierstoImplementa#on
2.8 2.8 2.92.6
1.0
2.0
3.0
4.0
FARYr1AverageRa4ng FARYr2AverageRa4ng InvYr1AverageRa4ng InvYr2AverageRa4ng
Ra#ngScale:4-ExtensiveChanges3-No4ceableChanges2-SomeChanges1-NoChanges
ExtentofPosi#veChanges
Family Assessment Response: Interim Evaluation Report 14
OneparticularreasonforthelowerratingsofpositivechangeinYearTwoisthatsomeinvestigatorsexpressedfrustrationwithnotbeingincludedasmuchastheycouldhavebeenintheFARofficerollout.FARimplementationhadadivisiveeffectwithinsomeoffices.ThishappenedforseveralreasonsbutwasmorepronouncedwheninvestigatorsfeltthatFARwasbeingapproachedasthenewest“great”thingandthattheirinvestigativeworkwaslessvalued.Additionally,shiftingcaseloadsandstaffvacanciesoftencreatedinitialhighcaseloadsthatoftenledtoconflictbetweenthetwogroupswithinsomeoffices.Overall,theresponsetoFARfrominvestigativeteamstendedtobemixed.SometeamsfeltthatsupportandcommunicationtoinvestigatorswasnotapriorityduringFARimplementation.Ascanbeseeninthechartbelow,FARcaseworkersinbothimplementationyearsreportedthatcaseworkerengagementhad“noticeable”change,whileinvestigativeworkersreported,onaverage,lessthan“some”change.
MostrespondentsreportedthatFAROfficeLeadswereabletomakesignificantprogresswithinthecommunityintermsoffindingresourcesandeducatingvariousstakeholdergroupsaboutCPSandtheFARmodel.Insomeoffices,theFAROfficeLeaddepartedafterthefirstseveralmonthsofimplementation.Caseworkersreportedtheseearlydeparturesashavingadetrimentalimpactontheirworkandtheofficerelationshipwiththecommunity.However,otherofficesreportedthatstrategiesputinplacebysupervisorsandworkershelpedthemcontinuetobuildrelationshipswithinthecommunityandtoidentifyresources.Overall,thefirsttwoyearsofimplementationdemonstratedthatFARofficesneedtoplanfortakingovercommunityoutreachresponsibilitiesoncetheFAROfficeLeadpositionexpires.
2.8 2.8
1.71.4
1.0
2.0
3.0
4.0
FARYr1AverageRa3ng FARYr2AverageRa3ng InvYr1AverageRa3ng InvYr2AverageRa3ng
Ra#ngScale:4-ChangedALot3-No3ceableChange2-SomeChange1-NoChange
HowMuchCaseworkerEngagementChanged
Family Assessment Response: Interim Evaluation Report 15
FamilyPerspectivesofFARImplementation
InadditiontoconductingkeyinformantinterviewsinFARofficestoexamineimplementationchallengesandsuccesses,TriWestalsoworkedwithparentallies(parentswithpreviousCPSinvolvementwhonowworkasfamilyadvocates)tosurveyFARfamiliesregardingtheirviewsofFARprocessesandoutcomes.ThissectionofthereportdiscusseskeyfeaturesoftheFARmodelandfamilies’perceptionsofhowwellthosewereimplemented.Itisimportanttonotethatkeylimitations(e.g.,theoptionalinclusioninthesurvey,problemswithdisconnectedphonenumbers,etc.)existinsurveyingfamilies.10OneimportantfacetofFARistousealessformalapproach(andtheabsenceofaformal“finding”)toincreasetrustandoverallengagementinthecaseprocess.Ascanbeseeninthegraphbelow,mostrespondents(88%)reportedbeingactivelyengagedinthecaseprocess“always,oralmostalways.”
“Iwasactivelyengagedwiththecaseprocess.”(N=231) Twootherimportantratingsaroundfamilyengagementaddressedtheextenttowhichthefamilyfeltthattheiropinionswerebeingconsideredwhendevelopingacaseplanorlinkingthefamilytoservices.Ascanbeshowninthetwochartsonthefollowingpage,morethanhalfoftherespondentsfeltthattheircaseworkerhadhelpedthemtoidentifythingstheirfamilyneeded.Morethantwothirdsreportedthattheircaseworker“alwaysoralmostalways”listenedtotheiropinionsaboutwhetherthefamilyneededservices.
10Surveymethodology,responseratesaswellasmorerecentlysurveydataarereportedintheDecember2016FamilySurveySummaryReport.ThisreportwillbeincludedwiththeWashingtonStateIV-EWaiverJanuary2017Semi-AnnualProgressReport.
AlwaysorAlmostAlways88%
SomeoftheTime7%
NotO7en2%Never3%
Family Assessment Response: Interim Evaluation Report 16
ChangestoFARDuringYearsOneandTwo
Asmentionedpreviously,severalchangestoFARoccurredduringYearsOneandTwoincludingasignificantsetofchangestargetingFARtrainingandcommunicationasbasedoninformationprovidedtotheCAfromourevaluationwork,caseconsultationswithoffices,andmoreinformalcommunicationswiththefield.Therewasclarificationoverthe“place”ofFARinchildwelfare,witharecommendationforstrongermessagingthatthisisstillaCPSresponseandthatchildsafetyneededtocontinuetobethesingularguidingpriorityinallcases.Additionally,trainingwasimprovedtoincludemoreinformationonthecontinuedfocusofchildsafety,clarificationaroundthevoluntarynatureofFAR,andimprovedprocessesforexplainingtheintakeprocessanddecisionmakingaroundtheassignmentofintakestoeithertheFARortheInvestigativepathway.Additionally,thelanguageintheFARagreementwaschanged(andtheagreementitselfshortened)toaddressconcernsthatitwasleadingfamiliestodeclineparticipationinFAR.Earlyindicationsshowthatthesechangeshaveimprovedimplementationinoffices,andweanticipateseeingcontinuedimprovementsintheassessmentofFARimplementationintothefuture.WecontinuetoworkcloselywithCAtodeveloparatingsystemtoassessfidelityofFARimplementationwithinofficesandtodeterminethedegreetowhichimplementationaffectsoutcomes.Thiswillbedetailedinfuturesemi-annualprogressreportsandwillbereportedintheFinalEvaluationReport.
Yes,VeryMuch45%
Yes,ALi3le25%
NotMuch12%
NotAtAll18%
AlwaysorAlmostAlways67%
SomeoftheTime22%
NotO8en5%
Never8%
Mycaseworkerhelpedidentifythingsthatcausedmyfamilyproblems.(N=228)
Mycaseworkerlistenedtowhetherornotmyfamilyneededservices.(N=225)
Family Assessment Response: Interim Evaluation Report 17
Also,CAoriginallyplannedtousetheChildandAdolescentNeedsandStrengths(CANS)tooltohelpdeterminefamilyserviceneeds.However,fewcaseworkersreportedusingthetoolthewayitwasintendedandfurtherreportedthatthetooladdedtoanalreadylengthydatacollectionprocesswithfamilies.Therefore,CAhasrecentlymadethedecisiontodiscontinueuseoftheCANS.OnechangetotheFARmodelthatdoesaffectthenumbersoffamiliesserved(whichwillbereportedinthenextsectionofthisreport)wasthedecisiontomovefamilies(regardlessofrisk)outofFAReligibilityiftheintakeinvolvedaphysicalabuseallegationofachildagedthreeyearsoryounger.ThisadjustmentdecreasedthenumberofFAR-eligiblefamiliesandthusloweredtheactualnumbersserved.AnalysisofMinorityDisproportionalitywithinFAR
Theissueofminoritydisproportionalitywithinthechildwelfaresystem,generally,isimportanttoCA.Thus,thisevaluationexaminedthedegreetowhichdecision-makingregardingFARdifferedacrossracialandethnicgroups.Onceanintake“screens-in”toCPS(whichistosay,theintakeworkerdeterminesthereissufficientinformationforaCPSresponse),theSDMIntakeToolhelpstheintakeworkertodeterminewhetherthecasemeetstheeligibilitycriteriatobereferredtotheFARpathwayor,noteligible,totheinvestigativepathway.Thetablebelowshowsthepercentageofallscreened-inallegationsthatareassignedtotheFARpathwaybyrace/ethnicity.NotethatthesepercentagesareassignmentstoFARacrossthestateeventhoughFARwasnotimplementedinalloffices.Afterfullimplementation,allpercentagesareexpectedtoincrease.
Race/EthnicityPercentof
IntakesAssignedtoFARPathway
TotalIntakes 30%
NativeAmerican 22%
AsianAmerican 28%
Black 29%
White 31%
Hispanic 27%
Multi-racial(Native) 30%
Multi-racial(Black) 33%
Multi-racial(White) 29%
Family Assessment Response: Interim Evaluation Report 18
ThepercentageofintakesthatareassignedtoFAR(asopposedtobeingscreenedtoinvestigationorassignedas“risk-only”)isconsistentacrosstheracial/ethniccategories,exceptforNativeAmericans,whohavealowerproportionofcasesscreeningtoFAR.Whilethedatadonotprovideanexplanationforthisdifference,thereisapossibilitythatthesecasesmaybetransferredovertotribalentitiesthatdonothaveaFARoptionormightbemorelikelytooccurinofficeswhereFARhasnotrolledout.ThisspeculationissupportedbythepercentageoffamiliesassignedtotheFARpathwaywhenchildrenaremembersofaWashingtonStatetribalentity,whichisevenlowerthantheoverallNativeAmericanrateofFARassignment(whichincludesWashingtonStatetribalmembers).TheCAcontinuestomonitorthisdifferenceasimplementationcontinues.DisproportionalityinRemainingintheFARPathway
OnceacaseisassignedtotheFARpathway,thevastmajorityoffamilies(91%)agreetoparticipateandcompletetheircaseunderFAR.However,insomecases,afamilymayeitherrefusetoparticipateormayhaveacasetransferredtoinvestigationsbyaworkerwhobelievesFARisnotanappropriatepathwayduetoaconcernforchildsafety.Thefollowingtableshowsdifferences,byrace/ethnicity,infamilies’pathwaydispositionaftertheirinitialpathwayassignmenttoFAR.
DisproportionalityofFARDisposition–Cohorts1-4(Years1&2)
Race/Ethnicity RemainFAR DeclinedFARTransferfor
Safety
Total 91% 6% 3%
NativeAmerican 86% 10% 4%
AsianAmerican 93% 6% 2%
Black 92% 5% 3%
White 91% 6% 2%
Hispanic 91% 5% 4%
Multi-racial(Native) 90% 6% 4%
Multi-racial(Black) 93% 4% 4%
Multi-racial(White) 92% 5% 2%
Asshowninthetableabove,theproportionofcasestransferredtoinvestigationsarevirtuallythesameforallfamilies.However,NativeAmericanfamiliesaresignificantlymorelikelytodeclinetoparticipateinFAR.IndiscussingthisphenomenonwithFARcaseworkers,TriWestlearnedthattheFARagreement,inparticular,seemedtobeasignificantbarrierforNative
Family Assessment Response: Interim Evaluation Report 19
Americanfamilies.CAisworkingtoamendthelegislativerequirementfortheFARagreementtotrytoalleviatesomeofthedisparityinNativeAmericanfamiliesdecliningtoparticipateinFAR.Preliminary Program Outcomes
ToassesstheimpactofFARonthegoalsofimprovingsafety,permanency,andwell-beingoutcomes,TriWesthasanalyzeddataonnewintakesintoCPSfollowingtheirinitialintakes,childremovalsfromthehome,andfamilyreportsofsuccessfuloutcomes.Dataarereportedforthefirstfourevaluationcohorts(seethetableonpage9forcohortdescriptions).Duetothetimingofintakes,Cohort4hasonlysixmonthsoffollow-updataatthistime.NewAcceptedIntakes
ThetablebelowshowstheproportionofFARandComparisongroupfamilieswithanewacceptedCPSintakewithinthreemonthsfollowingtheirinitialFAR(orinvestigative)case.TheComparisongrouphadasmall(butstatisticallysignificant)lowerproportionofnewintakeswhenconsideringallnewacceptedintakes.FARfamilieshadmorere-referralsingeneral,butmanycontinuedtobeFAR-eligiblereferrals,indicatingthatrisklevelshadbeenstayingthesameforthesefamilies.ComparisongroupfamilieswereeligibleforFARintheirfirstintakebutgenerallyhadfewersubsequentFAR-eligiblereferralsand,insomecases,hadsignificantlymorenon-eligiblereferrals,anindicatorthatthesefamilieswerefacinggreaterchallengeswhentheyreturned(asindicatedbyriskatintake).
FAROutcomes:FamilieswithNewCPSIntakesWithin3MonthsAfterInitialIntake,Cohorts1-4
FARMatched
ComparisonGroup
PercentoffamilieswithanynewacceptedCPSintake 12.9% 11.1%*
PercentoffamilieswithanewFAR-eligibleintake 9.8% 6.9%*
Percentoffamilieswithanewnon-FAR-eligibleintake 4.2% 5.1%*
Percentoffamilieswithanew“risk-only”intake 0.7% 0.7%
*Differencesaresignificantatthep<.05level.
Thesesamepatternsholdfornewintakesat6monthsand12months,asshowninthefollowingtables.Again,theComparisongrouphadalowerproportionoffamilieswithanynewintakes,butthisdifferencewasbeingdrivenentirelybyhavingfewerFAR-eligibleintakes.Comparisongroupfamiliescontinuedtohaveslightlylowerratesofnewnon-FAR-eligibleintakes.
Family Assessment Response: Interim Evaluation Report 20
FAROutcomes:FamilieswithNewCPSIntakesWithin6MonthsAfterInitialIntake,Cohorts1–4
FARMatched
ComparisonGroup
PercentoffamilieswithanynewacceptedCPSintake 19.8% 16.6%*
PercentoffamilieswithanewFAR-eligibleintake 14.7% 10.3%*
Percentoffamilieswithanewnon-FAR-eligibleintake 7.3% 8.2%
Percentoffamilieswithanew“risk-only”intake 1.4% 1.4%
*Differencesaresignificantatthep<.05level.
FAROutcomes:FamilieswithNewCPSIntakes12MonthsAfterInitialIntake,Cohorts1–3
FARMatched
ComparisonGroup
PercentoffamilieswithanynewacceptedCPSintake 28.4% 22.9%*
PercentoffamilieswithanewFAR-eligibleintake 21.5% 14.4%*
Percentoffamilieswithanewnon-FAR-eligibleintake 11.3% 12.1%
Percentoffamilieswithanew“risk-only”intake 2.4% 2.6%
*Differencesaresignificantatthep<.05level.
WhenanalyzingtheseparateeffectsofFARoneachcohort,wefoundthateachcohorthadahigheraveragenumberofacceptedintakesforFARfamilies.Thisincreasewasstatisticallysignificantforonlysomeofthetimeperiods(e.g.,3,6,and12months)andcohorts,anditdidnotpresentanobvioustrend.SeethetechnicalappendixforadetailedanalysisoftheeffectofFARbycohort.Findingsregardingnewintakesvariedthroughouttheotherstatesincludedintheliteraturereview.SomestatesdidfindsignificantimprovementsinnewintakesforFARfamilies,whileothersfoundnochangeorevenincreasednewintakeratesforARfamilies.Severalevaluationsalsoconcludedthatthebestpredictorofre-referralswaswhetherafamilyhadpreviousreferralswithCPS.Accordingtotheseevaluations,whenpredictingthelikelihoodofnewintakes,priorexperiencewithCPSdwarfedtheeffectsofpathway(ARvs.IR).Thisdistinctionisconsistentwithourevaluationfindings.WhenexaminingnewintakesbasedonpriorCPSinvolvement,therewerenosignificantdifferencesbasedonFARorComparisongroup
Family Assessment Response: Interim Evaluation Report 21
assignmentforfamilieswhohadnopriorintakes.FamilieswithpriorCPSinvolvementhadasignificantlygreaterlikelihoodofhavinganewintake.Seethetechnicalappendixofthisdocumentfordataregardingnewintakesbasedonpriorinvolvement.IndiscussingthesepreliminaryfindingswithFARfieldstaffandleadershipatCA,wefoundthattherewasaperceptionthatFARfamiliesmaycontinuetoreceivenewFAR-eligibleintakesatagreaterrateduetounmetservicesneeds.Thesefamiliestendtohavecomplicatedneedpatterns,whichoftencannotbeaddressedinthelimitedwindowof45days.ItisworthnotingthatstatesthathavefoundthatARhashadanimpactonreducingsubsequentintakesdonothavesuchstrictlimitsonthelengthoftimeacasecanbeopen.Theiroverallcaselengthaveragesarenotparticularlyhigh,buttheseotherstatesdohavetheflexibilitytokeepcasesopenlongerifnecessarytoprovideservices.CAdidaninternalreviewofFARcasesandfoundthat10percentwouldhavebenefittedfromservicesthatcouldhavebeenprovidedifthecasewereleftopenforalongerperiodratherthanclosedduetothe90-daytimelimit.ThisfindingsuggeststhatcreatingaprovisiontoallowanadditionaltimeextensiontoaFARcasewouldaffectarelativelysmallnumberofcasesbutinthosecases,couldprovidemoreneededservicestofamilies.FARleadershipisworkingonarequesttoamendtheprogramlegislationtoallowformoreflexibilitytokeepcasesopenlongerifthereisaneedforafamilytoreceivelonger-termservices.
Family Assessment Response: Interim Evaluation Report 22
RemovalRates
Asshowninthetablebelow,at3months,theComparisongrouphadaslightlyhigher,butstatisticallysignificant,rateofremovalsthandidFARfamilies.However,thispatternofasignificantdifferencedidnotpersistoverlongeroutcometimeframes(6monthsand12months).Itshouldbenoted,though,thatonlythefirstthreecohortshaddataavailableonremovalsforthefull12-monthwindowaftertheFARintake.
Removalsat3,6,and12MonthsAfterIntake
FAR
MatchedComparisonGroup
PercentofFamilieswithaRemovalwithin3monthsofintake
3.5% 4.0%*
PercentofFamilieswithaRemovalwithin6monthsofintake
5.1% 5.5%
PercentofFamilieswithaRemovalwithin12monthsofintake
7.0% 7.4%
(Cohorts1-2)FamiliesservedJanuary1–December31,2014(Cohort3)FamiliesservedJanuary1–June30,2015(Cohort4)FamiliesservedJuly1,2015–December31,2015(3&6monthoutcomesonly)
WhentheeffectofFARonremovalsisanalyzedseparatelybycohort,therearenostatisticallysignificantdifferencesinremovalratesbetweenFARandComparisongroupfamiliesforanyindividualcohortduringanyofthethreetimeperiods.WhiletheintenttotreatdesignnecessitatesthatallfamiliesinitiallyassignedtoFARareincludedinouranalysis,wedidexaminedifferencesinremovalratesbasedonwhetherafamilyactuallycompletedtheFARintervention.Asexpected,familieswhocompletedFARhadlowerratesofremovalsthatdidfamilieswhoeitherdeclinedparticipationorwhoweretransferredduetoconcernsregardingchildsafety.
Family Assessment Response: Interim Evaluation Report 23
FARRemovals,byCaseDisposition(3-and6-MonthRemovalRates)
3Months 6Months 12MonthsComparisonGroup3/6/12months
OverallRemovalRate 3.4% 5.30% 7.3% 4.5%/6.1%/7.7%
RemovalrateforfamilieswhoremaininFAR(89%ofallFARintakes)
2.6% 4.2% 6.5%
Removalrateforfamilieswhodeclinedorweretransferred(11%ofallFARintakes)
4.3% 7.1% 14.2%
(Cohorts1-2)FamiliesservedJanuary1–December31,2014
Family Satisfaction with FAR and Self-Reported Outcomes
Findingadifferentpathwaytoengagefamilies,toestablishtrust,andtoencouragefamiliestoacceptsupportandparticipateinservices,theFARmodelstressesworkingtogetherwithfamiliesandestablishingarelationshipthatislessadversarialthantraditionalCPSinvestigations.ToassessthedegreetowhichFARisabletoachievethisobjectiveandtoconsiderfamilies’perspectivesoftheirownimprovement,weaskedFARfamiliestoreportthedegreetowhichtheyweresatisfiedwiththeservicestheyreceivedfromFARandtheperceptionsofchangesintheirfamily’swell-being.TelephoneinterviewswereconductedwiththosefamilieswhoagreedtobecontactedbyresearcherswhentheyassignedtheinitialFARagreement.Thelargestchallengewithconductingtheseinterviewshasbeenreachingparents/caregiversbyphone.Inmanycasesphonenumberschangebetweencaseclosuresandourattemptstoconductsurveys.Inothercases,wemaydialanumbermultipletimeswithoutreceivingaresponse.11
11Itisimportantwhenconsideringtheseresultstonotethattherespondentsdorepresentthosefamilieswhowecanreachandwhoarewillingtotalktous.Inotherwords,therespondentsarenotnecessarilyfullyrepresentativeoftheentirepopulation.
Family Assessment Response: Interim Evaluation Report 24
ThemajorityofrespondentsreportedbothapositiveexperiencewithFARandpositiveoutcomesfollowingtheirparticipation.Asshownbelow,90%ofrespondentswereeitherverysatisfied(65%)or“mostlysatisfied”(25%)withtheywaythattheyandtheirfamilywastreatedbytheirFARcaseworker.Additionally,morethanhalfofrespondentsreportedthattheirfamilywasdoingeither“muchbetter”(38%)or“somewhatbetter”(23%)becauseoftheirFARparticipation.
Overall,howisyourfamilydoingbecauseofFAR?(N=228)
Verysa(sfied65%
Mostlysa(sfied25%
NA3%
Mostlydissa(sfied3%
Verydissa(sfied4%
Verysa(sfied65%
Mostlysa(sfied25%
NA3%
Mostlydissa(sfied3%
Verydissa(sfied4%
Howsatisfiedareyouwithhowyouweretreated?(N=228)
Family Assessment Response: Interim Evaluation Report 25
Morethree-quarters(79%)ofrespondentsreportedthattheywereeither“verysatisfied”(51%)or“mostlysatisfied”(28%)withtheservicestheyreceivedorwereofferedthroughtheirparticipationinFAR.
Moreover,63%ofrespondentswhohadhadapreviouschildwelfareexperiencereportedthatthisexperiencewithCPSwas“muchbetter”thantheirpreviouschildwelfareexperiences.ThisresponseindicatesthatFARisimprovingfamilyexperienceswithCPSovertime.
Overall,howsatisfiedareyouwiththeservicesyoureceived(orwereoffered)?
(N=225)Verysa(sfied
51%
Mostlysa(sfied28%
NA12%
MostlyDissa(sfied
4%
VeryDissa(sfied5%
Muchbe(er61%
Somewhatbe(er9%
Nochange18%
SomewhatWorse6%
MuchWorse6%
Overall,howwasthisexperiencebasedonyourpreviouschildwelfareexperiences?(N=88)
Family Assessment Response: Interim Evaluation Report 26
Cost Analysis FARhastwodistinctandoppositeeffectsonthecostofservices.ThefirsteffectisthatFARincreases,foralltimeintervals,theprobabilitythatfamilieswilluseaservicethatrequiresCAfunding.Thesecondeffectisthatforthosefamilies(FARandComparison)whodouseCA-fundedservices,FARfamilieshavereducedaveragecosts.Inotherwords,FARfamiliesaremorelikelytouseCAservices,butthoseservicestendtocostlessthancostsforComparisongroupfamilieswhouseCAservices.OnecomplicationwithanalyzingFARcostdataisthatservicecostsvarybycase.Formostfamilies(FARandComparison),thetotalservicecostsarezero;however,forsomefamilies,costscanbelarge.ThedistributionofthesedataisskewedsuchthatthemediancostofservicesprovidedbyCAforallfamiliesiszero.However,themean(average)costissubstantiallyabovezero.Themeanisthereforenot“typical”orrepresentative.Thevariancebetweenmedianandmeancanbeproblematicforanalysis.Manysimplestatisticaltests,suchasaT-testforthedifferenceinmeans,arepotentiallyinvalidwithdatathataremostlyzerosandhighlyskewed.Onecommontechniqueforanalysisofdataofthistypeisa“hurdle”model.Applyingthismodel,wehaveestablishedthatthefirsthurdlepredictstheprobabilitythatafamilywillrequireanycosts.Thesecondhurdlepredictsthemagnitudeofthecostsforanyfamilywithpositivecosts.Thetablebelowpresentstheoveralltwo-stephurdlemodelresultsforFARandComparisongroupfamilies.Dataforallofthecostanalysiscanbefoundinthetechnicalappendix(AppendixA)inthisdocument.
ServiceCostsAnalysis(WithoutSeparateCohortTreatment)
TimeInterval
HurdleExpectedValue
MagnitudeofEffect
FAR Comparison Hurdle1:DoesFARaffect
whetherfamilieshaveanypaidservices?
Hurdle2:DoesFARaffect
highercosts?
Combined
3Months $345 $228 Yes(morelikely) Yes $117*
6Months $645 $655 Yes(morelikely) No -$10*
12Months $1,258 $1,724 Yes(morelikely) No -$465**P-value=0.00
Howtoreadthecostdatatable:Thetableabovepresentskeyresults.The“HurdleExpectedValue”sectionisdividedintothetwogroups,FARandComparison.TheFARcolumnpresents
Family Assessment Response: Interim Evaluation Report 27
theexpectedcostsifeveryeligiblefamilywasservedundertheFARpathway.TheComparisoncolumnpresentstheexpectedcostsifeveryfamilyinsteadreceivedtheInvestigativeapproach.ThedifferencebetweenthetwocolumnsistheestimatedeffectofFAR.The“MagnitudeofEffect”sectionofthetablepresents,intheright-mostcolumn(“Combined”),thisdifference.Theothertwocolumnsin“MagnitudeofEffect”respondtokeycostquestions.Hurdle1designateswhetherFARhadapositiveornegativeeffectontheprobabilitythatafamilywillhaveanypaidservices.Foreachtimeinterval,FARfamiliesaremorelikelythanComparisonfamiliestouseservicespaidforbyCA.12Thiseffectisstatisticallysignificant(p<.00).Hurdle2considerstheaveragecostsofCApaidservicesfortherespectivegroups,FARandComparison.AnegativeeffectindicateslowercostsforFARfamilieswhohavecoststhanforComparisongroupfamilieswhohavecosts.Thisfindingisalsostatisticallysignificant(p<.00).The“Combined”columnpresentstheactualaveragevarianceincostspertimeintervalgiventhecombinedeffectofbothhurdles.Accordingtothesefindings,FARfamiliesaremorelikelytohaveapaidserviceofanykind.AndFARservicescost,onaverage,$117moreperfamilyovera3-monthperiod.However,over6monthsand12months,servicesforFARfamiliescost,onaverage,$10and$465lessthanComparisonfamilycostsperfamilyserved.TheseresultsareconsistentwiththeFARmodel:servicesareprovidedtofamiliestoresolveproblemsandpreventfutureinvestigationsandremovals. Summary and Conclusions
Atthetwo-yearmark,theFARprogramoffersseveralnotablefindings.Ononehand,bothcaseworkersandfamiliesservedbytheFARprogramreportoverallhighlevelsofsatisfactionwiththeimplementationoftheFARpathway.Ontheother,outcomesforfamilies,asmeasuredbyreductionsinnewintakesandremovals,havenotshownsignificantbenefitsforFARfamilies.However,thesenon-dynamicmeasuresmaynottellthefullstory.Forexample,therelativelystagnantmeasureofbenefitsincludesmeasuresfrompartialimplementationandearlyimplementation.Assuch,implementationadjustmentsbasedonearlyfindingsandincreasedfamiliaritywiththeFARmodelforcaseworkers,supervisors,administrators,andothersmayleadtomorepositiveoutcomesatfutureintervals.
12ThisincludesonlythosecostspaidforbyCA(notcommunityservicesnotpaidbyCA),butdoesincludeallcosts,includingthoserelatingtoplacement.
Family Assessment Response: Interim Evaluation Report 28
Asmuchasweremainoptimisticaboutthewaysthatgreaterfamiliarity,experience,andmodificationwillbenefitongoingFARimplementation,wedoofferrecommendationsforthisbenchmark.SomeofthespecificrequirementsputforthinFARenablinglegislationsmayhaveunintentionallylimitedtheeffectivenessoftheprogram.Basedonourfindingsininterviews,fieldresearch,literaturereview,andotherdata,weinclude,aspartoftheinterimevaluation,tworecommendationsthatarelikelytoaddresssomeofthelimitsrevealedinthisreport.Theserecommendationsarelistedasfollows:
EliminatetheFARAgreement.CaseworkersreportthattheFARAgreementcanbeasignificantbarrierforsomefamilies.OneparticularconcernisthatNativeAmericanfamilies,comparedwithothergroups,aredisproportionatelydecliningtoparticipateinFAR.Whiletheadministrativedatadonotallowustoconclusivelydeterminethattheagreementisthereasonforthisdisproportionality,interviewswithcaseworkersindicatethatNativeAmericanfamiliesareoftenreluctanttosignanofficialgovernmentdocumentinordertoparticipate.WhilewecannotguaranteethattheeliminationoftheFARAgreementwilleliminatetheobserveddisproportionalityforNativeAmericanfamiliesaltogether,evidencesuggeststhatthisadjustmentwillleadtoareductionindisproportionality.Allowforanadditionalcaseextensionforcasesinwhichtheadditionaltimeisneededtoprovideaspecificservice.Thecurrenttimelimitsignificantlyrestrictswhatservicescanbeprovidedtofamilies,particularlyEvidence-BasedPracticesthathaveservicedurationslongerthan90days.Evenforthoseservicesthathavea60–90-daytimeframe,thetimeneededtocompleteacomprehensivefamilyassessment,selectaservice,andmakeareferraltoaprovidercansignificantlytruncatetheavailabletime.
Finally,inadditiontotheabove,werecommendthatCAcontinueitsongoingeffortstomonitortrainingqualityandprovidefollow-upresourcesintheformsofcaseconsultations.TheseCAefforts,togetherwithimplementationoftheabove-listedpolicyrecommendations,willlikelyaidCAinitseffortsandassisttheFARprograminitseffectivenessandservicetothefamiliesofWashingtonState.