11
The Wraparound ProcessThe Wraparound ProcessAn in-depth look at fidelity patterns fromAn in-depth look at fidelity patterns from
a national dataseta national datasetKristen Kristen M. Leverentz-BradyM. Leverentz-Brady, M.A., M.A.
Jesse Jesse C.C. SuterSuter, M.A., M.A.University of Vermont Department of PsychologyUniversity of Vermont Department of Psychology
Eric J. Eric J. BrunsBruns, Ph.D., Ph.D.University of Washington Department of PsychiatryUniversity of Washington Department of Psychiatry
1818thth Annual Florida Mental Health Institute Annual Florida Mental Health InstituteResearch Conference on ChildrenResearch Conference on Children’’s Mental Healths Mental Health
March 7, 2005March 7, 2005
For more information: For more information: [email protected]@u.washington.eduwww.uvm.edu/~wrapvtwww.uvm.edu/~wrapvt or or depts.washington.edu/wrapevaldepts.washington.edu/wrapeval
22
IntroductionIntroductionJohn John BurchardBurchard’’ss Wraparound Fidelity Wraparound FidelityIndex (WFI-3) assesses adherence toIndex (WFI-3) assesses adherence toprinciples of the wraparound processprinciples of the wraparound processthrough interviews with caregivers, youths,through interviews with caregivers, youths,and providers.and providers. 11 Elements of Wraparound11 Elements of Wraparound 4 items per element4 items per element Responses scored by interviewer Responses scored by interviewer fromfrom 0 (low 0 (low
fidelity) to 2 (high fidelity) to 2 (high fidelity)fidelity)Requested by over Requested by over 250250 communitiescommunitiesUsed byUsed by at least at least 5050 communities nationally communities nationally
33
Prior Research on WFIPrior Research on WFI
WFI Total scores and most element scoresWFI Total scores and most element scoresfound to have good psychometric propertiesfound to have good psychometric properties(e.g., internal consistency, test-retest reliability)(e.g., internal consistency, test-retest reliability)WFI Total scores associated with externalWFI Total scores associated with externalexpert ratings of fidelity using more intensiveexpert ratings of fidelity using more intensivemethodmethodWFI Total scores discriminate WFI Total scores discriminate different types ofdifferent types ofprograms (e.g., wraparound vs. comparison)programs (e.g., wraparound vs. comparison)Emerging evidence for association betweenEmerging evidence for association betweenWFI scores and child and family outcomesWFI scores and child and family outcomes
44
Unanswered questionsUnanswered questionsFor communities using the WFI and researchersFor communities using the WFI and researchers
WhatWhat is the level of agreement between is the level of agreement betweenWFI respondents?WFI respondents?What is the underlying structure of theWhat is the underlying structure of theWFI? What does it tell us about the wayWFI? What does it tell us about the waywraparound occurs in the wraparound occurs in the ““real worldreal world””??What constitutes a What constitutes a ““goodgood”” (or (or ““acceptableacceptable””))fidelity score?fidelity score?
55
TodayToday’’s presentations presentationWe will present results of analyses of a nationalWe will present results of analyses of a nationalWFI-3 dataset for 667 youth from 10WFI-3 dataset for 667 youth from 10communities, focusing on:communities, focusing on:AgreementAgreement between youths, caregivers, and between youths, caregivers, and
facilitators,facilitators,How different hypothesized models for wraparoundHow different hypothesized models for wraparound
implementation fit with WFI-3 data (confirmatoryimplementation fit with WFI-3 data (confirmatoryfactor analysis), andfactor analysis), and
What the national dataset and previously publishedWhat the national dataset and previously publishedevaluation studies using the WFI-3 tell us aboutevaluation studies using the WFI-3 tell us aboutsetting standards for wraparound setting standards for wraparound fidelityfidelity
ImplicationsImplications for quality assurance, research,for quality assurance, research,and model developmentand model development
66
Extending understandingExtending understandingof the WFI:of the WFI:
Agreement between Agreement between WFIWFIrespondentsrespondents
Kristen M. Leverentz-Brady, M.A.Kristen M. Leverentz-Brady, M.A.University of Vermont Department of PsychologyUniversity of Vermont Department of Psychology
Wraparound Evaluation and Research TeamWraparound Evaluation and Research [email protected]@uvm.edu
77
Background: WFI Properties andBackground: WFI Properties andPsychometricsPsychometrics
What has been assessed:What has been assessed: Test-retest reliabilityTest-retest reliability Internal consistencyInternal consistency Concurrent validityConcurrent validity Criterion-related validityCriterion-related validityWhat has not been assessed:What has not been assessed: Stability over time and typical trajectory ofStability over time and typical trajectory of
fidelity scoresfidelity scores Inter-respondent agreementInter-respondent agreement
88
National WFI SampleNational WFI SampleN=667 families from 10 collaborating sites in 9N=667 families from 10 collaborating sites in 9statesstates N=622 RF interviewsN=622 RF interviews N=490 caregiver interviewsN=490 caregiver interviews N=367 YouthsN=367 Youths
Data collection method:Data collection method: Sites received manual and training materialsSites received manual and training materials Each site completed Memorandum of Agreement toEach site completed Memorandum of Agreement to
administer WFI in full and to adhere to interviewadminister WFI in full and to adhere to interviewprotocolsprotocols
Majority of sites administered interviews by phoneMajority of sites administered interviews by phone18% of RF, 14% of Caregiver, and 8% of Youth interviews18% of RF, 14% of Caregiver, and 8% of Youth interviewsconducted face-to-faceconducted face-to-face
99
National WFI study sampleNational WFI study sample
367490622667Total NsTotal Ns8151921Minnesota 25172427Florida
23313131Nevada27323434Missouri16222222North Carolina12232626Minnesota 129717474Massachusetts12141718Pennsylvania
212226339366Nebraska23394449Indiana
N YouthN CGN RFN familiesSite
1010
Results: Agreement BetweenResults: Agreement BetweenRespondentsRespondents
For all three respondents, a moderateFor all three respondents, a moderatecorrelation was foundcorrelation was found ICC = .58ICC = .58For Individual respondents:For Individual respondents: RF-CG = .44RF-CG = .44 CG-Y = .49CG-Y = .49 RF-Y = .45RF-Y = .45
1111
Findings and implicationsFindings and implicationsModerate agreement found for WFI Total scoresModerate agreement found for WFI Total scoresacross all respondentsacross all respondents Higher than found in a meta-analysis of parent-mental healthHigher than found in a meta-analysis of parent-mental health
worker cross-informant scores for children (Achenbach,worker cross-informant scores for children (Achenbach,McConaughy, & Howell, 1987)McConaughy, & Howell, 1987)
ConsistentConsistent with with agreementagreement found in a meta-analysis found in a meta-analysis ofof ratings ofratings ofadults (Achenbachadults (Achenbach,, Krukowski Krukowski,, Dumenci Dumenci, &, & Ivanova Ivanova, in , in press)press)
LevelLevel of agreement provides evidence for of agreement provides evidence forreliability of WFI, but also suggests thatreliability of WFI, but also suggests thatindividual respondent scores may differindividual respondent scores may differmeaningfully across individual respondentsmeaningfully across individual respondents Considering all respondentsConsidering all respondents’’ scores individually will scores individually will
likely be important in considering fidelity levelslikely be important in considering fidelity levels
1212
Recipes for Wraparound:Recipes for Wraparound:Comparing wraparound models usingComparing wraparound models using
confirmatory factor analysisconfirmatory factor analysis
Jesse C. Suter, M.A.Jesse C. Suter, M.A.University of Vermont Department of PsychologyUniversity of Vermont Department of Psychology
Wraparound Evaluation and Research TeamWraparound Evaluation and Research [email protected]@uvm.edu
1313
OverviewOverview
Wraparound developed asWraparound developed as value-based modelvalue-based model(e.g., community-based, family-focused)(e.g., community-based, family-focused)Service Service providers have struggledproviders have struggled to translate to translatephilosophy intophilosophy into effective real-world practiceseffective real-world practicesRecent effortsRecent efforts provided more clearly specifiedprovided more clearly specifiedmodels of wraparoundmodels of wraparoundCurrent study provides an empirical test for howCurrent study provides an empirical test for howwell two wraparound well two wraparound modelsmodels fit with fit with ratings fromratings fromcaregivers on the WFIcaregivers on the WFI
1414
OutlineOutline
Models of wraparoundModels of wraparoundMethod for comparing modelsMethod for comparing modelsFindingsFindingsConclusionsConclusionsImplications and future directionsImplications and future directions
1515
Wraparound ElementsWraparound Elements
1.1. VoiceVoice and choiceand choice2.2. Team-basedTeam-based3.3. Community-basedCommunity-based4.4. Cultural competenceCultural competence5.5. IndividualizedIndividualized andand
strengths-basedstrengths-based
6.6. Natural supportsNatural supports7.7. Continuation of careContinuation of care8.8. CollaborationCollaboration9.9. Flexible resourcesFlexible resources
and fundingand funding10.10. Outcomes-basedOutcomes-based
(Burns, B. J., & Goldman, S. K., 1999).
1616
Necessary ConditionsNecessary Conditions
Team uses documentation forTeam uses documentation forcontinuous improvement andcontinuous improvement andaccountability.accountability.
5.5. AccountabilityAccountability
TeamTeam is aware of wide array ofis aware of wide array ofservices, identifies/develops naturalservices, identifies/develops naturalsupports, and designs/tailors servicessupports, and designs/tailors servicesto expressed needs.to expressed needs.
4.4. Acquiring services &Acquiring services &supportssupports
Team members capably perform theirTeam members capably perform theirroles on the team.roles on the team.
3.3. Capacity buildingCapacity building
Regular attendance and participationRegular attendance and participationby appropriate people.by appropriate people.
2.2. CollaborationCollaboration
Team adheres to practice modelTeam adheres to practice modelwhile promoting team cohesivenesswhile promoting team cohesivenessand high quality planning.and high quality planning.
1.1. Adherence toAdherence tophilosophical modelphilosophical model
(Walker, J., Koroloff, N. & Schutte, K., 2003).
1717
Steps forSteps for Model TestingModel Testing
WFI-3 caregiver interviews (n = 444)WFI-3 caregiver interviews (n = 444)Specify modelsSpecify models Each item assignedEach item assigned to only one factorto only one factor Factors allowed to correlateFactors allowed to correlateItem-level CFA using WLSMV Item-level CFA using WLSMV estimationestimation(Levent, 2004)(Levent, 2004)
RMSEA of .06 or lower indicates good fitRMSEA of .06 or lower indicates good fit CompareCompare to one-factor modelto one-factor model (parsimony test) (parsimony test)
1818
Findings: Model SelectionFindings: Model Selection
0.0590.057
0.067
0.053
0.063
0.04
0.05
0.06
0.07
10 Elements Necessary
Conditions
"g"
Model
RMSEA
items 44
items 40
Cut-off for good model fit <= .06
1919
ConclusionsConclusions
Take-home findingsTake-home findings Both models showed adequate fitBoth models showed adequate fit Both models better thanBoth models better than one-factorone-factor Removing a poorly fitting element improved fitRemoving a poorly fitting element improved fitPotential limitationsPotential limitations Selected sampleSelected sample WFI items designed for one modelWFI items designed for one model Caregiver onlyCaregiver only
2020
Implications for quality assuranceImplications for quality assuranceand program developmentand program development
Results support presenting WFI fidelityResults support presenting WFI fidelitydata for individual elementsdata for individual elements Such results can be used to support trainingSuch results can be used to support training,,
supervision, supervision, and program developmentand program development Results may be useful for assessing andResults may be useful for assessing and
addressing organizational and systemaddressing organizational and systemconditions in a communityconditions in a community
2121
Implications for researchImplications for research
Results provide evidence for validity of the WFIResults provide evidence for validity of the WFIand its underlying elementsand its underlying elements Future research may illuminate how best to combineFuture research may illuminate how best to combine
elements into subscales that possess adequateelements into subscales that possess adequatereliability (internal consistency)reliability (internal consistency)
Results provide evidence for validity of theResults provide evidence for validity of thenecessary supports modelnecessary supports model Future research may investigate relationshipFuture research may investigate relationship
between necessary conditions subscales constructedbetween necessary conditions subscales constructedfrom WFI data and Portland State RTC system andfrom WFI data and Portland State RTC system andorganizational assessmentsorganizational assessments
2222
Implications for NWIImplications for NWI
National Wraparound Initiative: supportsNational Wraparound Initiative: supportsconsideration of value-based principlesconsideration of value-based principlesduring process ofduring process of Further defining wraparoundFurther defining wraparound Developing specific phases & activitiesDeveloping specific phases & activities Creating implementation supportsCreating implementation supports Creating next generation of fidelity toolsCreating next generation of fidelity tools
2323
““Is it Wraparound Yet?Is it Wraparound Yet?””Bootstrapping Bootstrapping wraparoundwraparound
fidelityfidelity standardsstandards usingusing the WFI the WFIEric J. Eric J. BrunsBruns, Ph.D., Ph.D.
University of WashingtonUniversity of WashingtonDivision of Public Behavioral Health and Justice PolicyDivision of Public Behavioral Health and Justice Policy
Wraparound Evaluation and Research TeamWraparound Evaluation and Research [email protected]@u.washington.edu
206-685-2477206-685-2477
2424
BackgroundBackgroundFidelity measurement is the Fidelity measurement is the ““natural unionnatural unionof scientific and practical needsof scientific and practical needs”” ( (SalyersSalyerset al., 2003)et al., 2003) Assist inAssist in interpretation of study interpretation of study results,results,
determine effective components of modelsdetermine effective components of models Help programs and trainers apply QualityHelp programs and trainers apply Quality
Assurance activitiesAssurance activities Help agencies or jurisdictions with policy,Help agencies or jurisdictions with policy,
funding, certification decisionsfunding, certification decisions
2525
Fidelity benchmarks forFidelity benchmarks forwraparound?wraparound?
Wraparound model has been slow to Wraparound model has been slow to becomebecomestandardizedstandardizedNo single model can probably yet be consideredNo single model can probably yet be considered““evidence-basedevidence-based”” by traditional treatment by traditional treatmentresearch standardsresearch standardsNonetheless, WFI items are based on coreNonetheless, WFI items are based on coreprinciples that are widely accepted (and principles that are widely accepted (and recentlyrecentlyrefined)refined)Both researchers and collaboratingBoth researchers and collaboratingcommunities are interested in assistance communities are interested in assistance inininterpretinginterpreting wraparound fidelity scores wraparound fidelity scores
2626
What methods can be used toWhat methods can be used todetermine fidelity standards?determine fidelity standards?
Norm-referencedNorm-referenced Comparison Comparison toto a large, representative a large, representative samplesampleCriterion-referencedCriterion-referenced Self-referencedSelf-referenced = compare scores to your = compare scores to your
own scores over timeown scores over time Content-referenced Content-referenced = compare scores to an= compare scores to an
absolute criterion (e.g., absolute criterion (e.g., ““90% fidelity90% fidelity””)) Expectancy-referencedExpectancy-referenced = Prediction of = Prediction of
performance based on external criteriaperformance based on external criteriae.g., compare scores to a score shown to predicte.g., compare scores to a score shown to predictdesired client outcomesdesired client outcomes
2727
Current Current approachapproach
Our ability to apply the describedOur ability to apply the describedapproaches is variableapproaches is variableOurOur aim is to aim is to ““bootstrapbootstrap”” the process the processthrough a combination of methodsthrough a combination of methods Study 1:Study 1: Norm-referenced Norm-referenced study usingstudy using our our
national WFI-3 datasetnational WFI-3 dataset Study 2:Study 2: Criterion-referenced Criterion-referenced reviewreview of of
studies employing the WFIstudies employing the WFI
2828
National WFI study sampleNational WFI study sample
367490622667Total NsTotal Ns8151921Minnesota 25172427Florida
23313131Nevada27323434Missouri16222222North Carolina12232626Minnesota 129717474Massachusetts12141718Pennsylvania
212226339366Nebraska23394449Indiana
N YouthN CGN RFN familiesSite
2929
WFI Overall Fidelity ScoresWFI Overall Fidelity Scores
0.020.767Meanb0.090.80110b0.070.8009b0.110.7978ab0.100.7957ab0.090.7946ab0.120.7535ab0.090.7514a0.070.7353a0.090.7352a0.110.7221
Post-hocSDmean
Site(Rank)
0.6
0.65
0.7
0.75
0.8
0.85
0.9
Site
WF
I O
vera
ll F
idelity
sco
re
F (9,656) = 5.951, p<.0001
Mean
3030
Total WFI Results: FacilitatorsTotal WFI Results: Facilitators
0.030.805Meanb0.050.85510b0.100.8359b0.090.8278ab0.100.8207b0.080.8176ab0.170.8125ab0.090.7994ab0.070.7783a0.090.7632a0.110.7461
Post-hocRF SDRF mean
Site(Rank)
0.6
0.65
0.7
0.75
0.8
0.85
0.9
Site
WF
I-R
F T
ota
l F
idelity
sco
re
F (9,612) = 7.452, p<.0001
Mean
3131
Total WFI Results: CaregiversTotal WFI Results: Caregivers
0.040.737Meanb0.090.79210b0.140.7859b0.160.7788ab0.130.7697ab0.150.7656ab0.100.7305ab0.140.7134ab0.140.6943ab0.200.6932a0.190.6491
Post-hocCG SDCG mean
Site(Rank)
0.6
0.65
0.7
0.75
0.8
0.85
0.9
Site
WF
I-C
G T
ota
l F
idelity
sco
re
F (9,480) = 3.195, p<.001
Mean
3232
Total WFI Results: YouthTotal WFI Results: Youth
0.030.736Meanb0.040.84310b0.140.8219b0.120.7938ab0.090.7737ab0.140.7516ab0.130.7445ab0.140.7074ab0.140.6893ab0.150.6742a0.170.5651
Post-hocY SDY mean
Site(Rank)
0.6
0.65
0.7
0.75
0.8
0.85
0.9
Site
WF
I-Y
T
ota
l F
idelity
sco
re
F (9,357) = 3.659, p<.0001
Mean
3333
Summary: Mean WFI-3 site scoresSummary: Mean WFI-3 site scores
80.5
73.7 73.6
76.7
60
65
70
75
80
85
90
95
100
WFI-RF WFI-CG WFI-Y Overall Fidelity
WF
I F
idelity
3434
Study Study 2:2: Criterion-referenced reviewCriterion-referenced review
Published, in press, or formally presentedPublished, in press, or formally presentedstudies presenting WFI scores forstudies presenting WFI scores forindependent samples that either:independent samples that either:
Predict inclusion in a group orPredict inclusion in a group orAchievement of an external criterionAchievement of an external criterion
Five studies foundFive studies found Bruns, Leverentz-Brady et al., 2004Bruns, Leverentz-Brady et al., 2004 Ferguson, 2005Ferguson, 2005 Peterson et al., 2004Peterson et al., 2004 Rast et al., 2004Rast et al., 2004 Rast & VanDenBerg, 2004Rast & VanDenBerg, 2004 (Bruns, Rast, Walker, Peterson, & Bosworth, in press)(Bruns, Rast, Walker, Peterson, & Bosworth, in press)
3535
WFI scores predicting groupWFI scores predicting groupmembershipmembership
60
76
50
55
60
65
70
75
80
85
90
95
100
Trad CW Wraparound
64
75
50
55
60
65
70
75
80
85
90
95
100
Trad CW Wraparound
Ov
era
ll W
FI
sc
ore
Ferguson et al (2004): Randomizedtrial of Wraparound in California
Peterson et al (2004): Matchedcomparison study of WA in Nevada
3636
WFI scores associated withWFI scores associated withexternal criterionexternal criterion
72
84
50
55
60
65
70
75
80
85
90
95
100
Poorer supports for
WA
Greater supports for
WA
72
87
50
55
60
65
70
75
80
85
90
95
100
RF with Poorer
outcomes
RF with more
Positive outcomes
Ov
era
ll W
FI
sc
ore
Rast et al (2004): Facilitator-levelassociation between WFI and outcomes
Bruns et al (2004): Association between fidelity& program and organizational supports
3737
WFI scores for a program overWFI scores for a program overdevelopmental stagesdevelopmental stages
64
72
86
50
55
60
65
70
75
80
85
90
95
100
Pre-training Post-training Training with
coaching
Overa
ll W
FI sco
re
Rast & VanDenBerg (2004): Impact of coachingand certification on wraparound fidelity
3838
Summary: Is it Wraparound Yet?Summary: Is it Wraparound Yet?
60
7275
72
84
64
7276 77
86
64
72
80
87
50
55
60
65
70
75
80
85
90
95
100
Non-WA /
pre-training
WA
WA with
poorer
support/
outcomes
WA as usual
from comp.
studies
National
sample (min,
mean, max)
Well-
supported/
outcome-
based WA
Overa
ll W
FI sco
re
Borderline
Non-wrap
Acceptable
High-fidelity
3939
ConclusionsConclusionsThere is a discernable pattern of WFIThere is a discernable pattern of WFIFidelity scores from across studiesFidelity scores from across studies Wraparound vs. non-wraparound programsWraparound vs. non-wraparound programs Wraparound programs with different levels ofWraparound programs with different levels of
supportsupport Facilitators and/or programs that achievedFacilitators and/or programs that achieved
more positive outcomesmore positive outcomesSite-level scores from the national WFISite-level scores from the national WFIdataset show significant variability, but falldataset show significant variability, but falllogically within the patternlogically within the pattern
4040
ConclusionsConclusionsBy combining data fromBy combining data fromthese norm- and criteria-these norm- and criteria-referenced approaches, wereferenced approaches, wecan make provisionalcan make provisionalguessesguesses What is wraparound?What is wraparound? What is What is ““high-fidelityhigh-fidelity””
wraparound?wraparound?Also can be done forAlso can be done forindividual respondent WFIindividual respondent WFIscoresscoresLikely to be useful forLikely to be useful forcollaborating sites as well ascollaborating sites as well asin research studiesin research studies
4141
LimitationsLimitations
Approach truly is art as much as scienceApproach truly is art as much as science especially in the absence of outcomes dataespecially in the absence of outcomes dataWe donWe don’’t know much about thet know much about theprograms in the national sampleprograms in the national sampleInconsistent methods used acrossInconsistent methods used acrossstudies reviewed and programs instudies reviewed and programs innational samplenational sample
4242
ConcernsConcernsHow to set fidelityHow to set fidelitythresholds forthresholds forwraparound withwraparound withlocal variations?local variations?How to reconcile withHow to reconcile witha lack of modela lack of modelstandardization?standardization?Also, how to avoidAlso, how to avoidfidelity standardsfidelity standardsproviding a providing a ““ceilingceiling””as well as a as well as a ““floorfloor””??
4343
Wraparound EvaluationWraparound Evaluationand Research Teamand Research Team www.uvm.edu/~wrapvtwww.uvm.edu/~wrapvt www.depts.washington.edu/www.depts.washington.edu/
wrapevalwrapeval [email protected]@u.washington.edu [email protected]@uvm.edu
Tell us what you think!Tell us what you think!