Evidence-Based Home Visiting Models
to Prevent Child Maltreatment – Assessing and Addressing Fidelity
February 12Anne Duggan, ScD
Research Supported by:NIH, CDC, HRSA MCHB, ACF
Annie E. Casey Foundation, Robert Wood Johnson Foundation, David & Lucile Packard Foundation
Hawaii Department of Health, Hawaii Family Support
Institute, Alaska State Health Department, Alaska Mental Health
Trust, Family League of Baltimore City; Safe & Sound Initiative;
United Way of Central Maryland
Today’s Talk
• Describe Evidence-based Models of Home Visiting
• Identify basic components of models and associated fidelity measures
• Introduce and demonstrate use of a conceptual framework for research to increase home fidelity and impact
Evidence-based Practices
An evidence-based practice, also
called EBP, refers to an approach to
prevention or treatment that is
validated by some form of
documented research evidence.
Rand Corporation Promising Practices Criteria for Research
Evidence Proven Promising
Types of Outcomes
Program directly impacts indicators of interest
Pgm impacts inter-mediary outcomes
Effect Size1 or more outcomes
changed by > 20% or > .25 sd
Change in outcome >1%
Statistical Significance
p<.05 p<.10
Comparison Groups
RCT or quasi-experimental Comparison group, but weaker
Sample Size
N > 30, each group N > 10, each group
Three Major HV Models
PAT
Pilot (1981) MI Statewide Scale Up (1985) National Scale Up (1986) RCT (1993)
NFPRCTs (1977, 1987, 1994) National Scale Up (1996) Nonprofit Replication Org. (2003)
HHS/ HFA
Demonstration Pilot of HHSP (1985) HFA National Scale Up (1992) HHSP RCT (1992, 1993) HI Statewide Scale Up (2001)
Designations of the Three Models
PAT NFP HFA
SAMHSA Pending PendingNot
Submitted
OJJDP PROMISING EXEMPLARY EFFECTIVE
Rand Corp. PROMISING PROVEN PROVEN
Red= All 3 programsBlue= 2 programsGreen= 1 program
Adoption of PAT, NFP and HFA Home Visiting Models
PAT: >3300 sites, 50 states
NFP: 118 sites, 25 statesHFA: 430 sites, 35 states
What are the Basic Components of a Home Visiting
Model?
• Families to be targeted
• Outcomes to be achieved
• Causal chain from inputs to outcomes
PAT, NFP, HFA Target Different Families
PAT All pregnant women and families with child <6 months old
NFP First time mothers, <26 wks gestation, and <19 years old or single or low SES
HHS/ HFA
CAN Risk per Family Stress Checklist >25Hx of abuse as child; SU, PMH, Criminal Hx; Low
self-esteem, poor coping ability; multiple life stressors; violence potential; unrealistic
expectations of child; harsh punishment of child; sees child as difficult/provocative
PAT, NFP, HFA Focus on Similar Outcomes
PAT
Increase parenting knowledge and behaviorIdentify developmental delays and health issuesPrevent CANIncrease school readiness and success
NFP
Improve pregnancy outcomes; Promote child health and development; Strengthen families’ economic self-sufficiency
HHS/ HFA
Improve family functioningPrevent CANPromote child health and development
Process
Screening & Assessment
Direct Home Visiting Services
Medical Home
Other Needed Community
Services
Family FunctioningParent Mental Health, SU, IPV
Social Support
Economic Sufficiency
Child OutcomesHealth and Development
Program Model
Outreach
Trust Building
Crisis Intervention
Case Management
Parenting Education, Role Modeling, Reinforcement
Framework for Influence of Home Visiting on Family and Child Outcomes
Parenting
Knowledge/Attitudes/Skill
Parent-Child Interaction
Environment for Learning
CAN
linkages
enrollment
Measures to Assess Fidelity to the Basic Components of an
EBHV Model
• How well families are targeted
• How well outcomes are achieved
• How well each element in the causal chain is carried out
What Do We Know about Home Visiting as a Preventive
Intervention?
1. Home visiting can improve outcomes, but effects tend to be modest and variable.
From Meta-Analytic Studies: HV can be effective, but effect sizes are
small.Domain ES
Cognitive Development (41 studies) .18*
Socio-emotional Development (24) .10*
CAN Prevention (7) .32
CAN – Prevention of Potential Abuse (13) .24*
Parenting Stress (4) .21
Parenting Behavior (37) .14*
Parenting Attitudes (15) .10*
Maternal Education (5) .13*
Maternal Employment (7) .02
Public Assistance (3) -.04
ES KeySmall .20
Medium .50
Large .80
*p<.05
Sweet & Applebaum, 2002
From single studies, we see that: Effects can vary over time, & across
subgroups.
Birth – 2 Yrs
All Families10% vs. 5%, NS
Poor, Unmarried Teens
19% vs. 4% p=.07
2-4 Yrs
All FamiliesNo group difference
Poor, Unmarried Teens
No group difference
Birth – 15 Yrs
All Families0.73 vs. 0.44
p<.05
Violent FamiliesNo group difference
What Do We Know about Home Visiting as a Preventive
Intervention?
1. Home visiting can improve outcomes, but effects tend to be modest and variable.
2. Programs like HSP/HFA target
– the right families and individuals,
– at the right time,
– focusing on the right outcomes
HHS/HFA Home Visiting Model - Who is Targeted, and When
• WHO IS TARGETED– Caregivers in Families with Multiple Malleable
Risks
– Kempe Family Stress Checklist (“at risk” > 25)
• WHEN ARE THEY TARGETED?– Prenatally, at Child’s Birth, Shortly Thereafter
HHS/HFA targets the right families – those with multiple, malleable risks for poor
parenting.
18%
34%41% 41%
19%24%
1%
13%
CAN Mother
Depressed
Child Language
Delay
Poor Classroom
Concentration
Not-at-Risk Families do better than At-Risk Controls
Outcomes in 1st Grade
HV Targets Right Individuals Caregivers = Primary Influence in EC
Child’s Developmental Trajectory
Brain Development
Caregivers
Exposure down with age
Community e.g. Socioeconomic and EC SERVICES
PeersExposure up
with age
Self-Regulation
Communicating and Learning
Making Friends & Getting Along
Environment
Risk & Protective Factors
Proximal
Distal
Genotype
Source: Adapted from Tremblay, R. E. 2006. www.excellence-earlychildhood.ca
HSP/HFA Targets Caregivers at the Right Time: Early Parenting Tracks into Grade School
AOR* p
Nonviolent Discipline 4.5 <.01
Psychological Aggression 5.9 <.01
Minor Assault 5.1 <.01
Severe Assault 7.6 <.01
Neglectful Behavior 3.7 <.01
Confirmed CPS Report 6.7 <.01
*AOR for Later Use if Tactic Used Birth–3 Years
Home Visiting Focuses on the Right Outcomes: Parenting and Its
Determinants
Parent’s Developmental
History
Partner Relations
Child Attributes
Social Network
PARENTINGPersonality / Relationship
Capacity
Child Outcomes
Adaptation of Belsky’s Framework
Stresses
The quality of parent-child interaction is most vulnerable to maternal relationship insecurity
under conditions of stress.
0
1
2
3
4
5
6
7
8
9
Low Stress High Stress
ContinuousSecure
EarnedSecure
Insecure
Mean and 95% CI
Phelps JL, Belsky J and Crnic K.
1998
HHSP/HFA Targets the Right Outcomes: Association of Maternal Depression & IPV with
Severe Physical Abuse of Child, Birth – 3 Years
Depression AOR 95% CI
None 1.0 Ref.
Possible (CES-D 16-23) 1.8 0.8, 3.8
Probable (CES-D >23) 3.7 1.9, 7.2
Intimate Partner Violence
Neither injured 1.0 Ref.
Only mother injured 0.8 0.1, 6.0
No partner 4.9 2.4, 10.0
Both partner & mother injured 6.4 2.9, 14.1
HHSP/HFA Targets the Right Outcomes: Depression and IPV Other Parenting
Behaviors
Parenting
Behaviors
Depressive Symptoms
Intimate Partner Violence
OR p OR p
Poor HOME Score 1.8 .05 1.2 .38
Poor NCAST 2.2 .02 1.1 .60
Neglect 1.7 .10 2.5 <.001
Assault on Esteem 2.4 <.01 2.8 <.001
Harsh Parenting 3.2 .01 2.3 <.01
What Do We Know about Home Visiting as a Preventive
Intervention? 1. Home visiting can improve outcomes, but
effects tend to be modest and variable.
2. Programs like HSP/HFA target– the right families and individuals,– at the right time,– focusing on the right outcomes
3. But even if the model seems right, desired outcomes might not be achieved.
Overall HSP/HFA Impact was Negligiblefor Most Outcomes
Hawaii Alaska
AOR p AOR p
Maternal Depression
0.97 .84 0.66 .16
Physical IPV 0.83 .19 0.82 .43
Poor HOME Score 0.87 .39 0.51 <.001
Poor NCAST Score 0.86 .29 0.79 .31
So what’s going on? Wrong model?
Type III Error Dobson and Cooke, Evaluation & Program Planning,
1980
• Unless fidelity of implementation is determined, it is not possible to determine whether negative impact is due to:
–Inadequacies in the model or
–Departures from the model
“Every system is perfectly designed to achieve exactly the results it gets.”
Donald M Berwick, M.D.Institute for Healthcare Improvement
• Let’s look at how home visiting services were provided – let’s look at coverage, duration and frequency of visits, visit content……
Percent of Families Screened, by Hospital
8984
88
64
82
67
H1 H2 H3 H4 H5 H6
% Screened
Overall(FY00-FY07) FY 05 FY06 FY07
Number of Births 153,403 18,023 18,364 19,069
Percent Screened 72% 77% 74% 76%
Percent Screen + 52% 50% 50% 52%
Percent Assessed 81% 85% 83% 83%
Percent Assess + 50% 47% 47% 47%
Percent Referred 70% 74% 82% 77%
HSP Screening & Assessment Rates
Attrition was higher than expected and there was substantial,
unintended variation across sites. (Hawaii)
0
20
40
60
80
100
120
0 4 8 12 16 20 24 28 32 36 40 44 48 52
Weeks of Age
Perc
ent
Agency BAgency AAgency C
Mean Number of Visits
Active AllAgencyFamiliesFamilies
B 22 16 A 19 11C 28 12
p <.01 <.01
Percent of Families Active
Visit Content also Differed from the Model: HVers Often Failed to Respond to
Parenting Risks
11%
21%14%14%
32%
8%
Poor Maternal MentalHealth
Domestic Violence Maternal SubstanceUse
All Families with Risk
All Families with Risk and High Dose
Programs Varied Substantially in Provision of Core Services
Hawaii Alaska
All Sites
Range All Sites
Range
Had 1st IFSP on Time 44% 28%-62% 24% 0% - 53%
Developmental Screenings 36% 23%-67% 48% 38% - 58%
Discussion of Risks
Poor Mental Health 15% 6%-30% 45% 9% - 88%
Domestic Violence 22% 13%-40% 24% 6% - 33%
Substance Use 15% 0%-33% 30% 11% - 42%
Factors forIntegrity
Model Complexity& Clarity
ImplementationSystem
ParticipantResponsiveness
Quality of Delivery
Service Integrity• Fidelity = Coverage
Duration, FrequencyVisit Content
• Competence
Component Analysis
To identify essential components of the model
Intervention
Model
Framework of Determinants of Integrity (Carroll et al., 2007)
Outcomes
Factors for Integrity•Implementation System
(Hiring, Training, Supervision, Curricula, Protocols, Monitoring, Linkage Agreements)•Family Attributes
Process
Screening & Assessment
Direct Home Visiting Services
Medical Home
Other Needed Community
Services
Family FunctioningParent Mental Health, SU, IPV
Social Support
Economic Sufficiency
Child OutcomesHealth and Development
Program Model
Outreach
Trust Building
Crisis Intervention
Case Management
Parenting Education, Role Modeling, Reinforcement
We need to understand how family attributes and the implementation system
moderate impact.
Parenting
Knowledge/Attitudes/Skill
Parent-Child Interaction
Environment for Learning
CAN
linkages
enrollment
Influence of Home Visiting Model
Complexity and Clarity (Hawaii)
Program impact was compromised …. by drift in the model itself that had arisen in taking the model to scale.
Original CAN Prevention Program
Designation as an EI Program for Children at Risk for CHCN due
to Environmental Factors
Case Plan focused on Risks that
Made Families Eligible
Parent-driven philosophy; IFSP with family as decision-makers in
setting goals & strategies
Influence of HV Model ClarityStaff and Recipient Understanding of
PAT – Similarity
• Belief that a strong relationship was important and beneficial to the parents.
– Differences
• Perception of home visitor’s expertise
• Perception of purpose of home visitor showing the child a new activity
- From Hebbeler & Gerlach-Downie, 2002
Influence of Implementation System - Training
SB6 HVers lacked basic knowledge of child development.
24
9
30
39
46
52
Social-Emotional
Language
Selected Age Too YoungSelected Correct Age RangeSelected Age Too Old
. Tandon et al. Success by 6 Evaluation, Baltimore, 2004.
Influence of Implementation System - Training
Training improved staff knowledge, at least short term.
19
24
5
9
49
30
65
39
32
46
30
52
SE-2006
SE-2004
Lang-2006
Lang-2004
Selected Age Too YoungSelected Correct Age RangeSelected Age Too Old
Comparison of Study Results with ICMQ Validation Study Results*
12 Months 24 Months• % Agreement Validation Studies 91% 89%
This study 88% 74%
• Sensitivity Validation Studies 72% 94%This study 0% 7%
• Specificity Validation Studies 92% 83%This study 90% 89%
*Bricker D and Squires J, the Effectiveness of Parental Screeningof At-Risk Infants: The Infant Monitoring Questionnaire.Overall sensitivity 4-36 months = 63%; overall specificity = 91%
The implementation system must include SKILLS TRAINING & FEEDBACK &
COACHING. Knowledge Shown
in Training Setting
Skill Shown in Training Setting
Skill Observed in Practice
Didactic & Discussion
10% 5% 0%
Skill Demonstrated in Training
30% 20% 0%
Practice/Feedback in Training
60% 60% 5%
Coaching in Practice Setting
95% 95% 95%
Joyce and Showers, 2002
RCT of Coaching to Promote Fidelity & Impact
• Statewide random assignment of home visiting teams
• Data sources: parent interviews; record review; observation of home visitors; surveys of and in-depth interviews with home visitors and supervisors.
15 Teams Stratified by
Caseloads and Retention Rates, then Randomly
Assigned to Three Study
Group
Usual HSP Services
Training in HFT + Usual Supervision
Training in HFT + Enhanced
Supervision
Influence of Participant ResponsivenessProgram outreach - most effective in a subset of
mothers.
0
20
40
60
80
100
16 32 48 64 80 96
Maternal Anxiety Score
Perc
ent
Acti
ve in P
rogra
m
Assertive OutreachRelaxed Outreach
An assertive outreach policy promoted retention of mothers with high relationship anxiety.
Home Visiting Impact Was Pronounced inMothers with High Relationship Anxiety
19%
57%
19%
31% 33%
58%
26%
68%
34% 32%
ALL AR Neither
High
Anxiety
High
Avoidance
High
Both High
Example: Depressive Symptoms
Where We Go from Here…
– Learn What Works, for Whom
– Improve Fidelity•Clarify existing models
•Build implementation system infrastructure
•Understand providers and recipients
•Build basic skills
– Enhance Home Visiting Models
Where We Are Going in Hawaii…
5-year ACF-funded project Staff training and supervision
CQI capacity
Targeting of families
Enhancements to the model
Reconciliation of funding incongruities