30
Data Tools for Evaluating, Presenting and Measuring Rapid Re-Housing Performance Jamie Taylor, Ph.D. Mark Johnston The Cloudburst Group
Data Tools for Evaluating, Presenting and Measuring
Rapid Re-Housing Performance
Jamie Taylor, Ph.D.
Mark Johnston
The Cloudburst Group
Learning Objectives I. Rapid Re-housing Evaluation Approach –
Using HIMS data and matched comparison groups
II. RRH Data Dashboard - Shared measurement tool to support collective approach to data-driven monitoring and evaluation
III. Establishing Local “HomeStat” cross-sector workgroup structures; presenting and analyzing data for continuous quality improvement of RRH service system
HMIS data = Gold Mine A great deal of information is
currently available for targeting local RRH resources and system improvement decisions in HMIS “gold mines”.
System-building skills are needed to illuminate HMIS data and to guide decision-maker questions and solutions Mining HMIS data is necessary to
determine which homelessness programs are most effective in preventing and ending homelessness?
Gold-mining Team!
RRH Evaluation - Necessary for Social Investment into Rapid Re-housing
Investors seek rigorous evaluation methods / mathcing
RRH Evaluation: What would have happened without RRH?
Matched groups help answer the counterfactual, what would have happened without a given intervention. Comparison groups serve as an estimate for what would have happened without RRH.
With matched groups, the difference in outcomes between treated (RRH) and control (non-RRH) groups can estimate true RRH impact.
Answering the counterfactual is critical. One reason: Very low-income families move often
Among extremely low-income families with
children, 43 percent moved into their current homes within past two years. (State of Nation’s Housing 2014)
40% of all renters with incomes below poverty had moved in the past year. Households with higher income levels do not move as often as very low income households. (American Housing Survey 2013).
How do we separate the effects RRH assistance has on housing movement into homelessness from usual housing patterns for very low income households?
RRH Evaluation Requires Seeing Process and Impact Outcomes
How do we assess what is happening
inside local RRH systems − by the aggregate and − by individual RRH program effects?
Build RRH Data Dashboards for Data-driven Planning
1) HMIS data is made visible
2) Allows for stakeholder assessment of current RRH services
3) Drives the development of system-level goals
4) Helps determine what strategies and improvements are necessary to achieve RRH system aims
5) Focuses decision-makers on RRH resources that effectively prevent and end homelessness
IOWA HomeStat Rapid Re-Housing Report
Last Update: February 20, 2015 HomeStat Presenters:
Iowa Homeless Service System Map Homeless Service System (Rapid Crisis Response) State of Iowa 2013 - 2014
Iowa RRH Providers and RRH Households
Map of RRH Providers
Two Year Total RRH households:
Aggregate RRH enrollments in 2014 a 26% increase over 2013.
RRH Program Enrollment 2013 & 2014
What might account for the fluctuation of overall enrollments in each quarter?
What might be an overall goal for increasing RRH enrollment in 2016?
What enrollment goals are set by individual providers?
4.1 3.4 4.1 2.8 5.5 3.7 3.7 4.9
35.6
1.5 0.5
8.4
21.9
% of All 2013 - 2014 RRH Households Enrolled in RRH by Provider
11
120
329
145
32
141 177
265 207
Iowa RRH Households Enrolled by Quarter
2014
Single Parents accounted for almost 1/4th of all RRH households.
Of all RRH households with a disabling condition, 46% had a mental health condition, 20% had a physically disabling condition and 24% had both mental health and physical disabilities.
Why is it important to know how many TANF families receive RRH assistance?
How do frontline providers define “disabling condition” ?
RRH Household Characteristics 2013 & 2014
• 41% of all RRH households were Black
• 50% of all RRH households were White
The majority of RRH returned households had returned within 6 months (71%).
12% of 2014 households had not yet exited RRH.
What is expected goal for % return to homelessness?
What might be affecting the variation seen in Return to Homelessness rates across providers?
How are these Return results influenced by entry dates into RRH?
RRH Return to Homelessness 2013 & 2014
5%
48%
2%
34%
Percent of Return to Homelessness RRH Households by Provider
Return to homelessness data tracked by program entry. All entries occurred in 2013 or 2014. HUD System Performance Measures directly impacted by RRH effectiveness: 1. Length of time persons remain homeless; 2. Extent to which persons who exit homelessness to permanent housing destinations return to homelessness
Are RRH and TH serving similar populations? What is average cost of TH vs. RRH households?
How can targeting of RRH vs. TH resources be evaluated?
Rate of Return to Homelessness by Program
19%
10%
9%
10%
Emergency Shelter
Transitional Housing
PSH
Rapid Re-Housing
Rate of Return to Homelessness by Program in 2013 & 2014
RRH System Assessment - Core Components
Data Quality Issue: Length of time in homelessness
Length of time in homelessness data could not be used in RRH analysis. It is suspected that most households with 0 days came from RRH outreach to households at imminent risk of homelessness.
Why is this data element critical to analysis of RRH impact?
How can data quality for “Length of time in homelessness” be improved?
Number Days Homeless before RRH
HMIS data showing % of all RRH households
0 Days 76%
< 30 Days 7%
30 – 89 Days 4%
90 – 179 Days 2%
6 Mo. – 1 Yr. 4%
1 Yr. – 2 Yrs. 4%
> 2 Yrs 2%
$2,500 / month
Data Quality Issue: Monthly Income
Monthly income at entry could not be used in RRH analysis. In chart, all blue dots above $2,500 level represent households with income >$2,500, a serious data issue.
Why is the income data element so critical to analysis of RRH impact?
How can data quality for household monthly income at entry be improved?
How can overall RRH data quality checks be instituted for frontline staff?
Monthly Household income
RRH Impact Evaluation: Iowa’s 2011 – 2012 HMIS data was analyzed using a statistical matching method*. The risk of return to homelessness for RRH households was compared to similar households who did not receive RRH.
*Propensity score match creates balanced comparison groups, ensuring household HMIS characteristics are similar in both groups to effectively estimate treatment effects of RRH.
RRH Comparative Impact Analysis
All households enrolled in RRH 2011 / 2012 were compared to similar households who did not receive RRH from same years. Return to homelessness analyzed 10/2014
What would have happened to RRH households if they
had not received RRH?
Rapid Re-housing significantly decreases the likelihood of a
return to homelessness. 7.1%
11.7%
RRH Households Non-RRH Households
Return to Homelessness Results
Why do Social Investors require rigorous matching evaluation results?
How can we use this evidence for expanded RRH funding?
Improvement Strategies - RRH Funding
50%
10% 0
10%
5%
20%
5% ESG
CDBG
TANF
TBRA-HOME
Foundations
State RRH
City RRH
Iowa RRH
Funding Sources 2013-2014
These funders and percentages are hypothetical for Iowa. Actual RRH funding sources are currently being calculated for each RRH program.
Does RRH Evaluation Impact Results lead to need for additional RRH funding?
What overall goal can be set for increasing RRH funding sources?
What RRH funding goals can be set by individual funders?
How can funding goals be monitored on RRH data dashboard?
Data Dashboard supports Collective Impact Approach
Source: F. Hanleybrown, J. Karmer (2012) Channeling Change: Making Collective Impact Work. Stanford Social Innovation Review
Collective Impact requires using data to take ownership / make decisions RRH Evaluation and Program Monitoring
Data Dashboards help drive system change decisions to:
− House more people − Improve outcomes for people served − Maximize use of existing funding − Attract additional funding with performance outcomes
(Federal, State, local) − Unify local stakeholders around neutral evidence
Collective Impact National Example: HUDStat
HUDStat encompasses a coordinated, integrated leadership team meeting quarterly to analyze data on shared performance goals; assessing progress and making cross-system course corrections to achieve shared aims.
Collective Impact Local Example: Iowa’s HomeStat
Steps to establish local HomeStat:
1. Form cross-sector decision-making group (RRH Providers, Funders, HMIS, Governments, NonProfits, homeless)
2. Identify shared agenda, set shared goals to accomplish
3. Use data to analyze and prepare agenda, create data dashboard (i.e. Iowa)
4. HomeStat group meets regularly to present, listen, discuss and decide issues
5. Leave with clear direction on next steps
Collective Impact: Use Data Dashboard to Set RRH Goals
NEED to formulate clear and compelling goals for RRH with cross-sector leadership / decision-making group
HomeStat Data dashboard process links RRH program outcomes to governance and action for RRH improvement goals
HUD System Performance Goal: To decrease annual return to homelessness rates
Analyze and compare local or state system outcomes to national return to homeless rates
Compare costs of local RRH to local TH projects
Sample Goal: Expand RRH services to additional 500 households/year
With HomeStat, Stakeholders can decide whether to:
Identify specific areas needing more information
Assign follow-up tasks, i.e. assessing need to improve HMIS data quality
Assign staff to identify other possible sources of RRH funding. i.e. TANF, HOME, etc
Identify topic(s) to be taken up at next session, i.e. assess targeting of TH and compare TH to RRH clients
Determine why some RRH programs have much higher return to homelessness rates that others
The majority of RRH returned households had returned within 6 months (71%).
12% of 2014 households had not yet exited RRH.
What is expected goal for % return to homelessness?
What might be affecting the variation seen in Return to Homelessness rates across providers?
How are these Return results influenced by entry dates into RRH?
RRH Return to Homelessness 2013 & 2014
5%
48%
2%
34%
Percent of Return to Homelessness RRH Households by Provider
Collective Impact Decision-making What HomeStat Is…a culture of evidence, building capacities to respond to data, not just report to HUD
Collective knowledge framing to address issues early-on
Forum to shhare local successes and best practices
Established relentless focus on data quality and problem-solving
What HomeStat is NOT.…
A “show-and-tell” that avoids the challenging issue
Focused ONLY on what is not working
A process that lasts only one or two sessions
WouldData Dashboards and HomeStat improve your system planning?
Thank you! Contact us for additional guidance on Local RRH Evaluation
Jamie and Mark
Cloudburst Consulting Group
Contact Jamie Taylor
Phone # 860-716-7392
