Using Data and Evidence to Strengthen Services for Homeless Youth at Good Shepherd Services Beyond Housing 2014: A National Conversation on Child Homelessness and Poverty January 16, 2014
Panelists: Elizabeth Garcia, Program Director, Chelsea Foyer/Edwin Gould/Street Outreach Adrian Rodriguez, Program Manager, Chelsea Foyer Whitney Welshimer, Program Analyst, Program Evaluation and Planning Department Miranda Yates, Director, Program Evaluation and Planning Department
AGENDA
o Good Shepherd’s Approach to Program Evaluation and Planning
o Overview Chelsea Foyer Program
o Steps in Using Data for Strengthening Outcomes
o Q & A Panel
2
3
WHO WE ARE
Good Shepherd Services is a leading youth and family development and educational support agency in New York City. Focusing on under resourced communities in Brooklyn and the Bronx, we provide a broad array of individual, family and school-based services that give young people and families the opportunity to take ownership of their future, making a difference today and for the next generation.
WHAT WE DO
26,037 participants were served through
81 programs operated across
3 boroughs (Brooklyn, Bronx, Manhattan).
4
As a multi-service agency, we operate two networks of community-based youth development, education, youth justice, and family service programs in Brooklyn and the Bronx; group homes for adolescents; supportive housing for young adults; foster care and adoption services; and an in-service professional training program.
5
HOW WE WORK
We surround youth and their families with a variety of services that keep youth connected to family, school, and the community. o Youth and Family Development Approach o Commitment to Community o Partnership/Shared Resources
6
We are a Learning Organization committed to continuous quality improvement.
Across all of Good Shepherd Services’ programs, we assess positive impact by focusing on three core Development Outcomes: o Safety
o Belonging
o Skill-building
APPROACH TO EVALUATION
7 Source: Satterfield, Spring, Brownson, Mullen, Newhouse, Walker, & Whitlock (2009)
APPROACH TO EVALUATION Transdisciplinary Model of Evidence-Based Practice
CHELSEA FOYER Program Overview
o Opened in 2004. Based on Foyer model developed in the UK
o Provides 40 homeless, runaway, and foster care youth, ages 18-25, with supported transitional housing
o Funding from multiple public & private contracts
o Residents participate in a personalized program of services for up to 24 months
o Services infused with Good Shepherd Services’ signature strength-based youth and family development practices
8
o Case Management Services o Life-Skills Development o Workforce Development o Community Building o Housing and Aftercare Services
o Rigorous Application Process o Contract and Action Plan o Limited Structure/High Expectations o On-Site Support Services o Program Fee o Workforce Development Culture
CHELSEA FOYER
Key Program Components On-Site Support Services
A. DEFINE Program Planning
B. MEASURE Data Collection & Management
C. LEARN Analysis & Reporting
D. IMPROVE Using Findings
and Insights
10 Source: Adapted from Eckart-Queenan & Forti (2011)
PERFORMANCE MANAGEMENT LIFECYCLE
11
LOGIC MODEL
12
PROGRAM OUTCOMES
13
OUTCOMES PROGRESS REPORT
14
OUTCOMES PROGRESS REPORT
15
USING DATA TO STRENGTHEN OUTCOMES: STEPS
1. Identify Priority Outcome to Strengthen
2. Deepen Understanding of this Outcome
3. Develop Strategy to Strengthen Results
4. Implement New Strategy
5. Re-Evaluate
1. Identify Priority Outcome to Strengthen
16
USING DATA TO STRENGTHEN OUTCOMES
41%
33% 31%
0%
20%
40%
60%
80%
100%
FY11 (n=68) FY12 (n=74) FY13 (n=69) %
Enr
olle
d Active Year
Enrolled in Educational/Vocational Program FY11-13
FY 2013 Priority Outcome: Employment FY 2014 Priority Outcome: Education
43% 43% 56%
0%
20%
40%
60%
80%
100%
FY10 (n=35) FY11 (n=30) FY12 (n=36) % E
mpl
oyed
at D
isch
arge
Discharge Year
Employment at Discharge FY10-12
2. Deepen Understanding of this Outcome
17
USING DATA TO STRENGTHEN OUTCOMES
• Performance trends across time • Performance variation by demographic characteristics • Performance variation by prior experience
3. Develop Strategy to Strengthen Outcome Performance
18
USING DATA TO STRENGTHEN OUTCOMES
FY 2013 Priority Outcome: Employment
Research evidence:
Practitioner expertise:
-WORC Model - Literature on workforce development
-Participant Satisfaction Survey -Internal Data
- Internal (Case Managers, Vocational Specialist) - External (Mentoring USA, Reciprocity Foundation)
Participant needs and values:
Decision Making
3. Develop Strategy to Strengthen Outcome Performance
19
USING DATA TO STRENGTHEN OUTCOMES
FY 2014 Priority Outcome: Education
- Case Managers, Director of Post-Secondary Initiatives - Cross-Agency Workgroup
Practitioner Expertise:
Research evidence:
- National Clearinghouse on Families and Youth - Community College Research Center
- Participant Satisfaction Survey -Internal Data
Participant needs and values:
Decision Making
3. Develop Strategy to Strengthen Outcome Performance:
20
USING DATA TO STRENGTHEN OUTCOMES
FY 2013 Priority Outcome: Employment
Youth Employed at Discharge Experience More:
• Workshops
• Internships/Fellowships
• Employment during first 30 days at Foyer
FY 2014 Priority Outcome: Education
Youth Enrolled in School Are More Likely To:
• Be employed at discharge
• Make a program payment
• Save more money by discharge
Using Internal Data to Understand Participant Success
4. Implement New Strategy
21
USING DATA TO STRENGTHEN OUTCOMES
FY 2013 Priority Outcome: Employment “Stepping Stones to Success”
FY 2014 Priority Outcome: Education
“Future Focus Initiative” In Progress…
• Assessment Process • Comprehensive WORC Assessments • Meetings with Vocational Specialist
• Skill-Building Workshops • Career Club (“Express to Success”) • Financial Literacy • Project Voice (new) • Mentoring USA (new)
• Career Exploration • Reciprocity Foundation Programming
• Supported Experiential Learning
• Expanded Internship Opportunities
• Training on Education-Specific Pathways and Options
• Creating Cross-Disciplinary Workgroup
• Identifying Priorities Across an Educational Continuum
• Establishing Programmatic Curriculum and Processes
• Defining Multi-Phase Process with Milestones
5. Re-Evaluate
22
USING DATA TO STRENGTHEN OUTCOMES
FY 2013 Priority Outcome: Employment FY 2014 Priority Outcome: Education
TBD 43% 43% 56%
66%
74%
0%
20%
40%
60%
80%
100%
FY10 (n=35)
FY11 (n=30)
FY12 (n=36)
FY13 (n=29)
FY14 (n=19) %
Em
ploy
ed a
t Dis
char
ge
Discharge Year
Employment at Discharge FY 10-14
23
10 VALUES GUIDING THE PROGRAM EVALUATION & PLANNING PROCESS
Mission-Driven
Aspirational
Partnered
Learning-oriented
Strategic
Evidence-informed
Iterative
Authentically empowering
Tailored
Flexible
PANEL CONVERSATION
24
•Dworsky, A. (2010). Supporting Homeless Youth During the Transition to Adulthood: Housing Based Independent Living Programs Retrieved from http://www.highbeam.com/doc/1G1-224934236.html
•Eckhart-Queenan, J. & Forti, M. (2011). Measurement as Learning: What Nonprofit CEOs, Board Members, and Philanthropists Need to Know to Keep Improving. Boston: MA: The Bridgespan Group. Retrieved from http://www.bridgespan.org/measurement-as-learning.aspx
•Good Shepherd Services (2012). Good Practice Produces Good Outcomes: Good Shepherd Services’ Commitment to Evidence-Based Practice. Retrieved from http://www.goodshepherds.org/about/results.html •Kirk, A. Visualising Data blog http://www.visualisingdata.com •Morton, M.M. (2012). Applying Evidence-Based Practice to Runaway and Homeless Youth Services. Administration for Children & Families. Retrieved from http://www.rhyttac.ou.edu/webinars/topical/968-applying-evidence-based-practice-to-runaway-and-homeless-youth-services
•Patton, M. Q. (2013). Utilization-Focused Evaluation Checklist. Retrieved from http://www.wmich.edu/evalctr/wp-content/uploads/2013/03/UFE_checklist_2013.pdf •Satterfield, J.M., Spring, B., Brownson, R.C., Mullen, E.J., Newhouse, R.P., Walker, B.B., & Whitlock, E.P. (2009). Toward a transdisciplinary model of evidence-based practice. The Milbank Quarterly, 87 (2), 368-390. Retrieved from http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2698591/?tool=pubmed
•The Urban Institute (2003). Keys to Outcome Management. Washington, DC: Urban Institute. Retrieved from http://www.urban.org/UploadedPDF/310776_KeySteps.pdf
•W. K. Kellogg Foundation (2004). Logic Model Development Guide: Using Logic Models to Bring Together Planning, Evaluation, and Action. Battlecreek, MI: Author. Retrieved from http://www.wisconsin.edu/edi/grants/Kellogg_Logic_Model.pdf
RESOURCES
25
THANK YOU!
For more information about Good Shepherd Services, please visit www.goodshepherds.org Contact: Elizabeth Garcia, LMSW Program Director, Chelsea Foyer/Edwin Gould /Street Outreach [email protected] Miranda Yates, Ph.D. Director of Program Evaluation and Planning [email protected]
26