21
What the NCWWI Evaluators have been up to… A love story Changing . . . Leading . . . Learning . . .
| Date post: | 13-Dec-2015 |
| Category: |
Documents |
| Upload: | miles-manning |
| View: | 212 times |
| Download: | 0 times |
What the NCWWI Evaluators have been up to…
A love story
Changing . . .
Leading . . .
Learning . . .
NCWWI Evaluation Team Updates | February 10, 2015
A Service of the Children’s Bureau
Evaluation Team
Anna de GuzmanResearch Coordinator for [email protected]
Robin LeakeLead [email protected]
Salvador ArmendarizResearch Coordinator for LAS, LINKD (webinars), and [email protected]
Shauna RienksResearch [email protected]
Amy GrenierResearch Coordinator for LAMM and [email protected]
Kate Lawrence, PhDUniversity at [email protected]
Ann ObermannPh.D. Student [email protected]
2
Still finishing NCWWI 1• Wrapped up the Leadership Academy
evaluation and prepared summary reports– 12 month surveys for 64 LAS participants– 12 month surveys for 35 LAMM participants– Summary reports completed and posted
online
• Traineeships– Last cohort of 36 students from 7 schools– Final report
NCWWI Evaluation Team Updates | February 10, 2015
A Service of the Children’s Bureau 3
NCWWI Evaluation Team Updates | February 10, 2015
A Service of the Children’s Bureau
LADD Evaluation Question Focus
• Working in partnership• Improvement of individual leadership
competencies• Implementing systems change• Implementing systems change in the
WE sites
4
LADD
•360 Feedback with 6 Deans and 7 Directors•Coaching Notes•Interviews in late spring, 2015•Analysis of jurisdiction change initiatives & alignment (flow down) in WE sites
NCWWI Evaluation Team Updates | February 10, 2015
A Service of the Children’s Bureau 5
NCWWI Evaluation Team Updates | February 10, 2015
A Service of the Children’s Bureau
UP Evaluation Questions Focus
• Implementation of innovative strategies for:o Child welfare educationo Workforce retention strategies
• University/Agency partnerships to support transition from school to worko Impact of facilitator support
• Support from LADD • Stronger outcomes in WE sites?
6
University Partnerships
Quantitative Data:• Baseline SSI from 63 students across 12 programs (93%)
Qualitative Data:• Facilitator notes, Semi-annual reports• Review of local workforce change efforts• Annual SSI spring/summer – Inclusion of NCWWI 1 graduates
• Interviews in spring of 2015 with PIs, agency partners, facilitators, students, and field instructors
• Study of programs’ field placement competency assessments
NCWWI Evaluation Team Updates | February 10, 2015
A Service of the Children’s Bureau 7
NCWWI Evaluation Team Updates | February 10, 2015
A Service of the Children’s Bureau
Organizational Intervention Evaluation Question Focus
• Implementation of Org Intervention• Changes in the health of the
workforce to drive implementation of CIs
• Value-added of Org Intervention• Improved readiness of the agency to
implement systems change
8
NCWWI Evaluation Team Updates | February 10, 2015
A Service of the Children’s Bureau
Organizational Intervention
COHA:o Baseline and 3-yr follow-up for all WE sites• Statewide Comparison group in Missouri
6-month surveys for site teamso Readiness to implement systems changeo 6 and 12 month satisfaction and Progress
Towards Goals surveysReview of implementation plans & facilitator notes
9
NCWWI Evaluation Team Updates | February 10, 2015
A Service of the Children’s Bureau
LAMM & LAS Evaluation focus
• Effective and efficient strategies for developing leadership skills
• Impact of integrated and focused leadership development
• Impact of additional supports for Managers (360 feedback and coaching)
10
LAMM and LAS
•Competency and TOL Surveys •Case studies with selected Managers and Supervisors (4 per site)o Interviews with participants and supervisors,
in-depth study of change initiatives
•Analysis of Change Initiatives•Review of LASLN and Coaching notes
NCWWI Evaluation Team Updates | February 10, 2015
A Service of the Children’s Bureau 11
Upcoming Interviews Schedule
NCWWI Evaluation Team Updates | February 10, 2015
A Service of the Children’s Bureau
Who? How Selected? When?
LAMM Participants (SF Cohort)
Random selection (5), 1-2 for case study
July 2015
LADD Deans and Directors All March-April 2015
LAS Participants (SF Cohort) Random selection (5), 1-2 for case study
July 2015
LAS State Coordinator (SF Cohort)
All March 2015
UP Principal Investigators and Agency Partners
All (except Arizona and Case Western Reserve)
March-April 2015
UP Facilitators All early May 2015
UP Field Instructors Random selection from each school
March-April 2015
UP Students Random selection (10) March-April 2015
UP Graduates Random selection (10) June-July 2015
Org Intervention COHA Interviews
By invitation, by job position June 8-12, 2015
12
WE Site: San Francisco
• COHA in summer of 2014• 339 surveys, response rate of
83%• 36 interviews and focus groups
across Seneca and HSA• Reports and dissemination of
findings
13
NCWWI Evaluation Team Updates | February 10, 2015
A Service of the Children’s Bureau
WE Site: San Francisco
LAMM:–Baseline surveys from 18 managers,
response rate of 100%–Post-training surveys from 15 managers–13 managers participated in 360
Feedback–Selection of managers for case studies
(2 from Seneca, 2 from HSA)• Case study interviews begin in Spring
14
NCWWI Evaluation Team Updates | February 10, 2015
A Service of the Children’s Bureau
WE Site: San Francisco
LAS:–Baseline surveys from 16 supervisors
response rate of 100%– LASLN Facilitation notes–SF plan to support implementation of
Supervisor Change Initiatives–Selection of supervisors for case
studies (2 from Seneca, 2 from HSA)• Case study interviews begin in Spring
15
NCWWI Evaluation Team Updates | February 10, 2015
A Service of the Children’s Bureau
Workforce Snapshot: San Francisco HSA
• Highly educated, highly paid, stable workforce• Underperforming
• Unclear roles and practice standards
• Low morale• CalSWEC• Resource-rich community• Heavy union involvement
16
NCWWI Evaluation Team Updates | February 10, 2015
A Service of the Children’s Bureau
WE Site: Missouri
• COHA in January of 2015–307 surveys, response rate of 86%–35 interviews and 17 focus groups
across four counties–Survey administered statewide to
more than 1,500 staff• 38% statewide
–Reports and dissemination of findings in May, 2015
17
NCWWI Evaluation Team Updates | February 10, 2015
A Service of the Children’s Bureau
Workforce Snapshot: Missouri Children’s Division• University-Agency partnerships getting
stronger• Workforce with few social workers• Poor training system• Lowest pay in the nation• High turnover• No practice model• 4 unique sites (morale, community
resources, leadership)
18
NCWWI Evaluation Team Updates | February 10, 2015
A Service of the Children’s Bureau
NCWWI Evaluation Team Updates | February 10, 2015
A Service of the Children’s Bureau
What have we learned so far about the 360 Feedback?• Supports the LAMM
o Aligns with the Strength Finder
• Incongruity between self and reporter ratings• Reports plus coaching is the RIGHT approach• Value added for LADD?
o Challenged by lack of focus on individual
Leadership skills
19
What have we learned so far about the COHA?
WDF and COHA fit together like a hand in a glove
COHA is FOUNDATIONAL to all of the WE activities
Need to find a way to better integrate the UPs
Need to include community partners/providers
20
NCWWI Evaluation Team Updates | February 10, 2015
A Service of the Children’s Bureau
What are we still figuring out?
• How to evaluate the alignment of the Change Initiatives – Focused on developing structured
approaches to developing and documenting CIs across the program areas
• How/whether to push for standardized approaches to competency assessment of traineeship students in field placements
• Final design for the Case Studies21
NCWWI Evaluation Team Updates | February 10, 2015
A Service of the Children’s Bureau
