How to Effectively Measure and Monitor Progress on Five-Year Grants
Presented by: Aaron Lieberman (Chairman of the Board);
Andrew Davis (Chief Partnership Officer)
Region V Head Start Association | October, 2015
Agenda
• Introductions & Learning Objectives
• Understanding the Requirements of 5-Year Grant Submission
• Setting Long-Term Goals
• Ongoing Monitoring and Use of Data
• Making it Real in Your Program
• Founded in 2002
• $50 million annual revenue
• 1048 employees
• Serving 5000 children in four states
Developing powerful new approaches to data, curriculum and family services approach
• Founded in 2009• $3.2 million annual revenue• 24 employees
• Intensive partnerships with over 30 Head Start programs serving more than 36,000 children
Leveraging Acelero Learning’s Head Start innovations to impact
larger early childhood community
Key Drivers of Success
• Extensive use of data– Proprietary reporting system includes weekly MBI report with 100
different summary indicators
– Monthly MBI + report with 54 different indicators
– Approach to data served as basis for development of Shine Insight data system
• Early childhood education focus on extensive coaching, intentional curriculum, and new approach to assessment
• Family services focus on leveling with families about the achievement gap and then setting outcome-based family life practice goals with every family
Introductions: About You!
• Name / Title
• Program / Children Served
• Years in Head Start
• Have you submitted your 5-year grant application already?
• What is one thing your program is focused on tracking better this school year?
• Share Out!
Today’s Goals
Desired Learning OutcomesBy the end of the session participants will be able to:• Articulate a practical understanding of how the use
of data will play a part in both setting and tracking long-term goals
• Build quantifiable measures related to each program goal, and set benchmarks / targets over the course of five years
• Articulate how quarterly check-ins on goal-related metrics and program data can enhance planning and programmatic self-assessment practices
BUILDING AND USING DATA DASHBOARDS
What’s new in Head Start
What is a data dashboard?
How can data dashboards be helpful in your program?
• Ongoing / regular tracking of compliance measures
• Structured, focused look at outcomes data for children and families
• Clear way to highlight key measures that will help you track progress toward your program goals
Manage by… Monitoring System
Manage by Information
Compliance & Process
Indicators
ShineStat
Quarterly Self-Assessment
Manage by Outcomes
Outcomes Indicators
What Makes Good Compliance Monitoring?
• Programs need a reporting tool which measures Head Start compliance indicators. Key aspects should:
� Include information by unit that is managed to and held accountable for results (i.e. classroom, or center, or zone, etc.)
� Show trends from prior month (or time period)
� Evolve over time as you use
� Be used as both a tool to identify areas that need more support and to hold people accountable
• To be effective, information must be up to date, real-time and run every other week (or monthly, at a minimum)
Types of Data to Enter, Capture, and Monitor
• ERSEA
– Current enrollment, openings, waitlist
– Eligibility, by category
• Disabilities / Mental Health
– Active IEPs / IFSPs
– Completed or incomplete or past-due screenings (Dev and SE)
• Family Services
– Completed Strength Assessments and Family Partnership Agreements
• Health
– Completed, Past-Due, Expired Health Screenings and Immunizations
– Medical / Dental Homes
• Early Learning
– Completed Home Visits and Parent/Teacher Conferences
– Completed Assessments, by checkpoint
Our Answer: Manage By Information Report
Rows represent centers and the dark purple row shows the program average or total.
Columns are numbered sequentially for reference
Historical trend data is included at the bottom.
Conditional formatting with colors help visually highlight areas.
4
3 2
1Frontline Staff / Coordinators input real-time data into
data system
Administrator(s) runs 20+ summary reports from data system and keys data into excel
spreadsheet (MBI)
MBI is distributed to wide audience
for review
Frontline Staff / Coordinators
respond to MBI issues and follow-up
Regular MBI Process (if not built into data system)
MBI Relies on Accurate, Current Data in Your Systems
If it is not in _________ (Shine Insight, Child Plus,
PROMIS, COPA, etc.)
it did not happen!
Managing by Information Not just reporting, but acting…
• Distribution of MBI report among key leaders and governance groups increases accountability for all players
• Ideally allows leadership to focus on data –not personalities – in problem solving and tracking progress
• Ultimately it is about identifying problem areas, and then making changes to improve, when key numbers indicate concerns
Basic Outcomes Tracking: The MBO
The MBI:The Limitations
• Almost exclusively focused on process indicators (in comparison to outcomes)
• Are short-term by nature – things that change every week
• Can lead you to “miss the forest for the trees” if there are no other measurement systems in place
The MBI:The Limitations
For example…
• What if we conduct all of your educational assessments, but children are leaving our program not prepared for kindergarten?
• What if we complete our family partnership agreements on time, but our families never meet those goals?
• What if we finish all our screenings on time, but never successfully help families follow-up on health issues?
What Does Good Outcomes Monitoring Look Like
Why it is needed…
• The new five-year grants structure, aligned monitoring system and new proposed performance standards place a greater emphasis on outcomes than ever before.
• For that reason, to be a GREAT Head Start program, it is NOT ENOUGH to just meet minimum standards for compliance
• The current HS Performance Standards are a floor, not a ceiling!
Our Answer: THE MBO Report
• The MBO Report allows us to:
– Set program-wide goals for key outcomes we wantto accomplish
– Manage and measure to see if we accomplish them, and how we can improve
– Review these outcome reports quarterly throughout the year, and adjust our work accordingly
Managing by OutcomesProcess vs. Outcome
Process� Completing
assessments on time
� Completing family strengths assessments
� Completing required screenings on time
Outcome� Children mastering skills
� Families achieving goals
� Children receiving treatment to help them become healthier
Table Discussion:
What will be your biggest challenge in terms of tracking outcomes in the four areas (child, family, health, special populations)?
Why?
Goals, Measures and Targets
Ctr
1:
PF
T
Ctr
2:
MLK
Ctr
3:
Ach
iev
e
How can this translate to tracking progress on five-year goals?
5 Year Grants Preview: Why does this matter?
Goals• Broad, inspirational statements that
describe what you seek to accomplish; targets to be reached.
Objectives
• Subparts of goals that are Specific, Measureable, Attainable, Realistic and Timely (= SMART).
Impacts
• The influence or effect on a specific Head Start population (e.g. staff, children, families, communities)
Creating a dashboard for your five-year goals
• What are the impacts you are looking to measure?
• What information must you collect in order to measure them?– Who is involved?
– When is the information collected?
• What units and levels are the most meaningful? (Child, classroom, center, program, etc.)
Group Exercise
Use one of the long-term goals for your program that you brought in. Discuss with your neighbor the answers to these questions:
• What impacts are you tracking?
• What types of information and systems are involved?
• When is it meaningful to collect and report / discuss this information during the year?
• What levels are the most meaningful to display? (Classroom level, Center level, region level, etc.)
• What would your ideal “dashboard” for this goal look like?
– It’s OK to include more than one area to track!
UNDERSTANDING THE REQUIREMENTS FOR NEW 5-YEAR GRANT SUBMISSIONS
What’s new in Head Start
Five Year Funding ApplicationsThe Big Picture: Baseline Applications
Section I. Program Design and Approach to Service Delivery
– Long Range Goals, Objectives and Program Impacts (4 criteria)
– Service Delivery (14 criteria)
– Approach to School Readiness (2 criteria)
– Parent, Family and Community Engagement (PFCE) (4 criteria)
– Governance, Organizational and Management Structure, and Ongoing Oversight (10 criteria)
Section II. Budget and Budget Justification (12 criteria)
Baseline Application: Long Range Goals, Objectives and Program Impacts
Specify the expected, measurable program impacts for each year of the five year period in the following areas:
– Outcomes for children and families including school readiness, health services, support for special populations, and effective parent and family engagement
Baseline Application: What has changed?
• Original application required applicants to specify measurable program impacts aligned with the long-range goals and objectives to be achieved for each year of the grant in the following areas:
– Outcomes for children and families, including school readiness, health services, support for special populations, and effective parent and family engagement
– Program and fiscal management systems; oversight and accountability
– Enhanced community involvement and resources
– Unique community and organizational goals
• Latest application (April 2015) only suggests these areas of focus – however, they kept the frame for alignment with program goals!
DEVELOPING 5-YEAR GOALS FOR CHILD OUTCOMES
Goal Development Example
5-Year Application (and Programmatic) Priorities
Year 1
• Describeprogram approach
• Defineanticipated outcomes
• Measuresof program progress
Year 2
• Focus on progresstowards goals, outcomes, and intended impacts
Year 3
• Focus on progresstowards goals, outcomes, and intended impacts
Year 4
• Focus on progresstowards goals, outcomes, and intended impacts
Year 5
• Provide evidence of the program’s cumulative impact on the children, families, communityGather evidence to submit in Year 5.
Example: Our ECE Approach
Structured Curriculum– Content-Rich – Intentional Teaching– Provide Scaffolds
Criterion-Based Professional Development– Self-Assessment– Coaching – Professional Learning Communities
Focused Assessment– Lean and Mean Approach– Ongoing outcomes monitoring
Next step: Define Anticipated Outcomes
• Given the focus of your approach, what outcomes do you expect? Why?
• Do you have a current “baseline”?
• What measures are in place to find out if you are on track to achieving those outcomes?
Managing by OutcomesProcess vs. Outcome
Process� Completing
assessments on time
� Home visits and parent-teacher conferences held twice per year
Outcome� Children achieving school readiness goals
� Parents understand school readiness goals and are prepared to support them at home
Four Key Outcome AreasChild Outcomes
How prepared will our children be for kindergarten as a result of
our program?
School Readiness Goals
Family OutcomesHow effective will our families be at preparing their children for
success in school as a result of our program?
PFCE Outcomes
Health OutcomesHow will our children be healthier as a result of
our program?
Special PopulationsHow will special populations be positively impacted as a result of
our program?
Sample Measures and Targets for Child Outcomes
Objective: Measure:
Ensure that
transitioning
children leave
our program
prepared for
kindergarten
Assessment on School Readiness Goals
Year 1 Target:
Demonstrate
that 75% of
children
enrolled in HS
for two years
meet school
ready targets.
Year 2 Target:
Demonstrate
that 80% of
children
enrolled in HS
for two years
meet school
ready targets.
Year 3 Target:
Demonstrate
that 85% of
Children
enrolled in HS
for two years
meet school
ready targets.
Year 4 Target:
Demonstrate
that 90% of
children
enrolled in HS
for two years
meet school
ready targets.
Year 5
Evidence:
External
assessment
validates Year
4 progress
(PPVT, etc)
FCE Program Approach
Year 1
• Describe program approach
• Define anticipated outcomes
• Measures of program progress
Year 2
• Focus on progress towards goals, outcomes, and intended impacts
Year 3
• Focus on progress towards goals, outcomes, and intended impacts
Year 4
• Focus on progress towards goals, outcomes, and intended impacts
Year 5
• Provide evidence of the program’s cumulative impact on the children, families, community
Our FCE Approach
Contracting with Families to Close theAchievement Gap
– Family Engagement Contract – Achievement Gap Dialogue– Family Life Practices– Family Engagement “Report Cards”
Differentiated Caseloads and Targeted FamilySupport
– Tiered Family Self-Assessment Instrument– Family Success Rubric– Outcomes-based Goal Setting
Our FCE Approach (cont.)
Competency-based Professional Development
‒ Success Rubrics for all staff
‒ Formal monthly observations and reflective coaching for Advocates and Coordinators
‒ Monthly data driven individual and group supervision
‒ Professional Learning Communities
Table Exercise: What is YOUR approach?
Family engagement?
Family self-assessment?
Professional development?
Anticipated Outcomes
Year 1
• Describe program approach
• Define anticipated outcomes
• Measures of program progress
Year 2
• Focus on progress towards goals, outcomes, and intended impacts
Year 3
• Focus on progress towards goals, outcomes, and intended impacts
Year 4
• Focus on progress towards goals, outcomes, and intended impacts
Year 5
• Provide evidence of the program’s cumulative impact on the children, families, community
Managing by OutcomesProcess vs. Outcome
Process
� Completing family strengths assessments
Outcome
� Increased family engagement (reading activities at home, PEER cards, progress on the Family services “Roadmap”)
Four Key Outcome AreasChild Outcomes
How prepared will our children be for kindergarten as a result of
our program?
School Readiness Goals
Family OutcomesHow effective will our families be at preparing their children for
success in school as a result of our program?
PFCE Outcomes
Health OutcomesHow will our children be healthier as a result of
our program?
Special PopulationsHow will special populations be positively impacted as a result of
our program?
Our Measures and Targets for Family Outcomes
Objective: Measure:
Ensure that all
families
accomplish
their family life
practice goal
to help their
child be
prepared for
success in
kindergarten.
Family outcome goal tracking system (Shine Insight)
Year 1
Target:
70% of
families will
accomplish
their family
life practice
goal.
Year 2 Target:
80% of
families will
accomplish
their family
life practice
goal.
Year 3 Target:
90% of
families will
accomplish
their family
life practice
goal.
Year 4 Target:
100% of
families will
accomplish
their family
life practice
goal.
Year 5
Evidence:
Outside
assessment of
impact on
families
Group Exercise
� What are your expected outcomes for children ORyour expected outcomes for families based on your ECE or FCE approach? (choose one area to focus on for this exercise)
� How would you measure them?
� What would a good target be for your first year of implementation?
� Given all of these inputs – how would you shape this into language for a long-term goal?
Example Measures and Targets
Objective: Measure:
??
Year 1 Target:
?
Year 2 Target:
?
Year 3 Target:
?
Year 4 Target:
?
Year 5
Evidence:
?
Other Goal Areas
• Other outcomes goals:– Goals related to health outcomes
• Obesity• Asthma• Local / specific health concerns
– Goals related to special populations served by your program
• Other program goals: – Program and fiscal management systems; oversight and
accountability– Enhanced community involvement and resources– Unique community and organizational goals
SHORT-TERM OBJECTIVES AND QUARTERLY TRACKING SYSTEMS
“Aim at Something High”
Goals and Measures
Goal # 1 Short-term Objectives
Ensure that transitioning
children leave our program
prepared for kindergarten
• Time Spent Coaching
• Coaching Sessions / Classroom
• CLASS scores
• Days Wastage (Attendance,
enrollment gaps / churn)
�Most important measure is in the goal itself
� Rest of measures track items we feel are key
contributors to achieving that goal
Goals and Measures
Goal # 2 Short-term Objectives
Ensure that all families
accomplish their family life
practice goal to help their
child be prepared for
success in kindergarten.
• ADA, retention, absenteeism
data
• Parent report card data
(participation in parent
meetings, etc.)
�Actual progress on families achieving family goals is
key outcome
� Parent participation in required meetings/events is
key contributor
How will program data allow us to consistently track these measures?
• Coaching
• Child Outcomes
• CLASS scores
• Days wastage
Education
• Family goals and progress
• Parent participationFamily
Services
Quarterly
Dashboard for
Program Results
Key questions
Where are we now?
According to our five-
year goals, where
should we be?
What are our next
steps?
Using Five Year Goals to Drive On-Going Self-Assessment
• If you align work with your goals, you should be able to measure progress quarterly
• In addition to other compliance measures, this quarterly data is ideal to use as the basis of on-going self-assessment and planning quarterly meetings
• You can use this to END the once a year self-assessment madness
Our Answer: Shine Stat
• Three times a year presentation of key measures
• Audience: Key staff, Board and PC members
• Focuses on actual data from previously established measures
• Cuts through clutter – allows you to see forest through the trees
• Increases accountability and engagement
Shine Stat Sample Measures
Early Childhood Education
• CLASS scores
• average # of coaching cycles per classroom
• average # of coaching-of-coaches cycles per coach
Family & Community Engagement
• # of parent meetings attended
• Progress on Family Life Practice goals
58
1.21.1 1.1 1.08
1.17
0.97
0.8
1.2
1.0
1.201.15
0.851.0
0.7
0.90.79
1.03
0.74
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
September October November December January March
Average # of Coaching Cycles Completed Within 24 Hours, per
classroom
Clark
CPHS
MMHS
Sample Shine Stat
Target : >1 cycle/month
Goal 1: Increase ECE impact. Are teachers/CDs receiving regular coaching?
1.3
2.5
1.4
1.732.00
1.55
1.0
2.3 2.3
1.861.57
0.86
0.4
1.31.5
1.25 1.33
2.00
0.0
0.5
1.0
1.5
2.0
2.5
3.0
September October November December January March
Average # 1:1 Coaching Sessions, per Center Director
Clark
CPHS
MMHS
Analysis� The general expectation
for teachers and CDs is monthly coaching, differentiated by need (regardless of Center size)
� The FMR – post CLASS notification – made February and March abnormal months
� Year to Date (through March)
• Coaching has consistently met our target
59
Sample Shine Stat
Target: 50%Goal 2: Increase FCE impactAre families making progress on FLP goals?
6% 7%12%
18%
25%
37%43%
55%
19%
0%
10%
20%
30%
40%
50%
60%
Family Life Practices
Family Goals
Self-Sufficiency
Family Goals
Support for Children
Family Goals
Support for Family
Members
Family Goals
Home-Based
Family Goals
% of goals with major progress,
Winter 11-12 (left), Fall 12-13 (middle), Winter 12-13 (right)
25%
9%
35%
0%
10%
20%
30%
40%
% of FLP goals with major progress,
Winter 2012-13
CC
CP
MM
Cla
rk
Analysis� 25% of Family Life Practice
Goals have seen major progress, compared to 6% at the same time in winter 2011-2012
� Family Life Practice Goals have seen less progress than other types of goals
• 37% of Self-Sufficiency goals have resulted in major progress
• 43% of Support for Children and Family goals have resulted in major progress
• 55% of Support for Family Member goals have resulted in major progress
60
Sample Shine StatTarget: 75%
Goal 2: Increase FCE ImpactAre parents attending Achievement Gap workshops and
Center Committee Meetings?
0%10%20%30%40%50%60%70%80%90%
100%
% of families attending center events YTD
PCC
FLP
48%
20%
42% 40%
72%
32%
61% 59%
41%
4%
19%26%
60%
27%32%
44%
0%
10%
20%
30%
40%
50%
60%
70%
80%
CC CP MM ACELERO
PCC through Dec 12
PCC through Apr 13
FLP through Dec 12
FLP through Apr 13
Analysis� Our target is for 75% of
enrolled families to attend Family Life Practice workshops
� Since January, the percentage of parents who have participated in at least one workshop has increased from 41% to 60%
� Participation in Center Committee meetings has been higher, with 72% of parents having participated in at least one Center Committee meeting
� Participation in Herb Kaufman, Gates and Sunflower is >80% for FLPs and >80% for PCC
Shine Stat: How do we run these meetings?
• Poster-sized boards with data in each area posted around a room
• Groups of leadership team members physically walk around to examine information, discuss
• Structured review of each service area –
– What stood out?
– What trends came forward?
– What should our takeaways be at this point?
Documenting Planning and Follow-Up
• After each Shine Stat self-assessment and planning meeting, we will document a clear action plan
• Notes include: Next Required Actions, written in who, what, by when format
• Summarize for annual planning meeting as our “self-assessment action plan” (or “Improvement Plan”)
• Annual summary forms a basis of resubmission to RO for tracking progress on five year goals!
Management by Monitoring System: Review
Manage by Information
Compliance & Process
Indicators
ShineStat
Quarterly Self-Assessment
Manage by Outcomes
Outcomes Indicators
Questions?
Thanks for your participation today!
Please stay in touch!
• Visit us at www.shineearly.com
• Sign up for our mailing list
• Follow us on social media (Twitter @ShineEarly, connect on LinkedIn)
• Get our business cards!