Results-Based Accountability
Using data to change policy and
program practice
Joseph McLaughlinResearch and
Evaluation Director
Lindsey LathropAssistant Director
Success Boston Initiative
A case study on measuring
systemic and organizational
impact
What is Success Boston?
• Success Boston is a cross-sector collaboration between The Boston Foundation, City of Boston, Boston Public Schools (BPS), nonprofit organizations, and higher education institutions (38 schools)
• The initiative’s work is organized along a “Getting Ready, Getting In, and Getting Through” framework
• Common goal to increase the number of BPS graduates who earn 2 or 4-year degrees, particularly among low income students of color
Graduation Rate Outcomes and Goals
• 2008 Study- “Getting to the Finish Line” revealed that only 35% of BPS college enrollees from the Class of 2000 graduated by 2007- seven years after high school.
• Success Boston initiative launched with two specific goals:
– Increase the graduation rate for the Class of 2009 to 52% (150% of Class of 2000)
– Double the graduation rate for the Class of 2011 (200% of Class of 2000)
• Transition coaching program piloted with the Class of 2009
Performance Measures(National Student Clearinghouse ‘NSC’ is primary data source)
• Enrollment rates
• Remedial course-taking rates
• Persistence rates
• Graduation rates
• College coaching impact measures (quasi-experimental studies)
*All measures are disaggregated for demographic and schooling subgroups
Cumulative College Enrollment Rates,
Seven Years after Graduation, NSC Data
67.2
73.8
77.0
60
62
64
66
68
70
72
74
76
78
80
2000 2003 2005
Pe
r C
en
t
Class
Six Year Graduation Rates of College Enrollees Who Enrolled in College in First Year after Graduation
40.6
46.547.4
49.2
30
32
34
36
38
40
42
44
46
48
50
2000 2003 2005 2006
Pe
r C
en
t
Class
One-Year College Persistence Rates,All and by Type of College Attended (in percent)
Graduating Class
(A)
AllColleges
(B)
Two YearCollege
(C)
Four YearCollege
2001 78.1 55.4 86.5
2003 81.6 61.0 87.5
2007 81.3 61.3 87.3
2008 82.5 66.5 87.5
Percentage Point Change +4.4 +11.1 +1.0
One- and Two-Year Persistence Rates of Class of 2009
BPS Graduates at Seven Selected Colleges, by Success
Boston Coaching Participation
Enrolled in Success Boston
Not in SuccessBoston Difference
Group
1-Yr Persistence
Rate
2-Yr Persistence
Rate
1-Yr Persistence
Rate
2-Yr Persistence
Rate
1-Yr Persistence
Rate
2-Yr Persistence
Rate N
All 86.4 73.4 66.0 49.9 +20.4 +23.5 711
Male 83.3 66.7 62.5 46.6 +20.9 +20.1 343
Female 88.1 77.1 70.0 53.6 +18.1 +23.5 368
Black 90.3 77.4 64.8 49.6 +25.6 +27.8 306
Hispanic 82.6 66.3 58.6 42.7 +24.0 +23.6 243
Estimated Percentage Point Impacts of Participation in
Success Boston Programs on the One Year and Two Year
Persistence Rates of BPS Graduates from the Class of 2009,
All and by Gender
Group
(A)
One YearPersistence
Rate
(B)
Sig.Level
(C)
Two YearPersistence
Rate
(D)
Sig.Level
(E)
Three YearPersistence
Rate
(F)
Sig.Level
All 16.7 .01 15.6 .01 16.3 .01
Men 18.6 .01 16.3 .05 11.1 .10
Women 14.8 .01 15.9 .01 18.8 .01
Black 21.9 .01 18.9 .01 22.0 .05
Hispanic 17.0 .01 13.7 .05 15.4 .05
10
Work with what
you have.
Since 2011, 700+
people have been
trained in RBA in
Vermont.
What is Results-Based Accountability?
A framework for:
• evaluation,
• planning, and
• collaboration
...that is simple, common sense and useful.
RBA uses data to:
• inform decisions,
• challenge assumptions, and
• explain actions to stakeholders.
From Ends Means
From Talk Action
Result or OutcomeWhat do we want?
Indicator or BenchmarkHow we will recognize it?
What will it take to get there?
Performance MeasureHow will we know we made an impact?
Pop
ula
tio
nPe
rfo
rman
ce /
P
rogr
am
ENDS
MEANS
RBA’s 2 Levels of Accountability
Population (All of Vermont):• Vermont has a prosperous economy.
• Vermonters are healthy.
• Vermont’s children and young people achieve their
potential.
(E) Youth successfully transition to adulthood.
Performance (What we can be held to):• % of youth with positive skills acquired and changed
behavior
• % of students know about and can access community
resources related to careers and college
• % of students who are prepared for employment or post-
secondary education after completing a Navicate program
7 Qs for Population Accountability
1. What are the quality of life conditions we want for the
children, adults, and families who live in our
community?
2. What would these conditions look like if we could see
them?
3. How can we measure these conditions?
4. How are we doing on the most important of these
measures?
5. Who are the partners that have a role to play in doing
better?
6. What works to do better, including no-cost and low- cost
ideas?
7. What do we propose to do?
7 Qs for Performance Accountability
1. Who are our customers?
2. How can we measure if our customers are better off?
3. How can we measure if we are delivering services well?
4. How are we doing on the most important of these
measures?
5. Who are the partners that have a role to play in doing
better?
6. What works to do better, including no-cost and low- cost
ideas?
7. What do we propose to do?
Navicate Performance Examples
Result or Outcome• All Vermont youth are engaged, inspired, and equipped to achieve
their career and educational aspirations.
• All members of the Vermont community value and nurture youth in the
process, thus assuring the educational, economic, and social vitality of
the whole community.
Indicator or BenchmarkFor our students: Rate of high school graduation, percent pursuing post-
secondary education, teen unemployment rate, college persistence rate,
percent ready for careers
By Program: Performance Measures:Three Questions:
1. How much did we do?
2. How well did we do?
3. Is anyone better off?
Program Performance Measures
% of youth with positive skills acquired and
changed behavior
% of students know about and can access
community resources related to careers and
college
% of students who are prepared for employment or
post-secondary education after completing a
Navicate program
Is anyone better off? = Why we do our work
BEFORECollegeConnections
Disagree
AFTERMany more agree!
Contributionrelationship
Appropriateresponsibility
THE LINKAGE Between POPULATION and PERFORMANCE
POPULATION ACCOUNTABILITY
Youth are aware of and can find jobs.Rate of teen employment, rate of internships
Businesses are engaged in work-based learning. Gain of # of employers offering WBL opportunities to
youth
Youth are Pursuing Post-Secondary EducationPercent students pursuing post-secondary education,
rate of second year success
# ofstudents engaged
% of students successfully completing internships
# / % of students that know which school & community resources can help them
PERFORMANCE ACCOUNTABILITY
High School Internship Program
POPULATIONRESULTS
CUSTOMERRESULTS
Alignmentof measures
Comparing your performance
1. To ourselves -- can we do better than
our own history?
2. To others -- when it is a fair apples to
apples comparison
3. To standards -- when we know what
good performance is
Overview: Population and Performance
Accountability
All Youth in Vermont
WHOLE POPULATION
OUTCOME: Youth successfully transition
to adulthood
INDICATORSStudents have exposure and
access to real life and life-long learning
People involved in Navicate PERFORMANCE MEASURE
How much did we do?How well did we do it?Is anyone better off?
(i.e. Dual Enrollment,Internships,
Employers involved,Changed student skills/behaviors,VT youth engaged in school and
learning about themselves through careers and college
Navicate’s
Contribution
Means
End
RBA in a nutshell
2 levels of accountability
3 performance measures
7 accountability questions
from ends to means
Work with the data you
ALREADY have. Start
NOW!
Navicate’s main lesson:
How much did we do?
# of customers served
(by customer characteristic)
# activities
(by type of activity)
How well did we do it?
(% common measures)
workload ratio, staff turnover rate, staff morale,
percent of staff fully trained, worker safety, unit cost, customer satisfaction)
% activity-specific measures:
percent of actions timely and correct
percent of customers completing the activity
percent of actions meeting standards
IS ANYONE BETTER OFF? (# and %)
Skills and Knowledge
Attitude/Opinion
Behavior
Circumstance
EFFO
RT
EFFE
CT
QUANTITY QUALITY
Quantity (How Much We Do)
How much service did we
deliver?
Quality (How Well We Do It)
How well did we do it?
EFFORT
• 183 high school students served
• 204 employers engaged in
internships, tours, job-shadows,
career presentations etc.
• 4 high schools
• 1 alternative program providers
• 143 internships completed
• 87% successfully complete internships
(71 of 82 in 2013)
• 5% increase out of school youth served
• 8% increase internships completed (132
in 2012, 143 in 2013)
• 19 % increase of employers
(100 in 2012, 119 in 2013)
• 6% did not complete an internship
(9 of 143 in 2013)
RBA Grid
TIPS Internship Program – 1 yr. of data
Activity Time!
In your team, 1. Decide on one program to focus on2. Answer the three RBA questions:
- How much do we do?- How well do we do it?- Is anyone better off?
3. Fill in 3 parts of the blank RBA grid for that program.
Don’t get hung up over which category things go in – just get it on the paper and then discuss how it should be reported or which of the questions it answers.
Database
Grant language
Marketing
Performance Review of Organizational Health
Strategic Planning
Board Development
Staff Professional Development
A common language on how
to talk about our impact!
We even
made an
infographic.