Date post: | 12-Apr-2017 |
Category: |
Government & Nonprofit |
Upload: | isaac-castillo |
View: | 232 times |
Download: | 2 times |
1
Identifying Collective Impact Outcomes
Isaac Castillo - @Isaac_outcomes
Isaac D. CastilloDeputy Director
DC Promise Neighborhood Initiative@Isaac_outcomes
December 4, 2015
2
Learning Objectives
• What is Collective Impact? • What type of Collective Impact are you talking
about? • What does it take to measure outcomes in a
collective impact framework?
December 2015 Isaac Castillo - @isaac_outcomes
3
What is Collective Impact?
• As defined by FSG (the nonprofit consulting firm that invented the term)…..
“Collective Impact is the commitment of a group of actors from different sectors to a common agenda for solving a complex social problem.”
December 2015 Isaac Castillo - @Isaac_outcomes
4
Five Conditions of Collective Impact• Collective impact is more rigorous and specific than collaboration among
organizations. There are five conditions that, together, lead to meaningful results from collective impact:– Common Agenda: All participants share a vision for change that includes a common
understanding of the problem and a joint approach to solving the problem through agreed-upon actions.
– Shared Measurement: All participating organizations agree on the ways success will be measured and reported, with a short list of common indicators identified and used for learning and improvement.
– Mutually Reinforcing Activities: A diverse set of stakeholders, typically across sectors, coordinate a set of differentiated activities through a mutually reinforcing plan of action.
– Continuous Communication: All players engage in frequent and structured open communication to build trust, assure mutual objectives, and create common motivation.
– Backbone Support: An independent, funded staff dedicated to the initiative provides ongoing support by guiding the initiative’s vision and strategy, supporting aligned activities, establishing shared measurement practices, building public will, advancing policy, and mobilizing resources.
December 2015
Source: FSG Consulting website (group that developed collective impact concept)
Isaac Castillo - @Isaac_outcomes
5
But is your Collective Impact the same as mine?
• Different people / groups interpret the concept of ‘collective impact’ in many different ways.
• I have found the three most common to be: – One problem / one geographic area– ‘Franchise model’ – multiple locations– Focus on core group of individuals
December 2015 Isaac Castillo - @Isaac_outcomes
DCPNI – Isaac Castillo - @Isaac_outcomes 6
One Problem / One Geographic Area
• Most commonly discussed form of collective impact.
• Single large problem is identified for solution in one area.
• Multiple agencies and organizations (government, schools, community, nonprofits) ‘collectively’ try to solve problem.
• Attempt to measure ‘impact’ of their efforts in solving the problem.
3/17/2014
7
One Problem / One Geographic AreaExample
December 2015 Isaac Castillo - @Isaac_outcomes
Reduce Gang Crime in City
Mayor’s Office
Government Agencies
Residents
Funding Community
Nonprofit Organizations
Law Enforcement
Schools Evaluators
8
One Problem / One Geographic AreaCharacteristics
• Commonly defined goal (with indicators) that everyone agrees with.
• Multiple agencies/groups all working towards this common goal.
• Some agency/group takes lead or serves as coordinator.
• Not every agency/group contributes the same – some do more than others.
December 2015 Isaac Castillo - @Isaac_outcomes
9
Franchise Model – Multiple Locations
• Most common model (not talked about as much).
• Same program/intervention replicated by single organization in multiple locations.
• Larger organization is trying to determine the ‘impact’ across all of their sites / locations.
• The aggregated outcomes of each location is then considered ‘collective’.
December 2015 Isaac Castillo - @Isaac_outcomes
10
Franchise Model – Multiple LocationsExample
December 2015 Isaac Castillo - @Isaac_outcomes
Collective Impact=+++++
11
Franchise Model – Multiple LocationsCharacteristics
• Parent organization sets goal(s) and local organizations agree to goal(s).
• Same program / approach in lots of locations. • Parent organization responsible for quality
control / fidelity.
December 2015 Isaac Castillo - @Isaac_outcomes
12
Core Group of Individuals
• Rare – put present in some locations.• Specified group of service recipients that are
served my multiple organizations. • Service recipients are pre-identified at the
individual level. • Attempt to measure the ‘impact’ of all service
providers on the same participants.
December 2015 Isaac Castillo - @Isaac_outcomes
13
Core Group of IndividualsExample
December 2015 Isaac Castillo - @Isaac_outcomes
After-school Tutoring
Pre-Identified Youth Service Recipients
RecreationExercise Programming
Health Education College ReadinessNutrition Programming
14
Core Group of IndividualsCharacteristics
• Service providers need to agree upon common set of outcomes/indicators.
• Some group / organization needs to coordinate.
• Intent is to provide extensive services to a limited set of participants.
December 2015 Isaac Castillo - @Isaac_outcomes
15
Common Concerns for All Collective Impact Approaches
• What kind of data will you collect? – Demographic and/or socioeconomic– Service amounts or types– Outcomes or changes in conditions
• Who is responsible for reporting data (and what is their motivation)?
• Who collects the data and in what format? • Who trains people on proper data submissions? • Who analyzes and then shares/reports data and findings? • Are there consequences for not sharing data (or not sharing
data in correct format or on time)?
December 2015 Isaac Castillo - @isaac_outcomes
16
How the Simple Can Create a Challenge
• Define each data point along with acceptable answer categories– Gender – what will you ask for and what will you
report? – Race / ethnicity – are these two different concepts
to be measured, or will they be grouped? And what are you answer categories?
– Neighborhood, census tract, census block group, or some other geographic area?
December 2015 Isaac Castillo - @isaac_outcomes
17
Agreeing on an Outcome
• Should come to consensus on what the outcome is that you want to make progress towards.
• Three at a maximum – the fewer the better. • This will be the hardest part – but important to
gain consensus. • Shouldn’t get too detailed or worry too much
on how to measure it. First just agree on what you want to change.
December 2015 Isaac Castillo - @isaac_outcomes
18
Next Agree on Indicators
• Indicators are what you will measure to determine if you are making progress towards your outcomes.
• They are more detailed than outcomes and can exist at multiple levels and time periods.
December 2015 Isaac Castillo - @isaac_outcomes
19
Examples of Indicators and Outcomes
• Outcome: Youth are college and career ready– Indicator: Youth graduate from college– Indicator: Youth graduate from high school – Indicator: Youth are proficient in math and/or reading
on state assessments• Outcome: Reduce poverty in defined area– Indicator: % of adults at or below poverty line– Indicator: % of children at or below poverty line– Indicator: % of people living at 150% or less of poverty
level
December 2015 Isaac Castillo - @isaac_outcomes
20December 2015 Isaac Castillo - @Isaac_outcomes
21
How Will You Measure?
• Once you have your outcomes and indicators, you can think about how to measure change.
• Will you….– Use publicly available data? (And how regularly
available is it)?– Collect data from multiple sources (and how will
you aggregate it)? – Ask multiple people to submit data (and what is in
it for them)?
December 2015 Isaac Castillo - @isaac_outcomes
22
Kenilworth-Parkside Community
DCPNI’s Collective Impact Approach
December 2015 Isaac Castillo - @Isaac_outcomes
Youth Parents
Residents
23
DCPNI Collective ImpactCharacteristics
• DCPNI serves as coordinating agency.• Some outcomes/indicators set by DCPNI
(Dept. of Ed.), others developed with partners. • Will determine which combinations of
partners are most effective.
December 2015 Isaac Castillo - @Isaac_outcomes
24
Measuring at the Community and Individual Levels
• How do these indicators help DCPNI evaluate its work in the community? That occurs at two levels: – At the community level – we look at change that
occurs over time for community level data. – At the individual level – we look at how each child
and parent in the community improves their outcomes over time.
December 2015 Isaac Castillo - @Isaac_outcomes
25
Community Level Evaluation
• Examine community level change that occurs.• Use existing data sets (Census, ACS, Vital
Statistics, Dept. of Health, Crime, etc.) • Conduct primary data collection: – Neighborhood survey– School climate survey
• Community interviews and focus groups. • Is there a clear link to changes observed and our
interventions?
December 2015 Isaac Castillo - @Isaac_outcomes
26
Individual Level Evaluation• Each partner / program defines success in alignment
with one (or more) of DCPNI’s indicators. • Each partner articulates outcomes and methods to
measure outcomes at individual (child, student, parent) level.
• Each partner shares individual level demographic, attendance, and outcome data with DCPNI.
• DCPNI combines with other data sets (school data) to measure individual level change over time.
December 2015 Isaac Castillo - @Isaac_outcomes
27
Individual Level Evaluation Example
December 2015 Isaac Castillo - @Isaac_outcomes
Math Tutoring
Effective Education – Students Proficient in Math
Report Card Grades
Chess Instruction
Critical Thinking Skills
School
Standardized Test
28
Individual Level Evaluation Example
December 2015 Isaac Castillo - @Isaac_outcomes
Tutoring
Effective Education – Students Proficient in Math
Report Card GradesChess
Critical ThinkingSchool
Standardized Test
How Does DCPNI Fit In?
Child 1
Child 3
Child 2
29
Individual Level Evaluation Insights
• Critical part is having student / resident names (or unique identifiers).
• Need to have common outcomes / indicators.• Each partner must have adequate data
collection and measurement capacity. • Can be used to identify effective combinations
of programs.
December 2015 Isaac Castillo - @Isaac_outcomes
30
Isaac’s Contact Information
December 2015 Isaac Castillo - @Isaac_outcomes
Isaac D. CastilloDeputy Director
DC Promise Neighborhood Initiative
On Twitter: @Isaac_outcomesEmail: [email protected]