Are we making a difference? Noticing Progress and Evaluating
Community-led
Action, Impact and Change Here are some ideas, tools and frameworks
that started sprouting in our work over 2015 - 2016 with
Good Cents Porirua, Great Start Taita and Tiakina Ttou Tamariki.
These ideas have continued to grow as we learn with each other
about noticing and communicating the difference CLD is making.
Community-led development (CLD) is not a model or service - it is
an approach based on five key practice principles which can make a
significant contribution to community transformation and
achievement of local aspirations. To be effective in locally-led
change, “what you do” and the “how you do it” are equally
important. So we must pay attention to who is involved and in what
roles, what is going on and how well we are living CLD principles,
not just our results. We need to keep noticing what is emerging,
and what lies behind the results and changes observed. We expand a
“what’s changed, for who and by how much?” lens to intentionally
notice and assess the broader factors that impact on/are impacted
by locally-led action processes. Taking a Developmental Approach to
Evaluation Traditional approaches to evaluation have generally
involved a linear process of review after action has been
taken:
Plan Act Evaluate
Complex community issues however require flexible and adaptive
approaches to project design, implementation and evaluation. We
need to assess what does/doesn’t work as action happens, and build
this into thinking and action in real time. For this purpose,
developmental evaluation1 approaches are really useful in
community-led journeys as they involve a continuous and
simultaneous learning process of:
In our experience, measuring and monitoring CLD impact and change
is a 'best fit' process. Each community and CLD initiative will
need to ‘pick and mix’ from a range of evaluative approaches, tools
and measures and decide what’s most relevant and useful for their
particular local context, aspirations and budget. We have provided
our own simple planning and evaluation tool. Some of
1 Developmental evaluation has been championed internationally by
Michael Quinn Patton, see
http://betterevaluation.org/plan/approach/developmental_evaluation
and there are a number of world-leading
practitioners in NZ including Developmental Evaluation Institute
https://www.developmentalevaluation.institute/
the most helpful approaches we’ve found are in Appendix three
below. Tamarack Institute and What Works elaborate more about
putting these tools into practice.
An Indicators Framework for Noticing and Capturing Community-Led
Change Each place and initiative has its own unique context, so
there is no magic CLD indicator set that can be applied to all
locally-led initiatives. But here’s some common framing that may
assist diverse community-led change efforts to notice, measure and
assess the difference they are contributing to. Our Inspiring
Communities 6 P indicator framework below has been influenced by
our early work with 8 CLD initiatives2, and Tamarack (our sister
organisation in Canada) who have been working on collaborative
efforts to reduce poverty. This framework helps us get clear about
what is our own indicators of success. What matters most may not
all be easy to measure, but our observation of trends and patterns
of change over time is just as important. We are observing a living
community – and like raising a child - all sorts of unexpected
things will happen! So we don’t limit our attention to only
observing what we have chosen as indicators of success, but it does
help us focus. These indicators help focus our attention on who’s
involved, what’s happening, how well we are doing, and what’s
changing at many levels – subtle changes in attitudes and beliefs
about community potential, improved wellbeing, wider systems
changes. Indicators should help us make sense of how and why things
happened and what we are learning from our past work to inform our
future work. Indicators at their best help us focus our
observations and reflections on our community as a living, growing
‘garden’ and how we as ‘gardeners’ can help create the conditions
for it to flourish. CLD Indicator Lenses We Find Useful 1.
Participation Indicators: engagement, involvement and ownership of
local people and
stakeholders in what's happening. (Who).
2. Progress Indicators: track actions taken to achieve local
visions and goals. (What happened).
3. Performance Indicators: track effectiveness of processes,
infrastructure and frameworks in place to achieve local visions and
goals. (How/how well).
4. Possibility Indicators: changes in belief about what’s possible
locally, new potential arising and a willingness to act. (Can do/do
next).
5. Population or People Indicators: changes in wellbeing outcomes
for local people. (What's changed).
6. Policy/Systems Change: tracks impacts and changes on bigger
picture thinking, funding, policies, approaches that local CLD
efforts have contributed to. (Now being done differently).
In some cases, indicators will be a specific quantified measure
e.g. % change of children under 5 enrolled in early childhood
education. In other cases, the indicator may be a story e.g. ‘How
have local visions positively influenced key stakeholder plans and
policies?’, with evaluation processes encouraging
reflection/looking back to assess and identify various
contributions and changes. See Appendix One for an example of an
outcome measure (A strong and connected community) and
2 For more see
http://www.communityresearch.org.nz/wp-content/uploads/formidable/Understanding-and-accelerating-
community-led-development-in-Aotearoa-New-Zealand..pdf
potential CLD indicator layers. See Appendix Two for an even wider
menu of potential measures for exploring progress and change in
relation to the five CLD practice principles. This creates a VERY
large matrix of possible measures and we are NOT suggesting that
initiatives try to assess them all. Rather, think of the matrix as
a useful guide for choosing what might be most useful to notice,
measure and assess within your locally-led framework. Try to adopt
a few measures across different indicator/principle areas that best
reflect your community initiative’s focus and what you most want to
use the evaluation evidence for. Use our CLD Planning and
Evaluation Tool to help you pull together a simple framework for
your use – and adapt as necessary as you go. Data gathered at the
beginning of your initiative (see Quick tips on making the most of
community feedback and Mapping community strengths and assets) can
provide an excellent baseline from which to identify changes over
time No matter what indicators are chosen, it’s important that they
are:
• grounded in the collective vision/definition of what success
looks like for each local community
• as simple, meaningful and useful as possible. We need to make
sure we’re putting effort and attention on the things that really
matter!
• utilised in ‘sense making’ conversations that enable local
communities and stakeholders to unpack and interpret the data
generated to inform future plans. This step is often overlooked and
yet it is key to informing ongoing development and effectiveness of
locally- led planning and investment in your initiative and action
planning.
Telling your story effectively One of the greatest rewards of doing
our evaluation work, is to be able to share our story with others
who might have an interest in or influence in helping us achieve
our vision e.g. members of our local community, funders, partners,
policy decision-makers or others previously marginalized from our
efforts. How we share our story can have a powerful impact on who
listens, engages and walks with us to engage with our vision. Here
are some great storytellers that have helped CLD initiatives tell
the story of the difference they are making in communities in a way
that people really sit up and listen. Moya Sayer Jones at
http://www.onlyhuman.com.au and Pixar at
https://www.khanacademy.org/partner-content/pixar/storytelling. The
second one is useful even if you are not making a film! The
difference we are making is all about impacts on people lives, our
planet, who has power and
much more. These are compelling human interest stories of joy and
frustration, successes and
challenges. A story which communicate the ‘before’ and ‘after’
difference we have observed
through our work, combining stories, statistics and your insights
over time about how those results
have been achieved, can help other communities work for change too.
Check out our stories from
Aotearoa communities using a CLD approach and we encourage you to
share yours with us here.
And here’s an example of insights harvested by a funder from across
the social innovators they
support that we can all learn from:
http://12lessons.mcconnellfoundation.ca/.
Noticing the Difference CLD Makes – helpful tools3
"The journey is as important as the destination. We need to give
them equal weight." Michael Quinn Patton
Throughout the process of learning together about CLD we’ve been
guided by “12 emerging principles of capturing and making sense of
outcomes” developed by Mark Cabaj of Tamarack.
12 Emerging Principles of Capturing and Making Sense of
Outcomes
1. Be as clear as you can (but no clearer) 2. Take a utilisation
focus 3. Use a contingency approach 4. Engage stakeholders from
beginning to end 5. Invest proportionally 6. Avoid trying to
re-prove the proven or testing the obvious 7. Gather hard and soft
data from multiple sources 8. Emphasise sense-making 9. Look for
contribution rather than attribution to change 10. Use goal
oriented and goal free evaluation 11. Strive for roughly right 12.
Be adaptable and flexible in your approach
Each CLD initiative we’ve worked alongside has used various tools
and frameworks to both gauge progress and assist their development.
Those that have proven most useful in helping to track changes and
to notice the difference CLD approaches are making include4:
1. Appreciative Inquiry: An intentionally strengths-based approach
that focuses on identifying and building on what an organisation or
community does well rather than on eliminating what it does badly.
By asking questions and envisioning the future, positive
experiences, relationships and other assets, including the
difference that is being made, are highlighted. This reveals
potential that can then be used to foster motivation for
improvement through a cycle of four processes:
1. DISCOVER/INQUIRE: Find out what already works well. 2.
DREAM/IMAGINE: Imagine what could work even better in the future.
3. DESIGN/INNOVATE: Plan and prioritize how that future vision
could actually work in practice 4. DELIVER/IMPLEMENT: Navigate the
change, and notice the difference being made.
Appreciative Inquiry is particularly useful in assessing change
when used as part of participatory evaluation approaches focused on
innovation and creativity with a wide range of stakeholders. It is
also useful in building the evaluation capacity of those
participating.
3 Adapted from Learning by Doing: Community-led Change in Aotearoa
NZ Inspiring Communities (2013), pages 154-59 4 See
http://whatworks.org.nz/ for more resources on these methods and
http://www.communityresearch.org.nz/ has excellent webinars that
help bring these alive in real examples
Diagram from: betterevaluation.org
3. Theory of change: A graphic representation of the change process
that also expresses causal connections between action and results.
But rather than describing the outcomes of a programme, a theory of
change focuses on achieving a change and conditions or approaches
needed to achieve that change.
(From http://www.theoryofchange.org/) A theory of change spells out
underlying assumptions and describes a process of desired social
change by making explicit the way we think about a current
situation or problem, its causes, the long-term change we seek, and
what needs to happen in society in order for that change to come
about. By uncovering the collective thinking about what
achievements are intended, interventions and actions can be
developed and adapted more proactively as potential weaknesses or
gaps are also identified. This focus on the intended change helps
develop more coherent and nimble approaches to achieve this change
that reach beyond any specific programme, especially if the theory
of change is collective, and regularly tested, reviewed and
evaluated. As well, successes and
lessons can be easily demonstrated along the way. Our Planning and
Evaluation tool elaborates an example of using a Theory of Change
approach.
4. Eco-cycle Framework: As well as helping to describe the
lifecycle of an initiative and the way decisions can feed into what
comes next, the eco- cycle can also be valuable in defining the
most useful kinds of evaluation for particular stages. For example,
developmental evaluation supports the design and progress of
innovation to guide creation and adaptation especially in emergent
and complex situations5 while a formative evaluation is focused on
improving, strengthening and fine tuning an intervention or
existing programme. There may be some performance monitoring and
process evaluation between exploitation and conservation when a
summative evaluation can be useful where its merits or worth are
assessed. The release phase allows for a harvesting of knowledge to
uncover patterns and/or principles of effectiveness and learnings,
ideally to feed into the next phase of the process. In this way an
initiative or organisation can continue to evolve in ways that can
most usefully contribute to the achievement of stated goals. (Image
from http://www.appartenance-belonging.org/) See our eco-cycle
resource for other ways this lifecycle framework can be
useful.
5. Outcome mapping helps with the 1) documentation of the journey;
2) reassessment (and revision) of the intended ‘destination’; and
3) improving capacities and capabilities to continue the journey.
The process is participatory and focused on outcomes rather than
impact, while recognising that impacts (desired changes in state)
are the ultimate goal.
Outcomes are understood as changes in behaviour, relationships,
activities or action of people, groups and organisations with whom
an initiative works directly. These outcomes can be logically
linked to the initiative but are not necessarily caused by them.
Guided by understandings of ‘boundary partners’ and spheres of
influence, outcome mapping has three core stages: intentional
design, outcome and performance monitoring, and evaluation
planning. By using outcome mapping, contributions to outcomes can
be revealed and social and organisational learning can be
influenced.
6. Outcomes Harvesting is a way of working out what has happened.
It does not measure progress towards predetermined outcomes or
objectives, but rather collects evidence of what has been achieved,
and works backward to determine whether and how the project or
intervention contributed to the change. It yields evidence-based
answers to the following questions:
• What happened?
• How do we know this? Is there corroborating evidence?
• Why is this important? What do we do with what we found
out?
5 See Michael Quinn Patton (2011) Developmental Evaluation. The
Guildford Press, New York.
owned; a quality, condition,
advantage, support,
Answers to these questions provide important information about the
contributions made by a specific program toward a given outcome or
outcomes.
7. Most Significant Change is a form of participatory monitoring
and evaluation. Many stakeholders are involved in deciding the
sorts of change to be recorded and in analysing the data. The
process occurs throughout the initiative cycle and provides
information to help people manage activities, monitor progress
towards goals while also providing information on outcomes and
impact. Essentially, the process involves the collection of
significant change stories from those directly involved and the
systematic selection of the most significant of these stories by
panels of designated stakeholders or staff. Various people then sit
down together, read the stories aloud and have regular and often
in-depth discussions about the value of these reported changes. The
process has been recognised as effective in identifying unexpected
changes; distinguishing prevailing values across groups and
organisations; encouraging analysis as well as data collection from
those closest; delivering a rich picture that requires no
specialist skills and useful in monitoring and evaluating bottom-up
initiatives that do not have predefined outcomes against which to
assess progress or achievement. Mangakino’s story is an example of
applying a most significant change approach.
8. Splash and ripple is a particular approach of outcome
measurement that uses metaphor to help grow understandings. The
rock is like a material Input, the person holding the rock is like
a human resource Input. The act of dropping the rock is like an
Activity. When the rock reaches the water, it creates a splash.
These are the outputs. The ripples, spreading out from the splash
are like the Outcomes, and then later the Impacts. The edge of the
pond represents the geographic and population boundaries of the
initiative. Splash and Ripple is an outcomes measurement approach
to planning and managing initiatives that encourages those involved
to be clear both about what they are doing and what they are
changing. Partly driven by a desire for greater accountability,
outcomes measurement also helps to influence wise planning and
management decisions. The Splash and Ripple approach includes five
guiding ideas: time, reach, control, context and learning and
improvement and has been used by police, justice and health
services as well as communities and community groups.
9. Impact stories are personal tales of transformation that share
insights, experience and changes via a story telling approach. They
are about individuals and are generally written in the first person
to simply communicate the ‘what happened/what changed, how and why’
in an honest, authentic and very ‘human’ way. They are often
accompanied by a picture. Written well, impact stories build
empathy and understanding about local situations and contexts by
encouraging a relational connection to the story. They can help
bring ‘big picture’ data and observations to life and are useful
for connecting cause, effect and contribution in ways that purely
statistical approaches are often unable to.
10. Asset Mapping Asset mapping is a process of identifying and
documenting the resources, strengths, gifts and assets that are
already in communities. Including those that are often not seen,
understood or counted. Asset mapping processes encourage dialogue
about what local people value and aspire to, the connections
between assets and who/how local communities are already using
their assets and how they might be used differently in the future.
See our asset mapping tool here.
Appendix 1: Sample Outcome Area and Key Indicator Layers
Intended Outcomes Participation Indicators: Progress Indicators:
Performance
Indicators:
Indicators:
Policy/Systems
Change:
happen.
stakeholders in what's
achieve local visions and
willingness to act on those
beliefs.
contributed to.
better place to live.
# residents who know the
neighbours
eg. (interviews, story
Sample outcome area and indicators:
Appendix 2: Indicators of Effective Community-led Development by
CLD Principle - 6 Note: indicators below are examples rather than
an exhaustive list of measures CLD Principles7
Participation/process Indicators (who)
Progress Indicators (what)
People/Population Indicators (What’s
• Processes developed to ensure inclusion and engagement in local
planning
• Engagement/participation rates in community engagement processes
to develop local visions/ priorities/plans (# people involved,
roles, sectors/groups, participation methods)
• Local planning and action projects designed by local people
• Local governance group in place involving range of local
stakeholders
• Local people in decision making roles
• Following engagement with community/ stakeholders, shared
vision/goals identified
• Action plan (s) developed to achieve vision and goals
• Resourcing secured/allocated to achieve local goals
• Milestones being achieved
• Reflective/developme ntal processes established in ways that
maximise community contributions
• Potential synergies between iwi/hap and community development
plans identified and activated
• Local people have a sense of ownership, can articulate future
vision, and are actively involved in making it happen
• Resourcing enables action on locally identified priorities
• Participants on governance/leadership groups enjoy their work,
acquiring & sharing new skills
• Learnings from action/reflection processes woven into next stages
of planning and action.
• Work and activities are clearly linked and contributing
effectively to achievement of vision
• Agreement that local community priorities and needs are well
known and driving change in effective
• Belief that community’s vision and associated actions will make a
difference
• Increased sense of community pride and belonging
• New leaders/ organisations wanting to be involved in locally- led
action
• New actions/goals identified as progress is made and are woven
into next steps and approaches
• Reported changes in sense of community, civic engagement, having
a voice and power in influencing local change
• Specific initiative outcomes in relation to improvements in
quality of life and vision, such as: stable school roll
rehabilitated
natural environments
improved living standards
• New local mechanisms/ structures in place for enabling and
activating CLD approach eg shared governance arrangements, matching
funds, reduced red tape/compliance costs, community
engagement/reporti ng processes.
• Local visions now influencing other stakeholder plans, practices
and investment
6 Note this table offers some example indicators only 7 For more
detail on the CLD principles and what underpins them see
http://inspiringcommunities.org.nz/ic_resource/nga-matapono-cld-principles/
People/Population Indicators (What’s
ways
2. Build from strengths
• Diversity of local people, groups and resources involved in the
initiatives(s)
• Strong understanding of history of this place from mana whenua
and tauiwi perspectives
• New projects that are bolted onto existing groups/ collaborations
etc
• Voter participation in local elections (Council and School/Kura
Board of Trustees, Community Trust)
• # of local organisations actively involved in community building
activities, going beyond their traditional /core functions)
• Rates/trends in parent volunteers/engagement at local events and
facilities eg. sports clubs
• Attendance at local community events
• Local residents actively part of local CLD governance
processes
• Range of uses/roles of existing local assets etc
• Extent to which community perspectives/views are gathered,
included and fed back to local people & and external
stakeholders
• # of actions taken by local people to progress issues/new
initiatives
• Community stories and feedback are part of local evaluation and
review processes
• # of actions/ processes that intentionally seek to engage &
involve diverse sectors of the community eg. new migrants, sports
clubs, young
• New processes/projects seen to build from existing local wisdom
and experience
• # and range of new people leading/involved in local community
projects/activities.
• People feel they have a real say/can get involved in local
issues
• People feel they can influence decisions that affect their
area
• Those championing CLD projects feel supported by other
leaders/groups in the community.
• Those championing community-led initiatives feel connected to
others who can help
• Perceived level of information sharing between groups/sectors/
neighbourhoods
• Agreement that young people (or other target groups) are valued
in this
• Local assets, gifts, talents identified and activated in new
ways
• Increased % of residents who believe they can influence change in
their community
• Increased % of residents/groups getting involved with local
action/projects
• New resources and assets have been identified to assist local
efforts
• Names of neighbours/children known in a street
• People feel well informed about local affairs
• People feel able to ask a neighbour for help
• Levels of trust and confidence within the community
• Locals feel they are/can contribute to their community
• Levels of community spirit and pride
• Existing local organizations feel supported and valued
• Perceptions of health and wellbeing at individual and community
levels
• Recognition of assets and strengths in a place/local people/local
iwi/hap
• Changes in agency work practices/ approaches to build on local
strengths and/or align with locally-led efforts
• Agency/funder support for local capacity & capability
building and leadership development
• Stakeholder plans include new initiatives that build onto
existing local initiatives rather than starting from scratch.
CLD Principles7
People/Population Indicators (What’s
Policy/Systems Change Indicators (Now done differently)
• Local residents stepping up to take on active 'doing' roles as
part of CLD efforts (can be paid/unpaid).
people, gangs, sole parents, businesses, churches etc.
community
• Mana whenua relationships well established
• New organisations/ agencies involved in community-led change
efforts
• Different combinations of people/organizations working
together
• Number of collaborative projects underway
• New strategic alliances/project groups advancing locally-led
projects (numbers of partners, sectors involved)
• New and/or increased level of funders/funding and resources
identified to support community change efforts
• Participation/leader ship of local residents in new initiatives
or projects
• Potential synergies between iwi and community development
plans
• Perceptions about quality of relationships and
collaboration
• Perceptions about individual/ organisational capacity and
effectiveness in collaboration
• Knowledge about how to volunteer/support community-led
activities
• Giving rates (time and money) to support local organisations and
initiatives
• Perceptions of the way people are working together to support and
improve their community
• Collaboration partners optimistic about what they can achieve
together
• Collaboration partners prepared to try new things together
• Local community feels supported by external partners
• Belief that XX community is/can be an even better place to
live
• New partnerships forming to explore new ideas/potential
projects
• % of residents who report regular social contact with their
neighbours in the last month
• % of residents who felt isolated or lonely over the past twelve
months
• % residents who report new local connections
• Volunteer hours supporting local projects/organisatio ns
• % residents who have done/ received a favour for/from a
neighbour
• Levels of trust in the community
• % of people who feel that in their local area people from
different
• Willingness of local boards/local council to act in support of an
issue, idea or proposal raised by the community
• Long term funding arrangements established to support local
efforts
• Changes to existing processes, plans, systems as a result of new
collaboration efforts
• New multi sector governance arrangements established to support
CLD efforts
CLD Principles7
People/Population Indicators (What’s
identified
backgrounds get on well together
4. Grow collaborative local leadership.
• Local residents stepping up and/or supported to step up to take
on active 'doing' roles as part of CLD efforts (can be paid/unpaid
roles).
• Processes established to regularly connect local leaders and
community contributors.
• Calls for help and assistance with CLD projects are increasingly
answered by willing volunteers
• Local community award/recognition processes in place
• Relationships established between iwi/hap leaders and local
community
• Number of forums held that intentionally bring diverse leaders
& community actors together to discuss local
issues/actions
• Local leadership/ people development strategy in place
• Communication channels created to actively share and cross
fertilise local news, activities and needs
• # of stories/ postings in (social)
• Quantity and/or quality of local coverage about locally-led
action & change generated in print, broadcast, or electronic
media
• General community recognition of community-led projects, events
and initiatives eg. Have you heard of XXX organisation entity, what
activities do you know of that are associated with this ….
• Recognition of community-led efforts by local authorities,
iwi/hap, businesses and community funders.
• Local leaders prepared to try new things/ideas they’ve
generated
• Confidence that conflicts/challenges that arise in the community
can be overcome
• Belief that local leaders work well together
• Levels of trust in community leaders
• Perceived depth of leadership in the community
• Leadership development integrated into all locally-led
initiatives
• Local leaders report increased skills, confidence and sense of
optimism
• % community members feeling positive about their community’s
direction and local leadership
• Intentional strategies developed to develop and support
collaborative local leadership development
• Mechanisms in place so that understanding/ knowledge about local
leaders and their contributions is widely known/shared
• Existing local leaders involved in training/ supporting next
tranche of local leaders (each one teach one).
• Agency/funder support for collaborative capacity building and
leadership development
CLD Principles7
People/Population Indicators (What’s
• Perceptions of power sharing
5. Learn by doing
• Multiple useful opportunities for reflection enacted (eg surveys,
1-1 chats, report backs, meetings)
• Active engagement/ participation of multiple voices/views in both
strategy/project reviews (who, sectors etc)
• Consistency/growth in participation of local people in
locally-led efforts
• Local people's contribution to results/ leadership is fed back to
empower further participation
• Participants in change processes enabled and/or resourced to
participate in reflection and development of next steps
• Failure and success are
• Number/quality of action-reflection sessions held to review
progress and learning to date, and adapt next steps/direction as
required
• Number of plans and projects reviewed and adapted/updated based
on new knowledge/local learning
• Community informed of progress and resulting changes to
• CLD stakeholders are engaged and empowered by what’s happening
and how
• Projects generate momentum and interest, as well as positive
outcomes
• Projects can be effectively sustained in ways that include local
people
• New areas for potential action identified and actioned through
action-reflection processes
• Outcomes lead to new projects and inclusion of different parts of
the local community
• Local people say that locally-led change efforts are making a
positive impact on local quality of life
• Local people feel their community is a great place to live/moving
forward
• New community resources or structures created to support local
change efforts
• New or adapted policies that local change efforts have
contributed to
• Quantity and quality of local community support in behind CLD
efforts
• New working relationships and ways of working with/within the
communities
CLD Principles7
People/Population Indicators (What’s
Policy/Systems Change Indicators (Now done differently)
equally valued and included in ways that empower and enable local
people
• Number/frequency of community celebrations