7/31/2019 Advancing Evaluation Practices in Philanthropy

1/24

Advancing EvaluationPractices in Philanthropy

This sponsored supplement was produced by Stanord Social Innovation Review or the Aspen Institute Program on Philanthropy and Social Innovation.

The supplement was underwritten by a grant rom the Ford Foundation.

Frai IssP. B Jane Wale

A Focs o ClrP. B Lu A. Uba

Lari froSilico VallP. 8B Ma Bannck & Erc Hallen

Sar OcosP. B Juh Rn & Nanc MacPhern

Ris BsissP. B Paul Bre

Assssi Os OwPrforacP. 0B Jame E. Canale & Ken Rafer

7/31/2019 Advancing Evaluation Practices in Philanthropy

2/24

AdvANCiNg EvALUAtioN PRACtiCEs i N PHiLANtHRoPy

Sponsored Supplement to SSIR

JAne WALeS is vice president, Philanthropy and Society, o the Aspen Institute anddirector o its Program on Philanthropy and Social Innovation. She is the ounding CEOo the Global Philanthropy Forum.

In recent years, the philanthropic sector has neared consen-

sus on the need to improve measurement and evaluation o

its work. Although the philanthropies they lead use dierent

methods, members o the Aspen Philanthropy Group (APG)

have agreed that basic principles and practices can inorm eorts

to monitor perormance, track progress, and assess the impact o

oundation strategies, initiatives, and grants. They hope to build

a culture o learning in the process.

Over the past two years these CEOs o private, corporate, andcommunity oundations have supported a series o meetings on mea-

surement and evaluation (M&E) with leaders o grantee organiza-

tions, issue experts, and evaluators. They have concluded that, when

done right, assessment can achieve three goals. It can strengthen

grantor and grantee decision-making, enable continuous learning

and improvement, and contribute to eld-wide learning. Below are

broad observations rom the workshop process, ollowed by articles

rom ve APG authors describing the M&E philosophies o the in-

stitutions they lead. Their articles will be among those to appear

in an edited e-volume, to be published by the Aspen Institute and

continuously updated to capture evolving oundation practice and

comments rom voices in the eld. This is what we learned.Denitions Matter| APG members ound that diering ter-

minology can undermine eorts by grantors and grantees to col-

laborate eectively in the design and implementation o an M&E

system. Many grantors and grantees use the terms evaluation,

impact measurement, and measurement and evaluation in-

terchangeably. In act, M&E encompasses distinct activities with

distinct purposes, methods, and levels o diculty. In his article,

William and Flora Hewlett Foundation president Paul Brest sepa-

rates M&E into three categories undertaken at three stages: theo-

ries o change described and logic models devised during the ini-

tial design o a project or oundation initiative; tracking progress

against the strategy set during the lie o the grant or initiative;

and assessing impact ater the act. The rst o these is essentialbackground or M&E, and the three together provide a useul

means o organizing the various activities and purposes o M&E.

The second enables a grantor and grantee to gain the inormation

needed to make mid-course corrections to the strategy and learn

throughout the process. The third activityassessing impactis

the most daunting. Brest notes that in some undertakings, such

as policy advocacy or Track II diplomacy, exogenous infuences

Framing the IssueBy Jane Walesmake it hard i not impossible to attribute impact to any one actor

or strategy. He argues or demonstrating contribution rather

than claiming attribution, where contribution means increasing

the likelihood o success, and notes that the true impact o such

risky grants may not be possible to ascertain. Nonetheless, they

are well worth pursuing.

Purpose Matters | At its best, M&E inorms decision-making

and provides or continuous learning. In his article, Matthew Ban-

nick, managing director o Omidyar Network (ON), discusses why

M&E is more likely to be usedand used to good eectwhen it is

designed collaboratively by grantor and grantee, and when data

are gathered and organized around decisions that each needs to

take. It is thereore critically important that they agree on their

evaluation approach at the outset. Ford Foundation president Luis

Ubias agrees, adding that rom the very beginning, grantees

should have a clear sense o what benchmarks o success are ex-

pected o them at each stage o initiative developmentand why.

The Cost-Beneft Ratio Matters | Ubias points out the costs

o M&E, arguing that in designing an evaluation system, careul

consideration must be given to the burdens on each party. Failure

to do so, he writes, can lead to excessive data gathering in which

grantor and grantee gather as much data as possible in search oevidence o impact. The costs are ourold. First, it is a burden

to grantees, creating surplus work or oten tightly staed and

nancially strapped nonprots. Second, it undermines quality

because grantees will provide the requested inormation to meet

their grant obligations, but may not have time to supply the in-

sight that is oten more valuable than the data. Third, it inundates

oundation sta with inormation but may leave them little time

to use it eectively. Fourth, it may not provide the inormation

that is actually needed to understand how eective our initiatives

and grantmaking are. According to Bannick, ON reduces the

burden by using a limited number o easily collected metrics, and,

as an alternative to the time-consuming, costly, and complicated

challenge o measuring impact, ON oten measures outputs asproxies. As or the price tag, Rockeeller Foundation president

Judith Rodin notes the eciencies gained by using technology

to gather real time data. Brest notes (inMoney Well Spent,co-

authored with Hal Harvey) that i you are a philanthropist with

a long-term commitment to a eld, it is well worth putting your

undsand lots o theminto evaluation.

Culture, Context, and Capacity Matter|M&E requires a commit-

ment to building capacity within oundations, grantee organizations,

and the eld o evaluation in general. The Rockeeller Foundation

invests in M&E teams in both the developed and developing world

7/31/2019 Advancing Evaluation Practices in Philanthropy

3/24

AdvANCiNg EvALUAtioN PRACtiCEs i N PHiLANtHRoPy

Sponsored Supplement to SSIR

to monitor oundation initiatives and to act as critical riends to

its grantees, establishing monitoring and learning systems where

none existed. The goal is to acilitate learning among grantees and

within the oundation aimed at improving perormance all round.

But most important, it is to leave behind greater capacity among

local M&E proessionals. Rodin reports that the oundation has

supported regional institutions that train and mentor local evalua-

tors and partner with similar institutions elsewhere, with the goalo building lasting capacity. Bannick speaks to the importance o

providing technical assistance to grantees. And, within a ounda-

tion, James Irvine Foundation president James Canales notes, it

is important or there to be leadership by trustees and senior o-

cers, as well as a readiness to devote time, dollars, and expertise

to assessing the philanthropys strategy, initiatives, and grants.

Doing so mandates ull institutional commitment and cannot be

the province o just the evaluation director. Beyond these impor-

tant tangible contributions lies the requirement or building what

Ubias calls an impact culture in which continuous learning and

adaptation are enabled, required, and rewarded. Goals, theories

o change, and operating approaches are all necessarily imperect;

only by learning rom our successes and our mistakes can we build

an impact culture, he writes.

The Unit o Analysis Matters | The goodand the badnews

is that almost any activity can be evaluated. It is important to sort

out the dierent units o analysis, as is done in the Bill and Melinda

Gates FoundationsGuide to Actionable Measurement,which notesthree distinct areas o ocus. At the level o oundation strategy, the

ocus should be on measuring outcomes over impact (as Bannick

describes), on assessing contribution rather than attribution (as Brest

recommends or certain grants), and on the degree o harmony that

can be achieved among grantees pursuing a given strategy. At the

level o oundation initiative, the oundation should use grantee re-

porting data on outputs and outcomes to signal whether the initia-tive is making progress; track the program teams activities other

than grants (such as convening and public speaking); use indepen-

dent evaluation; and capture both intended and unintended con-

sequences o the initiative. And at the level o the individual grant,

the oundation should align expected grant results with strategic

intent; work with the grantee to track grantee inputs, activities,

outputs and outcomes at critical points to manage and adjust each

grant appropriately; and measure the oundations input o human,

nancial and technical resources.

Timing Matters |Just as the units o analysis dier, so too do the

time horizons required to measure and evaluate the perormance o

a oundations strategy and grants. Many short and medium-term

metrics are useul in assessing how well an organization is managingprocesses or reaching target populations. Longer-term longitudinal

studies are critical to gauging the impact o a program and to estab-

lishing the causal relationship between intervention and desired out-

come. Such rigorous, long-term studies can be particularly useul or

those seeking to scale up innovations. Canales notes, however, that

the annual grant-cycle poses a structural barrier to the longer-term

undertaking o evaluating and learning, noting that program goals

and aspirations rarely ollow annual timelines, nor should they.. .i

they are suciently ambitious. He points to the importance o cre-

ating the space or consideration o broader progress assessment.

Feedback rom Grantees and Beneciaries Matters | M&E

must incorporate into the process the viewpoints and observa-

tions o the under, grantee, and ultimate beneciary through all

stages o workidentiying problems, co-creating solutions, and

implementing with a shared vision o outcomes. Under the leader-

ship o APG member Carol Larson, the David and Lucile Packard

Foundation solicits eedback rom the on-the-ground sta o the

grantee and has established written standards to help its programteam to communicate with grantees. Grantees, in turn, can bet-

ter assess community needsand their perormance in helping

to meet those needswhen program beneciaries provide them

quantitative and qualitative eedback. At Ford, Ubias notes that

the oundation selects grantees managed by those living and

working closest to where targeted populations are located. Noting

the Rockeeller Foundations commitment to evaluation practices

that include stakeholders voices, Rodin cites the consensus o

the Arica Evaluation Association: only when the voices o those

whose lives we seek to improve are heard, respected and internal-

ized in our understanding o the problems we seek to solve will

philanthropy achieve its purpose.

Transparency Matters | Although the goals o M&E are to in-

orm decision-making and enable continuous learning by those

immediately involved, there is a larger community to serve and

a larger purpose to pursue. By publicly sharing the data gathered

and conclusions reached, grantors and grantees can contribute to

eld-wide learning. APG members agree that this is an opportunity

to seize. The philanthropic sector has helped build communities o

practice that generate knowledge. These evaluators orm proes-

sional associations that set standards or the eld. Independent

organizations assess oundation and grantee perormance and

publish their ndings. Donor organizations and networks trans-

er knowledge among philanthropies, and between grant-making

institutions and individuals. And academic programs providethe intellectual underpinning or much o this work. Supported

by philanthropy, these and other institutions provide some o the

early hardware or wider impact. Moreover, the gradual evolution

o principles that guide and practices that enable rigorous evalu-

ation can contribute to its sotware.

But has a true systemor philanthropic impact been designed

and widely adopted? Perhaps not. And so, in publishing an e-volume

and opening a conversation, Aspens Program on Philanthropy

and Social Innovation will seek the wisdom o the crowd and ask

the questions: What might the components o such a system be?

Where will the breakthroughs occur? What sort o venture capital

will be needed to nance the prototypes? And what markets will

bring these innovations to scale? Or, building upon Ubiass aptphrase, under what conditions might an impact culture spread?

What would it take or its language to be adopted, its standards

embraced, its methods rened, and its potential realized? And i

that culture is to be global, dynamic, and enduring, how might

it be inormed and advanced by the new cadre o evaluators to

which Rodin reers?

None o the architects or beneciaries o modern-day philan-

thropy would claim that they alone can create such a world, but they

may well agree that it is one worth imagining. Doing so together

would make or a powerul beginning.s

7/31/2019 Advancing Evaluation Practices in Philanthropy

4/24

Sponsored Supplement to SSIR

AdvANCiNg EvALUAtioN PRACtiCEs i N PHiLANtHRoPy

making a dierence by delivering the results

we seek? To truly maximize impact, oun-

dations must work to build results-ocused

cultures that embed internally the risks and

demands aced by our grantees, and demand

o ourselves the same level o perormance

demanded o grantees.

At the Ford Foundation, we continually

work to answer these challenges. We make

long-term investments, understanding

that patient capital and well-reasoned risk

are required to chart bold new solutions to

complex social problems. Our goals are cen-

tered on social justice principles that rec-

ognize the inherent dignity o all people. In

accordance with these principles, the oun-

dations work seeks to ensure that social

systems and institutions give all people a

voice in decisions that aect their lives, and

the opportunity to reach their ull potential.

This is ar-reaching and essential work, butit is also, by nature, dicult to evaluate.

Our challenge, then, has been to create

a culture ocused on results within the Ford

Foundation without sacricing either our

ambitious objectives or our commitment

to maintaining certain core principles as

we pursue those objectives. In building and

ostering a results-ocused culture, the Ford

Foundation is guided by ve principles:

. Create a clear and ocused strategicvision.

2. Allocate resources on a dierentiatedand dynamic basis.

3. Build accountability based on clearlydelineated roles and responsibilities.

4. Put a premium on deep and eectivelistening.

. Implement a results-ocused cultureacross the entire organization.

Realizing these undamental principles

in an organizations culture is no simple un-

Philanthropic organizations conront

some o the most exigent and endur-

ing problems acing humankind.

Our mission-driven organizations

pursue broad societal goals, including reduc-

ing poverty, advancing human rights, oster-

ing educational achievement, and strength-

ening democratic principles and processes.

Pursuing these goals is a long-term commit-

ment, requiring aggressive problem solving,

sustained eort, and rm resolve.

Our philanthropic resources are excep-

tionally modest when measured against the

depth o the social and economic challenges

we tackle. And in a volatile economic cli-

mate, philanthropic leaders must be highly

disciplined in managing limited resources.

The need or strategic and eective philan-

thropic eort is urther heightened by the

unprecedented pace o change broughtabout by the seemingly limitless techno-

logical innovation o our time.

Now more than ever, results matter, and

how we as philanthropic leaders dene, pro-

mote and reinorce a commitment to results

in the culture o our organizations can pro-

oundly aect whether or not our collective

mission to make a dierence in peoples lives

succeeds. A results-ocused culture must be

predicated on an institution-wide commit-

ment to clearly dened objectives pursued

with strategic clarity and supported by dy-

namic resource allocation. Results can onlybe dened by end outcomes to the communi-

ties we are serving. Process or activity-based

results are valuable on an interim basis, but

the results that matter are the results that are

elt by people in need. How do we understand,

in appropriate timerames, whether we are

A Focus on CultureHow the Ford Foundation is engaging its global sta in building a shared culture o results

By luis a. uBias

LuIS A. uBIAS is president o the Ford Foundation.Beore joining the oundation in 2008, he was a directorat McKinsey & Co., leading the frms media practice onthe West Coast.

dertaking. It requires comort with uncer-

tainty and risk, and openness to elements o

accountability that at times call or signi-

cant shits in perspective. It demands that we

continually ask ourselves, at every stage o

our work and at all levels o the organization:

Are we making as great a dierence as we

can or the communities and people we are

entrusted with serving? Is there anything

we can learnand changeto achieve the

maximum results? And how do we dene

what constitutes results or the range o

work and goals to which we are committed?

CReAte A CLeAR And FOCuSedStRAtegIC VISIOnA culture ocused on results begins with a

clear understanding o objectives and the

development o strategies to achieve them.

During my rst year as president o the Ford

Foundation, the global program team andI went through a comprehensive review o

our grantmaking, motivated by the desire to

bring ocus and clarity to our work. We iden-

tied a core set o social justice issues that

together constitute a cogent way to be true

to the oundations mission. For each issue

we set clearly dened goals and strategies,

developed theories o change or achieving

them, and designed operating approaches

or working day-to-day. The results o this

eort can be seen on our website, which

provides a visual map o our grantmaking

objectives and strategies and directly linksthem to the nonprot and nongovernmental

eorts we are supporting.

The second phase o the eort was to

determine the scale o resources required

to achieve each strategy. Focusing our

resources meant we increased the aver-

age unding o initiatives rom just over $

million to more than $0 million each. We

brought the same kind o depth and concen-

tration to our stang, moving rom indi-

7/31/2019 Advancing Evaluation Practices in Philanthropy

5/24

Sponsored Supplement to SSIR

5AdvANCiNg EvALUAtioN PRACtiCEs i N PHiLANtHRoPy

vidual pursuit o initiatives to a team-based

approach. Perhaps most important, we were

able to thoughtully deploy the oundations

non-grant resources, including eld lead-

ership, public and private sector partner-

ships, communications, and convening

capabilities. Direct nancial support, our

traditional staple, became only one part othe portolio o tools the oundation could

mobilize to address a problem or challenge

once a strategic vision had been set.

One example o the results that can be

achieved by combining non-grant resources

with nancial resources is JustFilms, a new

Ford Foundation eort to advance social jus-

tice through the creative lens o emerging

and established documentary lmmakers.

Although the oundation has been a leading

under o social justice documentaries or

decades, supporting landmark lms such as

Eyes on the Prize, this eort is harnessing the

power o lm to create a national and global

dialogue on social justice issues.

Our rst step was to identiy clear, o-

cused outcomes that could be thoughtully

assessed, such as growing a new cadre o

lmmakers, bringing social justice lms to

market at strategic moments that maximize

awareness and social change, helping lms

nd an audience, and connecting new works

to a global network o social change makers

and movements. Evaluating how we were

meeting these objectives required us sub-sequently to change our strategy, more than

tripling our unding rom under $3 million

annually to more than $10 million annually

to support our aspirations; establishing a

creative collaboration with two leading lm

organizations, the Sundance Institute and

the Tribeca Film Festival; and assigning

two dedicated, ull-time, high-level and

experienced oundation sta members to

direct the initiative. In its rst year, Just-

Films had more lms in the 2012 Sundance

Film Festivals documentary competition

than any other producer, and one o ourlms won the award or Best Documentary.

ALLOCAte ReSOuRCeS In A dIFFeR-entIAted And dynAmIC mAnneRWith clear objectives, strategies, and op-

erating approaches in place, the central

question we asked was what scale o re-

sources was required or each strategy to

eectively achieve the results required.

From the start, the oundation allocated

resourcesnancial, sta, and other, such

as public voiceaccording to the nature

and scale o its objectives. We change that

allocation dynamically, on the basis o stra-

tegic need. As a result, the level o resources

can vary greatly across initiatives and over

time. Through ongoing assessments, we

engage in a dialogue with program sta

about whether our investments should be

augmented to achieve signicant results, or

reduced to allow or uture exploration and

course correction. This process creates adynamic internal marketplace o ideas and

inormation that inorms the evolution o

our grantmaking strategies.

For example, using results-based strate-

gic assessment and adjustment recently al-

lowed us to launch a $200 million investment

to help shit the approach o urban develop-

ment rom one ocused on isolated areas o

urban need to one ocused on regional solu-

tions that link potential workers to centers o

employment and housing. The goal is now

to tie economic opportunities to communi-

ties in need through better transportation,

housing, and zoning policies.

Once initial resources are allocated in

a dierentiated way, we immediately ace

the question o how to reallocate resources

as initiatives evolve, succeed, and, ail. A

results-ocused culture must integrate

course-correction methods, acknowledg-

ing successes and ailures and adjusting

resources accordingly. This process beginsby answering ve important questions:

. Does the initiative or eld oce havea clearly dened strategy and ap-

proach with explicit and achievable

objectives that can be documented

and evaluated?

2. Can additional unding or a denedperiod deepen or hasten desired re-

sults over time in an initiative or oce?

A culture ocused on results begins with a

clear understanding o objectives and the

development o strategies to achieve them.

7/31/2019 Advancing Evaluation Practices in Philanthropy

6/24

Sponsored Supplement to SSIR

AdvANCiNg EvALUAtioN PRACtiCEs i N PHiLANtHRoPy

3. Are the unds currently allocated be-ing deployed strategically? Have pe-

ripheral activities been eliminated?

4. To what extent has the initiative teamsuccessully engaged other partners,

including other oundations, busi-

nesses, and government?

. Is the initiative team strategically usingthe oundations non-grant resources

intellectual, convening, and communi-

cationsto pursue the initiatives goals?

The answers to these questions help

guide decisions about which initiatives are

best positioned to deploy additional unds

strategically, as well as which ones require

additional unds to achieve more signicant

results. But the answers may also lead to a

decision to change course or to end activities.

It is essential to use the same dieren-

tiated, dynamic approach in the allocation

o sta and other non-grant resources that

add value beyond direct nancial support.

One example is the strategic use o commu-

nicationsa core resource that is oten let

out o the results equation. Each strategy

must consider i and how the use o com-

munications activities is critical to achiev-

ing its desired goal. At the Ford Foundation,

we think about the deployment o external

communications assetshow we allocate

the oundations voicevery careully as

an integral part o how we work.Achieving results oten pivots on our

ability to communicate eectively and

strategically about promising solutions to

complex challenges that are eective and

realistic. An early example came in Novem-

ber 2010, when a critical mass o people

gathered in Cancun, Mexico, or the United

Nations Climate Change Conerence. The

Ford Foundation, along with our grantees,

saw this convening as an opportunity to

raise the prole o community orestry in

Mexicoa little known but signicant suc-

cess story in cutting greenhouse gases. Aoundation-supported media campaign

oered journalists a chance to visit com-

munity orestry sites in Mexico, generat-

ing important media coverage beore and

during the conerence. To complement this

eort, I authored an opinion article, which

ran in US and Mexican media, on the need

to invest in sustainable orestry programs

that respect and promote community

stewardship. Using the oundations public

voice continues to be a critical asset toward

achieving results in our work in commu-

nity orestry and other grantmaking areas.

BuILd In ACCOuntABILIty BASedOn CLeARLy deLIneAted ROLeSAnd ReSPOnSIBILItIeS

Clearly dening roles and responsibili-ties is central to a results-ocused culture.

For senior management, this kind o clar-

itycomplementary to clarity in strategy

and approachsupports the robust level

o oversight and partnership that is vital

to eectively align resources with goals.

Clarity about responsibilities is an essential

precursor to accountability.

At the Ford Foundation, we have expanded

and dened the role o directors to give them

explicit responsibility or initiatives. On aver-

age, a director oversees three initiatives, su-

pervising their teams and budgets. Our repre-

sentatives lead the work in our many regional

oces, collaborating with primarily, though

not exclusively, New York City-based direc-

tors to determine which o our initiatives aremost needed in a specic geographic area and

to ensure that initiatives are implemented e-

ectively. Although many people are involved

in grantmaking, responsibility or outcomes

in a particular initiative or geographic area

sits squarely with the team o directors and

representatives involved.

Our expectations o grantees are high.

We demand the best rom them, and as

oundation leaders we need to demand the

best rom ourselves as well. Every member

o the Ford Foundations sta, rom senior

leadership to the newest hire, is expected torecognize that being part o the organiza-

tion means holding himsel or hersel to the

highest standard in ensuring results. Each

sta person must know that, like grantees,

she or he is accountable or results.

Tying external results to internal out-

comes is central to the development o a re-

sults-ocused culture. In every organization,

high perormance can be driven by rewards

and incentives. In the or-prot sector this

oten takes the orm o increasing economic

rewards. Nonprot organizations and oun-

dations have a dierent model but must be

no less committed to creating a culture that

rewards perormance and contribution to

orwarding the mission. It is critical to ac-

knowledge high perormers who achieve

disproportionate resultsinnovative leaderswho oster internal and external innovation,

build movements, and establish national

and international dialogues as a new norm.

Our ability to reward such high perormers

is no less powerul than in the private sector

because we can provide the added resources

that these leaders need to deepen their work.

At the same time, any culture that ocuses

on perormance results and standards o ac-

countability must have an equally strong

organizational commitment to sta devel-

opment. Philanthropic organizations must

invest in building the skills and organiza-

tional tools needed to evaluate progress

or each initiative and or each individual

grantmakerindeed, or every member o

the sta, rom administrative assistants tothe president. Every willing staer has the

potential to be a change leader.

An important caveat, as we think about

results, is the need to remain aware o the

risky nature o our work. Many o our most

important initiatives are extraordinarily

demanding, and their outcomes are highly

uncertain. We ace the challenge o main-

taining high standards o accountability in

an environment where speculative inno-

vation is encouraged and oten necessary.

In a results-ocused culture, it is critical

or leaders to make positive use o noble,well-planned, and well-executed eorts

that nonetheless were not able to succeed,

recognizing and understanding that ailure

is oten the potent seed o uture success.

Put A PRemIum On deePAnd eFFeCtIVe LIStenIngA results-ocused culture puts a premium

on listening. At the Ford Foundation, we

know that our grantees have the deepest

We believe that our early eort to defne

and instill a culture o results has relevance

or other oundations, both large and small.

7/31/2019 Advancing Evaluation Practices in Philanthropy

7/24

Sponsored Supplement to SSIR

7AdvANCiNg EvALUAtioN PRACtiCEs i N PHiLANtHRoPy

understanding o the issues that aect the

people they serve. That is why we support

initiatives managed by those living and

working closest to where targeted prob-

lems are located. We recognize the critical

need to maintain a rich dialogue with our

grantees and others in the elds and com-

munities in which we work.The Ford Foundations worldwide net-

work o oces is designed to help us real-

ize this aspiration by ostering both ace-

to-ace and virtual meetings. In a typical

year, in our New York City oces alone, we

host more than 0,000 visitors who come

together with our teams to work on existing

initiatives and to explore new ones. Invest-

ing time in listening and learning enables

us to make real-time strategic adjustments

and to plan highly inormed uture direc-

tions. This emphasis on listening, in turn,

enables us to improve how we are working,

and demands o ourselves the same excel-

lence we demand o our grantees.

Deep and eective listening is more about

shared dialogue than inormation gathering.

Traditional reporting and data collection can

sometimes be excessive and ultimately unin-

ormative or misleading, at times yielding ad-

versarial relationships with grantees rather

than a constructive collaboration. Excessive

data gathering has several costs. First, it is a

burden to grantees, creating surplus work or

oten tightly staed and nancially strappednonprots. Second, it undermines quality,

because grantees will provide the requested

inormation to meet their grant obligations,

but may not have time to supply the insight

that is oten more valuable than this orm o

data. Third, it inundates oundation sta with

inormation but may leave them little time to

use it eectively. Fourth, it may not provide

the inormation that is actually needed to

understand how eective our initiatives and

grantmaking are.

Data reporting and design do play an

important role in building a culture o re-sults, but just as resources must be stra-

tegically targeted, so too must reporting

requirements be ocused and shaped by

certain basic principles. The inormation

and indicators collected must be aligned

with program goals and strategies to e-

ectively show whether an activity is on

the orecasted path or has deviated rom it.

From the very beginning, grantees should

have a clear sense o what benchmarks o

success are expected o them at each stage

o initiative development. Such inormation

and indicators are not exclusively or always

quantied or quantiable. Thereore we

must be creative in articulating what kind

o inormation will allow us to make these

assessments and subsequent decisions.

The skill o deep listening is vital notonly to our individual initiatives, but to our

organization as a whole. It is a central part

o the culture anchored in our initial strat-

egy-setting eort. At the Ford Foundation,

we engaged more than ,000 grantees and

other experts in our strategy-resetting eort

in 008, and we believe every one o them

played a role in reshaping and revitalizing

our work. Goals, theories o change, and

operating approaches are all necessarily

imperect; only by learning rom our suc-

cesses and our mistakes can we build an e-

ective culture o results. Our objective, as

we listen to grantees and others, should not

be merely to gather data and evaluate others

but to learn about ourselves and to evolve.

ImPLement A ReSuLtS-FOCuSedCuLtuRe ACROSS the entIReORgAnIzAtIOnLeading a oundation is a complex task. At

the Ford Foundation, we make more than

$500 million in grants and contribute to

change in a range o other ways, rom pro-

viding individual and team leadership, tobringing voice and convening, to building

awareness and partnerships. But excellence

in these dimensions does not adequately

dene a results-ocused culture.

Our success in implementing and sup-

porting program initiatives depends on

the oundations outstanding day-to-day

operations and exceptional endowment

management, and consequently we have

worked hard to ensure that our culture o

results reaches beyond the program team

to include all sta. A philanthropic orga-

nization must bring to all o its operationsthe same level o rigor that it uses to dene

strategy, allocate resources, gather inorma-

tion, and evaluate results.

More than $00 million o the ounda-

tions 0-grantmaking was the direct re-

sult o the changes we made in operations

and endowment management in 2008 and

2009. For example, shiting $40 million

rom internal operating costs to external

grantmaking in 2009 enabled us to launch

and ully und the programs discussed in

this article. In addition, careul stewardship

o our endowmentincluding changing our

allocation o unds and the management o

those allocationshas allowed it to ully

recover the investment losses that resulted

rom the economic downturn. The decrease

in our endowment that has occurred sincethe beginning o the recession is now exclu-

sively the result o our grantmaking activity.

WhAt We hAVe LeARnedThe Ford Foundation has a complex global

ootprint and a broad scope. Nonetheless,

we believe that our early eort to dene

and instill a culture o results has relevance

or other oundations, both large and small.

The lessons we have learned over the last

several years and the ve principles we

have put in place have charted a well-de-

ned course or the oundation.

Although the ve principles are now

clear, nothing about our endeavor to build

a culture o results has been easy. Listening

means hearing criticism as well as praise;

dynamic resource allocation oten means

making hard and even painul choices; and

a commitment to accountability means

taking responsibility or perormance or-

ganizationally and individually. But when

we consider the benets o this careully

developed approacha greater ability to

help the people we servewe think they aroutweigh the challenges o implementation.

As the president o the Ford Foundation,

leading a results-ocused culture is a con-

stantly evolving challenge. Very little is static.

Strategies change and resources shit as

strategic need and results demand. Account-

ability is maintained, based on individual and

team perormance. Grantees and others need

to be continuously engaged and heard.

A culture ocused on results is not al-

ways comortableit has many o the

stresses o a marketplace that demands e-

ectiveness and eciency. In a time denedby technological change and economic dis-

continuity, however, we have no choice but

to demand the most o ourselves. Cultural

transormation is, undeniably, hard work,

but the rewards o such an endeavor are

great. Any leader trying to achieve strategic

results at a oundation or nonprot should

consider the hard-won but invaluable ben-

ets o building a culture that osters, rein-

orces, and rewards that goal.s

7/31/2019 Advancing Evaluation Practices in Philanthropy

8/24

Sponsored Supplement to SSIR

8AdvANCiNg EvALUAtioN PRACtiCEs i N PHiLANtHRoPy

Learning romSilicon ValleyHow the Omidyar Network uses a venture capital model to measure and evaluate eectiveness

By Matt Bannick & eri c Hallstein

When visitors step out o the

elevator at the Omidyar

Network (ON) oice in

Redwood City, Cali., therst thing they see is a quotation on the wall:

Every person has the power to make a di-

erence. A belie in the undamental value

and power o each individual shapes nearly

every acet o ON, the philanthropic invest-

ment rm created and unded in 00 by

eBay ounder Pierre Omidyar and his wie,

Pam. At ON, we strive to create opportuni-

ties or individuals to unlock this power.

ONs approach to measurement and eval-

uation is inseparable rom our approach to

philanthropy. We start with the notion thatunlocking the potential o individuals is an

indispensable way to improve the world.

Given opportunities, people will tap their

talents to improve their lives and conse-

quently, the lives o our amilies and com-

munities, which ultimately benets society

more broadly.

Omidyar Network strives to scale up in-

novation by applying many o the best prac-

tices rom the venture capital industry to

philanthropy. In rapidly changing markets,

we believe the truly game-changing innova-

tions will emerge rom entrepreneurs whoare empowered to identiy changing situa-

tions and rapidly adapt their organizations.

Our investment model is to invest in and

then help scale up innovative nonprot and

or-prot organizations with the potential

to create opportunity or hundreds o thou-

sands or millions o people. Instead o und-

ing others to implement programs related to

our strategy, we seek to enable entrepreneurs

to identiyand iterate ontheir own strate-

gies or creating social impact.

meASuRement And eVALuAtIOnON has tried many approaches to mea-

surement and evaluation over the last ew

years, and rom that experience we are

convinced that an organizations approach

to measurement and evaluation must fow

rom its investment model. ONs approach

is to conduct an up-ront, detailed due dili-gence process to assess an organizations

potential and decide whether to invest. We

then develop trusted partnerships with our

investees to measure their progress in real

time and provide additional, non-nancial

support as appropriate.

ON provides grants to nonprot organi-

zations and invests debt and equity capital in

or-prot ventures. ON is an impact investor

whose bottom line is social value. We take a

systems view; we are driven by the problem

that needs to be solved and have the fex-

ibility to tailor the highest impact solutionusing the most appropriate type o capital.

Philanthropy through grants can benet

society in many ways. Grants can help non-

prots provide pure public goods. For exam-

ple, the ree legal tools oered by Creative

Commons, which allow content creators

to grant copyright permissions to creative

work, increase the amount o content avail-

able to the public or sharing. Grants can help

subsidize goods and services that produce

positive societal outcomes but are undersup-

plied by the market. For example, Endeavor

is helping to transorm emerging markets by

supporting entrepreneurs. Grants can help

disadvantaged populations that are unable

to participate in the market, as Landesa and

other property rights organizations do. And

grants can spur investments in high-risk

ventures, as they did with the initial develop-

ment o the micronance industry.

Philanthropy through or-prot ven-

tures, in contrast, can benet society by

leveraging the power o markets. When the

primary motive is to generate prots, busi-

nesses will strive to deliver value in excess

o costs and scale up. Customer willingness

to pay or a product or service not only in-dicates value creation, but also serves as a

valuable eedback mechanism.

ON has ound that grants and or-prot

investments can play complementary roles

in delivering social impact, even, or maybe

particularly, in diicult socioeconomic

environments. Socially responsible grant

capital can help potentially high-impact in-

novation gain market traction, and or-prot

equity and debt capital can then help that

idea scale up.

The phenomenal growth o micronance

is a good example o such complementaryroles. Since 2004, ON has invested more

than $00 million in micronance orga-

nizations: 15 nonprots and 11 or-prots.

In the 980s and 990s, most micronance

institutions were grant-unded NGOs. As mi-

cronances impact and commercial viability

became apparent, business investorsmany

with strong social motivationsinvested

heavily in commercial micronance institu-

tions, helping them grow rapidly. According

mAtt BAnnICk is managing partner at OmidyarNetwork, where he leads all aspects o the philanthropicinvestment frms strategy and operations. Beore joiningOmidyar, Bannick was president o PayPal and presidento eBay International.

eRIC hALLSteIn is a vice president at Imprint CapitalMarkets and a venture partner at CalCEF Clean EnergyAngel Fund. He was previously director o investments atOmidyar Network.

7/31/2019 Advancing Evaluation Practices in Philanthropy

9/24

Sponsored Supplement to SSIR

9AdvANCiNg EvALUAtioN PRACtiCEs i N PHiLANtHRoPy

to a 009 Monitor Institute study, the per-

centage o the worlds top 0 micronance

institutions that were or-prot banks in-

creased rom percent to percent be-

tween 1998 and 2008. In other words, grants

sparked and nurtured micronance, and

or-prot capital helped it scale up.

ON takes equity stakes in or-prot com-panies and typically provides negotiated

general operating support to nonprot or-

ganizations. This support gives organiza-

tions great fexibility to change their tactics

and reallocate their resources in response to

new inormation or changing market condi-

tions. It also enables them to invest in critical

overhead unctions, such as developing man-

agement talent and inormation technology,

which may otherwise not get unded because

these unctions are not directly related to

delivering specic programs.

A grant or investment is only the begin-

ning o ONs relationship with our portolio

organizations. Although vital or growth,

scale, and capacity, money alone cannot

solve every problem or bring about the posi-

tive social impact we seek. Pierre and Pam

Omidyars vision was to create an organiza-

tion that augments the impact o its nancial

investments by deploying human capital and

leveraging the knowledge, expertise, and in-

novation o all those involved in solving the

worlds most challenging problems.

When requested by a portolio organi-zation, ONs human capital team provides

strategy, governance, and leadership as-

sistance tailored to meet the grantees or

investees needs. The team also creates

opportunities or its investees to tap the

power o the network by regularly organiz-

ing orums or them to exchange inorma-

tion and share best practices.

SeLeCtIOn CRIteRIA Anddue dILIgenCeONs investment selection criteria and pro-

cess provide a oundation or measurementand evaluation o our investments. To make

investment decisions, we consider actors

such as innovation, scalability, and viability

o business or revenue models. ON encour-

ages nonprot investees to develop earned-

income streams whenever possible. Earned

income is oten a powerul tool or ensuring

that a nonprot ocuses on understanding its

constituents (customers); benets rom clear,

market-based signals about the value o its

product or service; and delivers products

and services with values exceeding costs.

Like traditional venture capital rms,

ON perorms extensive due diligence be-

ore investing in an organization. Although

traditional approaches to measurementand evaluation do not view due diligence

as evaluation, organization-wide, up-ront

due diligence is central to our approach. In

considering a unding investment, we care-

ully assess the organizations management

team, operational planning and governance,

target market and competition, technology,

product or service, and nancials. We meet

with the organizations management team;

interview customers, key channel partners,

and board members; read industry and peer-

reviewed reports; and analyze company -

nancial and operating data.The due diligence process is primarily

about whether to invest, but it also aords

an opportunity to explore strategically im-

portant issues that ultimately help ON sup-

port an organization. Detailed discussions

with an organizations management team

help to ensure strategic alignment between

ON and the investee. During this process

the management team and ON agree on

the metrics that will be used to evaluate the

organizations progress. Due diligence also

helps ON determine how best to use its hu-

man capital to support an organization. Thus

the diligence process has a direct bearing on

all subsequent measurement and evaluation.

ReACh And engAgementON, like all philanthropic unders, consid-

ers social impact the single most important

investment criterion. Although our specic

metrics vary across sectors and organiza-

tions, we have developed two overarching

metrics that help us understand the social

impact o portolio organizations: reach

andengagement. Reach is a measure o how

many individuals are touched by a product

or service. Engagement is a measure o the

depth o that interaction.

Take or example the nonprot Wikime-dia Foundation, which operates Wikipedia.

Wikipedia engages hundreds o millions

o people in creating educational content,

sharing inormation, and learning online.

In 2010, a measure o Wikipedias reach was

the roughly 400 million dierent people who

visited its website. A measure o engagement

was the 13 minutes that an average visi-

tor spent visiting Wikipedia every month.

Other measures o reach and engagement

7/31/2019 Advancing Evaluation Practices in Philanthropy

10/24

10AdvANCiNg EvALUAtioN PRACtiCEs i N PHiLANtHRoPy

Sponsored Supplement to SSIR

or Wikipedia include the number o active

contributors o content and the level o their

activity on the site.

Neither the reach nor engagement metric

directly measures Wikipedias social impact,

however. To determine impact we would need

to understand the extent to which a product

or service aects the end user. In this case, apotential impact metric or Wikipedia would

be the improvement in students test scores

ater they use the site. Accurate estimation

o Wikipedias impact would require an ex-

perimental or quasi-experimental approach,

typically a randomized trial in the absence o

a natural experiment, and might take months

or even years to perorm.

As an alternative to the time-consuming,

costly, and complicated challenge o directly

measuring impact, we oten measure out-

putsreach and engagementas proxies

or impact. Outputs are units o production,

which oten can be readily measured by an

organization in the normal course o business.

Our use o a very limited number o eas-

ily collected metrics is a departure rom

some o our early evaluation rameworks

in which we asked entrepreneurs to track

as many as 18 dierent metrics. The earlier

approaches not only were more costly to

implement, but had less impact on invest-

ees success because neither they nor ON

was able to prioritize appropriately.

We have learned that an organizationssocial impact is best estimated using a com-

bination o reach and engagement metrics

tailored to the industry, the organizations

business model, and its growth stage. Some

organizations, such as Wikimedia, have wide

reach and relatively limited personal engage-

ment. Other organizations, such as Endeavor,

have modest reach and deep engagement.

Selection o appropriate reach and engage-

ment metrics can be dicult and nuanced.

SCALABILIty And SOCIAL ImPACt

ON invests in highly scalable solutions withthe potential to reach millions o people,

and in which costs per unit generally de-

crease as the volume produced increases.

Because organizations need a reliable

revenue source to achieve scale, nancial

sustainability is a ocus o our evaluation

processbut there are some important di-

erences in how we think about nancial

sustainability in or-prots and nonprots.

For ON to consider investing in a or-

prot company, the critical rst lter is its

potential or positive social impact, and the

second is its ability to protably and quickly

scale up. We look at nancial metrics that

are standard or a prot and loss statement,

such as revenue, operating margin, and net

income. In particular, operating income

is a leading indicator o impact because itdemonstrates a companys ability to deliver

value to customers above cost.

In the or-prot organizations that ON

unds, nancial metrics are indicators o

nancial sustainability and social impact.

A companys revenue and margins are

driven by the number o customers served

and how highly those customers value the

product, both o which provide a quantita-

tive measure o the opportunities a com-

pany is creating.

To supplement standard prot-and-loss

measures, ON uses metrics captured by

the companytypically, key perormance

indicators that a or-prot company tracks

to understand the underlying drivers o

its nancial perormance. At Bridge In-

ternational Academies, a Kenyan startup

that operates a ranchise-like network o

low-cost, or-prot private schools, these

metrics include the number o schools, the

number o students and trained teachers,

academic perormance, and the turnover

rate o students and teachers.

ReVenue StReAmSIn nonprot organizations, social impact is

the most important evaluation criterion, and

we spend considerable time with investees

to dene reach and engagement metrics that

measure it. We also encourage and work with

nonprot investees to develop diversied

sources o revenue as well as to generate

earned income and other revenue sources

to increase their nancial sustainability.

Earned income is valuable to nonpro-

its not only because it provides revenue to

cover operating costs, but also because it isa conduit or clear, rapid, and market-based

eedback about whether the organization is

providing a highly valued product or ser-

vice. In this sense, not all revenue is equal.

We view revenue tied to an organizations

products and services to be the highest value.

Similar to revenue and prot in or-prot

organizations, market-based signals tied to

products and services can be a powerul

mechanism or getting nonprot manage-

ment teams to prioritize the activities o

highest value to the end user. Market-based

eedback also is likely to be more useul and

valuable or both ON and investees because

it inorms real-time decisions without wait-

ing or inrequent ormal reviews.

For example, consider GuideStar, a non-

prot in which ON rst invested in 00.GuideStar oers inormation on 1.8 million

US nonprots and private oundations and

had more than 10 million website visitors in

2010. Although 98 percent o GuideStar data

uses are ree, users pay a ee or more so-

phisticated tools and services. Fee revenue

covered 90 percent o GuideStars operating

costs in 00. Charging ees or premium

services has given GuideStar resources to

grow quickly and enabled its management

team to ocus on delivering high-quality

services to paying customers rather than

raising money through donations. Guide-

Star now uses oundation grants primarily

or special projects and opportunities.

Several o ONs portolio organizations

have developed innovative donation models

that are a twist on the earned-income model.

For example, DonorsChoose.org matches do-

nors with school classroom projects in need

o support. When donors support a specic

project, DonorsChoose.org invites them to

contribute a small amount toward the costs

o running the organization. In scal 0,

DonorsChoose.org covered a remarkable 102percent o its operating expenses through

this optional ee. Voluntary donor support o

operating ees is a type o immediate eed-

back indicating the value the donor places

on a product or service.

COSt-tO-SeRVeCost-to-servethe cost o providing a

product or service to a single customeris

another metric that we typically track in

or-prot and nonprot investees. Cost-to-

serve is an important measure o produc-

tivityspecically, how eciently an orga-nization is delivering a particular product

or service. In general, we nd that despite

its potential value or driving nancially

sustainable growth, ew nonprots ocus

on reducing cost-to-serve.

We calculate the cost-to-serve as the quo-

tient o reach and the trailing months o

operating expenses. Cost-to-serve can vary

signicantly across organizations. For ex-

ample, we estimate that in 2010, Wikimedias

7/31/2019 Advancing Evaluation Practices in Philanthropy

11/24

11AdvANCiNg EvALUAtioN PRACtiCEs i N PHiLANtHRoPy

Sponsored Supplement to SSIR

cost-to-serve each visit to its website was

$0.02, whereas DonorsChoose.orgs cost-to-

serve was $5.23, or roughly 250 times greater.

Cost-to-serve can oer some insight

when comparing two businesses with similar

models in the same sector, but is less useul

when comparing across sectors. Typically,

we look at an organizations cost-to-serve tohighlight opportunities or eciency or asset

productivity improvements. DonorsChoose.

org, or example, is working on reducing its

cost-to-serve by lowering new user acquisi-

tion costs and increasing the donor retention

rate. Bridge International Academiesa new

organization with relatively high startup-re-

lated overheadhas high costs relative to the

number o students served. As Bridge opens

more schools and benets rom the result-

ing economies o scale, we expect the cost-

to-serve each student to drop dramatically.

COLLABORAtIVe eVALuAtIOnA successul relationship between ON and

each organization is highly collaborative

and based on trust. Collaboration starts

with the due diligence process, through

which ON evaluates the leadership teams

ability and commitment to practice exem-

plary governance, operational eciency,

transparency, and disciplined nancial

planning. This process builds relationships

between ON personnel and the investees

management team, establishing the basisor a positive, long-term relationship.

The spirit o collaboration and trust car-

ries over to the measurement and evalua-

tion process. Our investment proessionals

work closely with each investee to develop

milestones jointly and then measure the in-

vestees progress against them. For-prot

companies usually track the desired reach

and engagement metrics during the normal

course o business. For some consumer In-

ternet companies, or example, the number

o unique visitors to a site (reach) and num-

ber o page views per visit (engagement)are critical revenue drivers. The organiza-

tions perormance relative to established

milestones is an important criterion or ON

when we consider subsequent investments,

but specic impact metrics typically are not

captured in any agreement between ON and

a or-prot company.

In nonprots, ON rst engages the lead-

ership team in a dialogue about the organi-

zations potential social impact, and then

works with the team to speciy appropriate,

related reach and engagement metrics that

are codied in the grant agreement. These

milestones provide a mechanism to both

hold the organization accountable or achiev-

ing its agreed-upon objectives and identiy

when it might need additional human capital

support rom ON to be successul.ONs investment proessionals regularly

review each investees perormance against

these metrics. ON holds board seats or board

observer rights in about 0 percent o our

portolio organizations, and metrics are usu-

ally reported by the management team at

board meetings. I ON does not hold a board

seat, the grant agreement typically requires

the investee to provide quarterly reports o

its progress against the metrics.

To supplement ongoing perormance

tracking and comprehensively evaluate

our investees progress, ON holds an an-

nual, internal portolio review. The review

covers the perormance o each portolio

organization and the overall perormance

o all organizations by sector.

ONs human capital contributions keep

us in requent contact with our portolio or-

ganizations management teams. The close

proessional relationships with investees

and the tight integration through these gov-

ernance roles engender a sense o shared

commitment between ON and each investee,

building a oundation or eective, real-timeevaluation and response. The requent con-

tact creates a continuous fow o inormation

between ON and the investee, providing a

substantial context or important decisions.

Equally important, ONs investment proes-

sionals have the added benet o being able

to look broadly across the entire market and

competitive landscape, and can highlight

specic issues that a more narrowly ocused

management team might miss.

WhAt WeVe LeARned

ONs investment model undamentallyshapes how we approach measurement and

evaluation. Developing and applying ONs

approach has taught us to:

Foster partnerships. Trust-based part-

nership is the key to establishing a pro-

ductive approach to measurement and

evaluation. ON osters partnerships

by listening careully to our investees

needs rom the beginning o the due dil-

igence process and investing heavily in

our human capital unction.Exercise judgment. Data is rarely a

substitute or good judgment. Only

so much can be measured, much that

matters cannot be measured, and any

amount o data alone cannot provide

all the answers. ON invests in talentedmanagement teams and works with

them to strike the right balance be-

tween the robustness and complexity

o a companys approach to evaluation.Be fexible. Markets, competitors, and

companies can change rapidly. What

is important to measure today may not

be meaningul tomorrow, so we cannot

rely on static measurements in a dy-

namic environment.Embrace eedback.We manage what

we measure. ON takes the time to un-

derstand what the data says about a

companys product or service. We then

help our investees use that inormation

to improve their product or service to

deliver even greater value to custom-

ers, who will ultimately determine

their success or ailure.

At ON we continue to discuss and debate

many topics related to how we approach

measurement and evaluation, including

the appropriate role or independent im-

pact assessments, the optimal resourceallocation across sectors and companies,

the best approach to meaningul inorma-

tion and data sharing across unders, and

whether to evaluate ONs contribution to

helping shit sectors.

Our view is that an organizations ap-

proach to measurement and evaluation

fows directly rom its investment model. ON

regularly re-evaluates whether its investees

are having a positive impact on the sector

and whether other investments are neces-

sary to break bottlenecks. ONs approach to

measurement and evaluation thus remains awork in progress. We continue to iterate and

innovate, jointly with our colleagues in the

impact investing sector, with the objective

o investing in and scaling organizations

that have the ability to create opportunity

or millions o people.s

The authors wish to thank Tie Kim and Gowri Pai orresearch assistance; Je Bradach, Suzanne Lippert, PaulaGoldman, Annis Steiner, and Greg Pershall or editorialcomments; and Arjuna Costa, Chris Bishko, Amy Klement,Bill Barmeier, and Todor Tashev or valuable discussions .

7/31/2019 Advancing Evaluation Practices in Philanthropy

12/24

1AdvANCiNg EvALUAtioN PRACtiCEs i N PHiLANtHRoPy

Sponsored Supplement to SSIR

Shared OutcomesHow the Rockeeller Foundation is approaching evaluation with developing country partners

By JuditH rodin & nancy MacPHerson

Across every sector o society,

decision makers are struggling

with the complexity and velocity

o change in an increasingly in-

terdependent world. The context o decision-

making has evolved, and in many cases has

been altered in revolutionary ways. In the

decade ahead our lives will be more intensely

shaped by transormative orces, includ-

ing economic, environmental, geopolitical,

societal, and technological seismic shits.1

As part o its response to this global dy-

namism, the Rockeeller Foundation has

translated its 1913 mission o promoting the

well-being o humanity into two overarch-

ing goals: expanding opportunity through

more equitable growth, and strengthen-

ing resilience to acute crises and chronic

stresses, whether man-made or ecological.

Our vision is a world in which globaliza-tions benets are more widely shared and

the inevitable challenges that accompany

a world that is ast changing, diverse, and

complex are more easily weathered.

The Rockeeller Foundation structures

its work around time-bound cross-sectoral

initiatives that seek innovative solutions

and support enabling environments to

bring about change. The oundations struc-

ture refects its view that todays problems

and solutions are multi-dimensional in

scope and nature, and that they require

multi-disciplinary responses at the inter-section o elds.

Just as the Rockeeller Foundations

approach to philanthropy has evolved, so

too has its approach to evaluation. With its

mission to improve the well-being o hu-

mankind, its ocus on impact, and much o

its grantmaking in developing countries,

the Rockeeller Foundation is commit-

ted to evaluation practices that are rigor-

ous, innovative, inclusive o stakeholders

voices, and appropriate to the contexts in

which the oundation works. This article

discusses how the Rockeeller Foundation

integrates the views o developing-region

evaluators into its evaluation approaches,and highlights ve key strategies:

. Engaging stakeholders to developshared outcomes.

2. Expanding capacity through use onon-sta monitoring and evaluation

specialists to partner with grantees.

3. Sharing knowledge through learningorums and communities o practice.

4. Strengthening developing countryevaluation practice and ownership o

results.

. Developing innovative methods andapproaches to evaluation and learning.

RethInkIng, ReShAPIng,And R eFORmIng eVALuAtIOnIn November 2011, the Rockeeller Founda-

tion brought together leaders rom philan-

thropy and development at the Future o

Philanthropy and Development orum, held

at its conerence center in Bellagio, Italy.

In one o the keynote papers, Evaluating

Development Philanthropy in a Changing

World, Robert Picciotto, ormer vice presi-

dent at the World Bank and now proessor

at Kings College London, squarely tackled

the role o evaluation in philanthropy and

development. The changing context and

thinking on development has proound

implications or development evaluation

itsel, and or the contribution evaluation

can bring to the empowerment o people;

and the eectiveness o development in-

terventions by national governments and

international partners and, increasingly, bynon-state actorsoundations, philanthro-

pists, and agencies that promote investing

or impact.

Picciotto continued, Pressing human

needs are not being met by an ocial aid

system that is short o resources, catering

to multiple interests, and hobbled by mas-

sive coordination problems. By contrast,

private giving or development is growing

and has proven nimbler and more results-

oriented than ocial aid. However, the

philanthropic enterprise will not ulll its

potential unless it identies and taps intoits distinctive comparative advantage and

coordinates its interventions with other de-

velopment actors; embeds evaluation in its

processes to achieve operational relevance,

eectiveness, and eciency; and demon-

strates that it is accountable and responsive

to its diverse stakeholders.

Developing country evaluation leaders

have also articulated the need or a new ap-

proach to evaluation and the role it plays in

JudIth ROdIn is president o the Rockeeller Founda-tion. Beore joining the oundation, she was president othe University o Pennsylvania, and preceding that, pro-vost o Yale University. Rodin is on the board o Citigroup,Comcast, and AMR.

nAnCy mACPheRSOn is managing director, evalu-ation, at the Rockeeller Foundation. Beore joining theoundation she spent 25 years in development evaluationor international nonproft organizations in Asia andArica and or the United Nations.

The value o evaluation must ultimately be

judged by its useulness in helping to

improve outcomes or target benefciaries.

7/31/2019 Advancing Evaluation Practices in Philanthropy

13/24

1AdvANCiNg EvALUAtioN PRACtiCEs i N PHiLANtHRoPy

Sponsored Supplement to SSIR

improving the wellbeing o humankindin

particular, the lives o the poor and vulner-

able in developing countries. At the Janu-

ary 2012 gathering o the Arica Evalua-

tion Associations biannual conerence in

Accra, Ghana, Arican evaluation leaders

and policy makers highlighted ve steps

oundations and development agenciesmust take i they aspire to play a meaning-

ul role in social transormation.

. Broaden the inclusion o key stake-holders in evaluation. Only when the

voices o those whose lives we seek to

improve are heard, respected, and in-

ternalized, will we be able to eectively

evaluate what success should look like

or the people we are most concerned

about. Foundations and agencies need

to take practical steps to include key

stakeholders in the design o, conduct

o, and learning rom evaluation.

2. Regard evaluative knowledge asa public good and share it widely.

Learning with our partners and

stakeholders about what works and

what doesnt work should be seen

as a global public good not limited

to boards and program teams, but

shared widely with grantees, part-

ners, and peers.

3. Address evaluation asymmetries

between developed and developingregions. The majority o human and

nancial resources or evaluation em-

anate rom agencies and oundations

based in the developed world. With

evaluators rom developing countries

playing a minor role, i any, many o

them do not get sucient experience

to move into leadership roles. Men-

toring, coaching, and training can

strengthen the role, capacity, and re-

sources o developing-country evalu-

ators so that they can play key roles

in conducting and using evaluationresults or social transormation and

accountability in their own countries.

4. Broaden the objects o evaluation tolearn more beyond the individual

grant or project to a more strategic as-

sessment o portolios o investments,

policy change, new nancing mecha-

nisms, and sector-wide approaches

that tell us more about what works and

what does not in dierent contexts.

Framing evaluation to take into ac-

count the drivers o unsustainability

and causes o the challenge being ad-

dressed provides greater learning than

narrower evaluations that ocus only

on the unders specic intervention.. Invest in the development and appli-cation o innovative new methods and

tools or evaluation and monitoring that

refect multidisciplinary and systems

approaches to problems and complex-

ity; invest in methods that assess net-

work eectiveness and policy change;

and use and adapt new technology to

enable stakeholders to provide close to

real-time data and eedback.

the ROCkeFeLLeR

FOundAtIOnS APPROAChWith its long history o supporting devel-

oping country institutions,the Rockeeller

Foundation has responded to the call to ac-

n frm eelpn-ren ealuar by

adopting the ollowing approaches to plan-

ning, monitoring, and evaluating its work.

Shared Outcomes | An important un-

derpinning o the Rockeeller Foundations

initiative-based approach is a undamental

recognition that the worlds greatest chal-

lenges cant be solved alone. These chal-

lenges involve a complex mix o actors that

are oten globally interdependent across sec-

tors and geographies. Networks, alliances,

and coalitions o diverse stakeholders rom

governments, oundations, civil society, andbusiness are increasingly seen by the oun-

dation as a more powerul way to mobilize

the vast range o resources and actions re-

quired to bring about sustained and trans-

ormational change on a signicant scale.

Increasingly, the Rockeeller Founda-

tion brings together grantees and partners

rom developed and developing countries

to establish a common vision o the prob-

lem, outcomes, and indicators or success.

Grantee agreements now include reerence

to the common vision o results and shared

outcomes to which the grantee contributes,and oundation teams are expected to man-

age portolios o grants and relationships

with grantees towards that common vision.

This shared-outcomes approach orms the

basis or ongoing monitoring, evaluation,

and reporting, and or learning dialogues

with grantees and partners. (See Shared

Results Framework on p. .)

Monitoring and Evaluation | Most

oundations have capacity limitations on

7/31/2019 Advancing Evaluation Practices in Philanthropy

14/24

1AdvANCiNg EvALUAtioN PRACtiCEs i N PHiLANtHRoPy

Sponsored Supplement to SSIR

the amount o time that can be devoted to

monitoring and learning with grantees and

partners, visiting eld projects, and working

collaborativelyactivities that we know con-

tribute to greater collaborative learning and

eective relationships. Recognizing these

limitations, the oundation awards grants

to monitoring and evaluation (M&E) groupsand specialists in developing and developed

countries who act as monitoring partners, or

what we call critical riends, 2 throughout

the lie o initiatives (typically, a ve- to six-

year period). They work with grantees to

identiy key learning questions, help to set

up monitoring systems, and provide support

in analyzing monitoring data. The most sig-

nicant eature o the critical riends is that

they build trust with grantees and partners

to ask tough evaluative questions, and they

support grantees in seeking and using eed-

back to make improvements throughout the

lie o the initiative. Periodic evaluations are

conducted by independent teams to provide

an objective assessment o progress towardoutcomes and impact.

For example, the India-based nonprot

Participatory Research in Asia, in collabora-

tion with the Ghana-based Institute or Policy

Alternatives, works alongside Shack/Slum

Dwellers International (SDI) which directly

represents millions o urban poor slum dwell-

ers in countries. The aim o this critical

riend partnership is to strengthen the par-

ticipatory learning, monitoring, and evalua-

tion systems and abilities o the urban poor

networks to better capture and systematize

learning and strengthen accountability with

the goal o empowering the urban poor to

achieve wider positive impacts. The critical

riend role underpins a belie that ederationso the urban poor are capable o changing

their own situation or the better. As a result

o this partnership, SDI has strengthened its

ability to democratize learning, monitoring,

and evaluationcontinuing to place the tools,

responsibility, and ability or change in the

hands o its members.

Learning Forums and Communities o

Practice | Most o the Rockeeller Founda-

tions initiative teams convene grantees and

partners annually to review progress, high-

light lessons and challenges, celebrate suc-

cesses, and identiy improvements needed.

Through these orums grantees learn rom

others in the eld, meet new resource people,

and adjust their strategies going orward.

Although this practice does not guarantee

impact, it increases the likelihood o it by

creating a greater sense o ownership and

shared outcomes, and it increases leverage

by connecting grantees with new resource

people, unders, and mentors. Increasingly,

M&E grantees produce high-quality knowl-

edge products as a public evaluation good

to highlight what works, what does not, orwhom, and under what conditions. Our aim

is to establish with grantees a body o col-

laborative knowledge, shared lessons, and a

culture that values evaluation as a resource

or learning as well as or accountability.

For example, the Rockeeller Foundation

has aligned with the South East Asia Com-

munity o Practice in Evaluating Climate

Change Resilience (SEA Change), acili-

tated by the nonprot organization Pact, to

work on urban climate change resilience in

ten Asian cities. This community o prac-

tice brings together evaluators, programmanagers, grantees, and policy makers

concerned with learning what works in in-

terventions aimed at adapting and building

resilience to the eects o climate change

and extreme weather events in Southeast

Asia. Resources and lessons are shared

through online learning, onsite convening

o SEA Change participants, and coaching,

mentoring, and training provided by mem-

bers o the community o practice through-

Shared Results Framework

This gure illustrates the ramework around which Rockeeller Foundation staf,

grantees, and partners develop a common vision o the results and impact that they

seek to achieve collectively. The top rame represents the mission and strategy othe oundationpromoting the well-being o humanity in two overarching goals: ex-

panding opportunity through more equitable growth, and strengthening resilience.

The middle rame represents the medium-term outcomes that the oundation seeks

to achieve during the lie o the initiative (these change rom initiative to initiative).

The lower rame represents the work that grantees, partners, and staf do in their indi-

vidual organizations to collectively bring about outcomes and ultimately improve the

lives o benefciaries. These shared results rameworks anchor the ongoing dialogue

with grantees about progress toward achieving this vision and their contribution to

the shared outcomes. It also serves as a ramework or managing portolios o grants

and monitoring changes during the lie o the initiative.

Mission

& Strategy

Initiative

Outcomes

& Impact

Work of

Grantees,

Partners,

& Sta

Resilience

More Equitable Growth

Initiative Goal

Policy Networks Capacity

Non-GrantActivities

GrantsPortfolio

7/31/2019 Advancing Evaluation Practices in Philanthropy

15/24

15AdvANCiNg EvALUAtioN PRACtiCEs i N PHiLANtHRoPy

Sponsored Supplement to SSIR

out the countries o Southeast Asia.

Addressing Asymmetries | The Rock-

eeller Foundation is supporting the orma-

tion and strengthening o regional devel-

oping-country networks and the rst-ever

regional institutions to train, coach, and

mentor evaluators, and to partner with eval-

uators rom other regions. Through theseplatorms and networks, the oundation

aims to help rebalance the asymmetries

o choice and opportunity or developing-

country evaluators to control the evalua-

tion process in their own localities and to

improve the quality o evaluation by part-

nering with evaluation leaders globally.

One example o this is the Arican Evalu-

ation Association (ArEA), a pan-Arica um-

brella organization comprising more than 25

national M&E associations in Arica, and a

resource or individuals in countries where

national evaluation bodies do not exist.

ArEA, which has more than ,000 evalu-

ators rom all regions o Arica, receives

Rockeeller Foundation unding to enable

the ormalization o its organizational, op-

erational, and management structure, and

to build communities o practice among

its membership to tackle the most press-

ing evaluation challenges on the continent.

The Centers or Learning on Evaluation

and Results, located in Arica and Asia,

are another example o an eort aimed

at addressing asymmetries in evaluationin developing countries. Together with

a consortium o unders committed to

building developing country capacity or

taking charge o the evaluation agenda in

their regions, the oundation is supporting

regional centers3 in East and West Arica

and South Asia to strengthen their skills,

networks, and experience in monitoring

and evaluation and results-based manage-

ment capacity o public, private, and civil

society development in the global south.

New Methods and Approaches| Tradi-

tional evaluation methods and approachesto learning, accountability, and eedback

have not kept pace with the advances in

technology and social media. The majority

o evaluation practice is still largely paper-

based despite great strides in technology,

interactive web-based platorms, and mul-

timedia tools that make real-time eedback

rom grantees and beneciaries possible

and accessible. The Rockeeller Foundation

and its partners learned a great deal rom

Ushahidi, an open-source crowdsourcing

project that allows users to send crisis in-

ormation via mobile devices to map reports

o violence or suering. Inspired by the po-

tential o these kinds o tools to democratize

evaluation inormation, increase transpar-

ency, and lower the barriers or individuals

to share inormation and stories, the oun-dation is supporting a number o innovative

approaches to evaluation.

One example is GlobalGivings Story

Telling project, an innovative way to gather

local eedback rom people in developing

countries and to share it with communities,

implementing organizations, and donors to

create real-time eedback. With support rom

the Rockeeller Foundation, GlobalGiving

successully deployed a network o people

in Kenya, Uganda, and Tanzania that has

generated more than 20,000 tagged nar-

ratives rom thousands o people. Some o

GlobalGivings partners are deriving ac-

tionable intelligence rom these stories, and

GlobalGiving is discovering patterns in the

stories that inorm its own operational andstrategic decision-making processes.

Another example is BetterEvaluation, an

online interactive community o evaluation

practice developed by the Royal Melbourne

Institute o Technology in partnership with

the Institutional Learning and Change Ini-

tiative and the Overseas Development In-

stitute, with support rom the Rockeeller

Foundation and Pact. BetterEvaluation

provides advice, online support, and good

practice examples to evaluators in develop-

ing and developed countries.

ReShAPIng deVeLOPmenteVALuAtIOnPhilanthropists and development practi-

tioners have a golden opportunity to join

together with grantees and partners in de-

veloping countries to reshape evaluation

to better respond to global change and to

serve our missions and goals more eec-

tively. To do this, we must be prepared to re-

think and reshape our evaluation practice

in at least our ways. We must:

. Embrace a broader set o voices inraming our approaches to evaluation.

2. View collaboration and partnershipsbetween developed and developing

areas as mutually benecial toward a

common goal o expanding and shar-ing evaluation knowledge as a public

good aimed at achieving better devel-

opment outcomes.

3. Recognize the need to address issueso accountability, transparency, eth-

ics, culture, and independence.

4. Address asymmetries in individualand institutional capacities or under-

taking, driving, and owning evaluation

in developing regions by promoting

opportunities or proessional excel-

lence, networks, and sustained global

partnerships in the discipline o devel-

opment evaluation.

The value o evaluation must ultimately

be judged by its useulness in helping toimprove outcomes or target beneciaries.

The quest or impact is currently in the

spotlight among oundations and devel-

opment agencies as we seek collectively

to maximize the positive benets o our

resources. We are privileged to work in

an expanding eld in which our evalu-

ation ndings can change lives or the

better. Together with our peers, partners,

and grantees, we can and should rethink,

reshape, and reorm the practice o evalu-

ation to better meet that challenge.s

1 For more discussion o this topic, see Global Risks

2012, Seventh Edition, World Economic Forum, 0.

2 The term critical riend reers to a partner that

builds trust and engenders a refective evaluativeculture that is a