Impact Evaluation for policy-making
Promoting uptake of
impact evaluation
findings: the importance
of relevance, utility and
engagement
What 3ie is learning
Beryl Leach, deputy director, 3ie
18 February 2015
Istanbul, Turkey
Overview
• Who we are and how we promote high-quality,
relevant and useful studies that are taken up
and used
• What we are learning about impact evaluation
and policy influence
• How think tanks can produce, engage and use
impact evaluation in policy advocacy
What is 3ie?• International grant-making NGO founded in 2008
• Filling evidence gaps in what we know works,
why, how and the costs in development
• We fund particular types of evidence production
– high-quality experimental and quasi-experimental,
theory-based, mixed-method impact evaluations
– systematic reviews
– evidence syntheses
Why are IEs important for policy-making?
Policy relevance and usefulness
• Evidence that can improve the
effectiveness of development
policies and programmes
• Must be policy relevant
• We expect policy influence and
impact on policy and programming
because of how we approach
design and implementation
What do we mean by useful?
RELEVANT: Helps answer a
specific policy question
CONTEXTUAL: Appropriate
political context—it makes
sense
CLEAR: message gives
options
FEASIBLE: Affordable and
possible
TIMELY: interest exists
3ie funds high-quality policy-relevant studies
3ie encourages researchers to engage with key stakeholders
Uptake of study findings and improved policies and practice
Applications
- Ask policy
relevant
question(s)
- Have potential
for policy impact
- Study team has
experience in
policy influence
ASSUMPTIONS
Researchers
-Committed
- Understand how
policy influence
happens
- Have the tools
and resources to
invest in policy
engagement
- Decision-makers
are interested
Study
-Makes policy-
relevant
recommendations
– Answers main
IE questions
- Proposes
feasible
solutions
- Had ongoing
engagement
ASSUMPTIONS ASSUMPTIONS
What we learned from analysing early
influencing in our grants
• 3ie needed to be directly engaging with
researchers and much earlier
• Researchers needed to be engaging with
implementers much earlier and differently
• Paper-driven processes were not effective
• 3ie approach needed to be based on evidence
about policy change and research uptake
Main ways we are ensuring policy
relevance and usefulness
• Early and direct engagement with implementing agencies
during the development of funding windows
• Require country nationals on the research teams in
substantive roles; more funding is for country-based
teams – role and opportunity for think tanks
• Preparation phase
• Direct engagement with researchers about PIPs from the
start and ongoing dialogue with the team
Communicating impact evaluation
evidence: ongoing, integrated, multi-level
• Engage from the start
• Explain the study and why it will be useful--build interest
• Report preliminary findings-get feedback-promotes
ownership
• Engage a range of key actors: beneficiaries and other
local actors, civil society, media from early phases
• Translate into plain language and produce in a variety of
formats and disseminate through multiple channels:– Meetings, conferences, social media, multiple briefs, papers, reports
Think
Tanks
Impact evaluation and
think tank policy
influencing: opportunities
and constraints
Bridging researchers & policymakers
Translating, relationship building, networking
Using impact evaluation evidence for
policy advocacy: continued
• Constraints
• Two-sided coin– Rigor as the basis for being high quality
and for sound decision-making
– IEs seek to measure causal
relationships, we need to be able to
assess validity
• Two-part problem– Poor or weak designs mean you can’t
really know which cause
– Users have limited critical appraisal
capacity
Using impact evaluations for advocacy
• How much can you generalise based on one IE study?– Not much, even though we stress generalisability a lot
– More than IE evidence needed to know about potential for scaling up
– Researchers should always bring knowledge of existing evidence to the
analysis, so that the new evidence is situated in the existing body of
evidence, doesn’t always happen
• Evidence for programme change can be more
immediately useful than for policy change– Especially the case with theory-based evaluation, which is what we
require
– Modification to existing programme theory of change and operations
Benefits of increased production and use
of impact evaluations by think tanks• IEs particularly need a strong emphasis on policy relevance and
utility that think tanks can provide
• More think tanks doing quality IEs and being effective
intermediaries in policymaking will help increase uptake
• Participating in IE teams and commissioning can strengthen
relevance and utility
• Big potential for improving think tank understanding of IE evidence
and critical appraisal to be knowledge translators in advocacy
process
Thank you
www.3ieimpact.org#3ieNews #TTIX2015
END OF TTIX PRESENTATION
What are we looking at next?
• Developing our approaches to working with implementers
• Doing case studies on the cumulative impact of IEs done
over time: direct and indirect contributions to what types of
change
• Evaluating evidence uptake from 3ie studies– Finding an appropriate method and framework
– Exploring QCA
– Why we probably will not be doing IEs of policy influence
– Why we will look at synthesising evidence
Preparation phase-why is it so important
• Implementing agencies do not necessarily understand
impact evaluations
• Researchers do not necessarily speak the implementers’
language or understand their evidence needs well enough
• Fund working together
• Monitor through direct contact
• Have inception workshops
Policy influencing using impact
evaluations• Evidence does not even play a major role in most
decision-making
• Research uptake is a political process, not a technical one
• Researchers are vital to translating and building trust and
credibility
• Single study evidence most often does not result in major
policy change, nor should it
• 3ie does not advocate for wider change based on single
studies, nor should you
Limitations
• Expensive
• Work best with large
programmes
• Highly specialised
methods
• Demands of
counterfactuals
• Need to translate for policy
and programming uptake
Prevailing view of policy engagement in
the IE community
Get evidenceDisseminate
as reports and papers
Change happens
Assumption: You need the evidence
before you engage and promote uptake
3ie funds high-quality policy
relevant studies
Applications
- Ask policy relevant
question(s)
- Have potential for
policy impact
- Study team has
demonstrated
experience in
policy influence
ASSUMPTIONS
Increased scoring for policy aspects
National researchers on team
Preparation phase
How we
support
3ie requires stakeholder engagement
Researchers
-Committed
- Understand how
policy influence
happens
- Have the tools
and resources to
invest in policy
engagement
- Decision-makers
interested
ASSUMPTIONS
Inception workshop includes implementing agencies
Identify useful evaluable questions
Develop a policy influence plan
Earmark study budget
How we
support
Uptake of study
findings and
improved policies
and practice
Study
-Makes policy-
relevant
recommendations
- Answers what
works and why
- Proposes
feasible solutions
- Has had ongoing
engagement
ASSUMPTIONS Ongoing engagement with 3ie as part of grant monitoring– graduating to a more dynamic model of interaction
Intensive review and feedback of study reports by internal and external reviewers
How we
support
Using impact evaluation evidence for
policy advocacy: 3ie perspective
• Benefits
– Policymakers can understand numbers
– Talking in effect size and costs can be
compelling
– Policymakers often are looking for cost
effectiveness evidence
– Programme managers want the IE and
are invested in the questions and want
to be able to act on findings
– They have credibility in today’s climate