Post on 07-May-2015
description
transcript
Measuring policy influence: like measuring thin air?
John Young: j.young@odi.org.uk Arnaldo Pellini: a.pellini.ra@odi.org.uk
RAPID
Power, Politics and evidence use
Evidence production and communication
Knowledge intermediaries and interactions
Policy processes are...
Monitoring and Evalua/on
Agenda Se3ng
Decision Making
Policy Implementa/on
Policy Formula/on
Civil Society
Donors Cabinet
Parliament
Ministries
Private Sector
The Cynefn Framework
4
Policy change
• Discursive: Client- focused services
• Attitudinal: Farmers have good ideas
• Procedural: Participatory approaches to service development
• Content: UU20, UU25. New guidelines
• Behavioural: Approach being applied in practice
5
Focus on behaviour change
Inputs Activities Outputs
Other Actors
Project Team
Outcome Impact Outcomes Impact Outcomes Impact Behaviour Change 6
RAPID Outcome Mapping Approach
Define your policy
objectives
RAPID Outcome Mapping Approach
Academic research communications
Develop a network or partnership
Online communications
Media strategy
More research
Policy advocacy coalition
Why do M&E?
• To learn about what works • To manage better • To account:
– to donors – to recipients
9
Methods
• Classical case studies (IDRC, IFPRI) • Episode studies (ODI/RAPID) • Stories of Change (Denning) • Micro-Narratives (Snowden) • Impact matrices (Davies) • Peer evaluations (CHSRF) • Systematic reviews? • RCTs?
10
Outcome Mapping
OUTCOME MAPPING: Building Learning and Reflection into Development Programs Sarah Earl, Fred Carden, and Terry Smutylo
http://www.idrc.ca/en/ev-9330-201-1-DO_TOPIC.html
11
Social Network Analysis
www.odi.org.uk/RAPID/Tools/Toolkits/KM/Social_network_analysis.html 12
RAPID Outcome Assessment
www.odi.org.uk/RAPID/Publications/RAPID_WP_266.html 13
1. Strategy and direction –are you doing the right thing?
2. Management –are you doing what you planned to do?
3. Outputs – are the outputs appropriate for the audience?
4. Uptake – are people aware of your work?
5. Outcomes and impacts –are you having any impact?
A systematic approach
1. Strategy and direction
2. Management
3. Outputs
4. Uptake
5. Outcomes and impacts
14
Conclusions
Research to influence: • Clear objectives
• Understand the context
• Theory of change
• Iterative / learning approach
Measuring impact • Clear objectives • Theory of change • 5-levels • Multiple methods • Triangulation • Expect the unexpected
15
Recommendations
• Strategy: theory of change, impact pathways, peer review, log frame
• Management: process records, appreciative inquiry, AARs, PRINCE2
• Output: logs, peer-review,
• Uptake: logs, webstats, surveys
• Impact: outcome mapping, stories of change, episode studies, peer review
16
VDR 2010
VDR 2010: assessing policy influence
VDR 2010: Modern Institutions
• Research output of a large governance programme (DfID)
• 12 months of research and consultations
• Presented by WB + 13 donors in 12/2009
• Policy influence assessment in 03/2011
VDR 2010
Confirm the policy change
Attitudinal change
Procedural change
Policy content
Behaviour change
Attitudinal change
Policy influencing approaches Evidence / science
based
Interest / values based
Inside track / cooperation
Outside track / pressure
Advising Advocacy
Lobbying Activism
Start and Hovland (2004)
VDR 2010
Areas of M&E
Strategy and direction Logframes, ToC
Management processes Project Governance, Research Management
Outputs Peer reviews, After Action Reviews
Uptake Citation Analysis, Uptake logs, User Surveys, …
Outcomes and impacts Stories of Change, Episode Studies, …
Hovland 2007
Strategy and direction Logframes, ToC
Management processes Project Governance, Research Management
Outputs Peer reviews, After Action Reviews
Uptake Citation Analysis, Uptake logs, User Surveys, …
Outcomes and impacts Stories of Change, Episode Studies, …
Areas of evaluation
Strategy and direction
Management processes
Outputs
Uptake
Outcomes and impacts
Planned
Data on uptake of VDR 2010
Stories of Change
Areas of M&E
What we did
• Collaborate with a local research/advocacy organization
• Citation analysis Google search
• Interviews with donors representatives
• Interviews with GoV advisers involved in the VDR 2010 process
• Interviews with GoV officials
What we did/learned
• Cultural norms influence the process
• Citation, Google search, and time
• Uptake and www traffic data
• Management of the process emerged as an important theme
• Perception of the quality of the report
• Too early to speak of impact
Areas of evaluation
Strategy and direction
Management processes
Outputs
Uptake
Outcomes and impacts
Planned
Data on uptake of VDR 2010
Story of Change / Episode Study
What we did/learned
Perception about the timing of the VDR 2010
Perception of the process
Perception of the quality of the VDR 2010
Data on uptake of the VDR 2010
What we did/learned
Assessing the policy influence of research: a case study of governance research in Viet Nam ODI Background Notes, May 2012 (http://bit.ly/LNLHE0)
(http://bit.ly/LNOJZ1)
Assessment of the delivery of the Vietnam development report 2010 (Full report)
What we did/learned
• ODI has 12 core research programs
• 145 publications / 74 blogs / 13 issues Development Policy Review and Disasters (2009-2010)
• 20 public events / 4500 participants (2009-2010)
• Advisory work
M&E in ODI
Strategy and direction
ODI Strategy / Program Strategy / Reviews by SMT / ToCs
Management processes
Institute-wide management processes / staff surveys / project completion reports / after
action reviews / internal peer review
Outputs Peer reviews / quality assurance / communication team input / user surveys / end of project report
+ after action reviews
M&E in ODI
Uptake Note by communication team / CommStat / regular updates on
uptake / uptake log
M&E in ODI
CommStat ODI RAPID July – September 2011
M&E in ODI
M&E in ODI
Outcome / Impact
Stories of Change page / Review by TT heads (2008) / Program evaluations
M&E in ODI
http://www.odi.org.uk/work/stories-change/
M&E in ODI
Thank you