Making a difference:M&E of policy researchJohn Young: ODI, Londonj.young@odi.org.uk

What should you measure?It depends what you’re trying to do….

“If you don't know where you are going, any road will get you there”

Should be:

Specific

Measurable

Achievable

Realistic

Time-bound

(Objective)

Whatever you measure

M&E of policy research1. Strategy and direction: Logframes;

Social Network Analysis; Impact Pathways; Modular Matrices

2. Management: ‘Fit for Purpose’ Reviews; ‘Lighter Touch’ Quality Audits; Horizontal Evaluation; Appreciative Inquiry

3. Outputs: Evaluating academic articles and research reports; Evaluating policy and briefing papers; Evaluating websites; Evaluating networks; After Action Reviews

4. Uptake: Impact Logs; New Areas for Citation Analysis; User Surveys

5. Outcomes and impacts: Outcome Mapping; RAPID Outcome Assessment; Most Significant Change; Innovation Histories; Episode Studies

www.odi.org.uk/RAPID/Publications/RAPID_WP_281.html

Logical frameworksGoal Indicator MOV Assumptions/Risks

Purpose Indicator MOV Assumptions/Risks

Output 1 Indicator MOV Assumptions/Risks

Output 2 Indicator MOV

Output 3 Indicator MOV

Output 4 Indicator MOV

√√

...and many projects fail when the inputs cease...

Change takes a long time

Inputs Activities Outputs Outcomes Impact

Other Actors

Project Effort

Behaviour Change

Focusing on change

www.odi.org.uk/RAPID/Tools/Toolkits/KM/Outcome_mapping.html

Emphasis on “learning”

“…every time we do something again, we should do it better than the last time…”

Goals ResultsActivities

Learnduring

Learnafter

Learnbefore

External networks; Colleagues; Information assets; Own knowledge

www.odi.org.uk/RAPID/Tools/Toolkits/KM/Index.html

Starts with the attitude that someone has probably already done what I am about to do.

I wonder who?”

Learning before: Peer Assist

www.odi.org.uk/RAPID/Tools/Toolkits/KM/Peer_assists.html

• What was the situation?

• What was the challenge?

• What was done?

• What was the result?

• What lessons can be drawn?

Learning During: Stories

www.odi.org.uk/RAPID/Tools/Toolkits/KM/Stories.htmlwww.mande.co.uk/docs/MSCGuide.pdf

Most significant change1.Best stories at each level

2.Synthesis

Stories of change

Horizontal evaluation• Peer review

– Choose the moment– Choose your peers– Limited criteria– e.g. ODI Peer Review

Diagram 8. The Appreciative Inquiry ‘5-D’ model

• Appreciative enquiry– Self-evaluation– CGIAR/CIAT– Workshop

www.odi.org.uk/RAPID/Tools/Toolkits/KM/AAR.html

An after action review asks 4 simple questions:

15 minute team debrief, conducted in a “rank-free” environment.

Learning after: AAR

• What was supposed to happen?

• What actually happened?• Why was there a

difference?• What can we learn from it?

Case & Episode Studies• Classical case studies: how did

evidence shape policy decisions?– e.g. IFPRI & IDRC– Overestimate the role of research

www.idrc.ca/en/ev-26606-201-1-DO_TOPIC.html

www.ifpri.org/impact/impact.htm

www.odi.org.uk/RAPID/Publications/BRP_ITDG.html

www.odi.org.uk/RAPID/Projects/PPA0104/Index.html

www.gdnet.org/middle.php?oid=175

• Episode studies: retrospective tracking back from policy change– e.g. PRSPs, SL, AHC– Underestimate the role of research

Social Network Analysis

www.odi.org.uk/RAPID/Tools/Toolkits/KM/Social_network_analysis.html

RAPID Outcome Mapping

www.odi.org.uk/RAPID/Publications/RAPID_WP_266.html

Impact log• Partner’s research quoted in media or on a

blog: ebpdn research on Japanese Aid on BBC Afrique; Waldo Mendoza of CIES quoted in Géstion.

• ebpdn used to find position: MSc student in US got intern position in Uganda; a member cites active membership of ebpdn in his on-line CVs (on other web sites)

• RAPID framework in academia: Garrett, J. “Improving Results for Nutrition” in the Journal of Nutrition (2008)

• RAPID approaches incorporated into organisations: CIDA research studies to make it more participatory

Stories

RSS Feeds

Webstats

Other approaches: Public• Citations, webstats, media

logs etc• Surveys

– Quantitative– Qualitative

• Distribution lists and attendance records

• Meeting evaluations• Logs:

– The expected– The unexpected– How you have changed

Evaluation: Practical Guidelines, Research Councils UK. 2002

www.rcuk.ac.uk/cmsweb/downloads/rcuk/publications/evaluationguide.pdf