1

A review of Evaluation and Impact Assessment A review of Evaluation and Impact Assessment by DAC Heads of Information in 2006 by DAC Heads of Information in 2006

Ida Mc Donnell

Annual Meeting of DAC HoI

Canberra May 2006

2

3

Structure

Challenges and Next Steps

I

II

III

Objectives of the Review

Survey Results

4

Challenges and Next Steps

I

II

III

Objectives of the Review

Survey Results

5

Objectives Objectives

• Overview of current practice in evaluating communications, awareness raising, development education

• Identify common lessons and challenges

6

Challenges and Next Steps

I

II

III

Objectives of the Review

Survey Results

7

Why evaluate?Why evaluate?

Source: Questionnaire administered to DAC Heads of Information October 2005

Purpose of Evaluations for Aid Agencies and NGOs Funded by Agencies (n=19)

1716

11 1110

1314 14

11

0

02468

1012141618

Lessons forfuture

Monitoring Accountability Impact Other

nu

mb

er o

f re

spo

nse

Aid agency

NGO

8

Common criteria for evaluating an activity Common criteria for evaluating an activity (Govt & NGO) in 11 DAC Members(Govt & NGO) in 11 DAC Members

New/pilot activityCost/size/importance

Depends on activity

Ongoing,L-T activity

Belgium * * *

Canada * *

Denmark * (NGOs only) *

Finland * * (NGOs) *

Germany * (> than euro 7K for NGOs)

*

Luxembourg *

Netherlands * (>euro 20K) *

Norway *

Sweden *

Switzerland * *

UK *

Source: Questionnaire administered to DAC Heads of Information October 2005

9

Tools & IndicatorsTools & Indicators

16

15

13

12

11

11

11

7

3

2

2

0 2 4 6 8 10 12 14 16

Polls

Visits to website

Number participants

Media coverage

Focus group

Interview target group

Newspaper refs

Request information

Other

Invitations to show activity

Questions in Parliament

Figure 2 Tools and Indicators for M & E in DAC Members

no. of response

Source: Questionnaire administered to DAC Heads of Information October 2005

10

Challenges and Next Steps

I

II

III

Objectives of the Review

Survey Results

11

Top challenges Govt Evaluation of CommunicationTop challenges Govt Evaluation of Communication

• How to measure impact• Attribution• Resource issues - cost and time• Indicators, incl. process, short-term, long-term and

comparable between different activities.• Culture of evaluation• Status of communications; not treated seriously/seen as

part of development cooperation so less status…

12

Top challenges Evaluation by NGO of Top challenges Evaluation by NGO of CommunicationCommunication

• Similar to Govt. (cost, time, impact, culture, comparability…)

With a few specificities to NGOs: • lack of interest in evaluating long-term reach, especially

when funded for short-term activities• diversity of NGO activities and targets • difficulty in having standard indicators • no established methodologies• few trainers• no expertise in evaluation• trust to report negative conclusions from evaluation

13

HOI Network can helpHOI Network can help

• Sixteen responses

• 12 think HOI can exchange best/good practice, experiences, and information on evaluation and impact assessment.

• Nine think HOI can identify relevant methodologies, common indicators, and best practice guidelines on evaluating effective communication and development education.

• Other: testing/experimenting with new methods; solving the issue of attribution; organise workshop on the art and theory of evaluation; better coordination; share good external evaluators.

14

Options for Next Steps…Options for Next Steps…

• Organise workshop in early 2007 with expert evaluators, including members of the DAC Evaluation Network

• HOI network work on/agree on common guidelines/best practice on evaluation?

• Investigate the possibility of experimenting with methodologies and create a knowledge-base on which activities are more effective in changing behaviour than others?

• Other?

15

Thank you!Thank you!

www.oecd.org/dev/opinion