Module 6 Module 6 – – Evaluation Evaluation Methods and TechniquesMethods and Techniques

23-04-18 2

Questions and

criteria

Methods and

techniques

Quality

How the evaluation will be done

Overview of the ModuleOverview of the Module

1- Methods, techniques and tools2- Methods for the evaluation of impacts3- Techniques commonly used for evaluations

23-04-18 3

Questions and TechniquesQuestions and Techniques

Descriptive Available data Case studies Statistical survey

Normative Available data Case studies Statistical survey Cost-benefit analysis Cost-effectiveness analysis

Impact-focused Search for causal relations Statistical analysis Forecast analysis

Descriptive Available data Case studies Statistical survey

Normative Available data Case studies Statistical survey Cost-benefit analysis Cost-effectiveness analysis

Impact-focused Search for causal relations Statistical analysis Forecast analysis

Each type of evaluation question is Each type of evaluation question is associated with specific techniquesassociated with specific techniques

23-04-18 4

Each evaluation question implies

a specific approach which will allow the evaluator to gather the elements that he/she needs to build

a line of thought that will allow the evaluator to formulate

a sound (or convincing) judgement

What Does an Evaluation What Does an Evaluation Question Include?Question Include?

23-04-18 5

Did the implementation of this program unfold as planned?

1) I will identify what was planned in the original documents

Document ReviewDocument Review

Interviews with various people in charge to ensure that I have a good understanding of the

situation

Interviews with various people in charge to ensure that I have a good understanding of the

situation

How can you give a valid answer to this question?

Document review, review of archive filesDocument review, review of archive files

Interviews with people involved in the different modules and the different phases, and with

different sensitivities

Interviews with people involved in the different modules and the different phases, and with

different sensitivities

Examples of QuestionsExamples of Questions

2) I will reconstruct what happened in reality

23-04-18 6

Did the implementation of this program unfold as planned?

3) I will compare the plan with what happened, identify the gaps, and check that the opinions collected on this issue lead to the same conclusions

Preparation of a retrospective time

chart

Preparation of a retrospective time

chart

Interviews with various stakeholders to check

the validity of the conclusions reached

Interviews with various stakeholders to check

the validity of the conclusions reached

Organization of the arguments

Organization of the arguments

This is how I propose to proceed to answer this question

4) I will decide what constitutes a minor change and a notable change. I will only flag the changes that have had visible consequences on costs and delivery time

Examples of Questions Examples of Questions (cont’d)(cont’d)

23-04-18 7

Did the implementation of this program unfold as planned?

Various techniques must be used in this approach. These techniques call for some specific tools

Examples of Questions Examples of Questions (cont’d)(cont’d)

TechniquesTechniques• Document review• File review• One-on-one interviews• Discussion groups

TechniquesTechniques• Document review• File review• One-on-one interviews• Discussion groups

ToolsTools• Interview guides• Time charts

ToolsTools• Interview guides• Time charts

23-04-18 8

Second question, to be selected with the group, preferably on efficiency

Second ExampleSecond Example

Follow the reasoning together:• How do you answer the question (consider

going all the way back to how the question is worded)?

• What process do you follow?• What techniques and tools do you use?• Compare with the previous approach.

23-04-18 9

The term ‘method’ is generally used to designate the process for evaluating

impacts

The whole evaluation process consists in tools, techniques and a

method

23-04-18 10

How do you measure the impact (effects) of a

program? This is one of the key methodological issues for evaluations

At the Heart of Evaluation: The At the Heart of Evaluation: The Measurement of the EffectsMeasurement of the Effects

The rationale of a program is to produce an effect or an impact

Lon

g-

an

d s

hort

-term

im

pacts

Impact

Effects

Directresults

Activities

Means

Purpose

Objective

23-04-18 11

Impact Assessment MethodsImpact Assessment Methods

1. To what extent is it possible to identify the effect (revenues increase, the prevalence of a disease goes down, etc.)

2. To what extent can this effect be

attributed to the program (and not to

some other cause)

Solutions must be found for two problems:

To find the best answers possible to these two questions, methods that are specific to evaluation are used

23-04-18 12

What would have happened with the beneficiaries if the program

had not existed?

What would have happened with the beneficiaries if the program

had not existed?

Impact Assessment MethodsImpact Assessment Methods

The evaluator’s key question:

How do you rewrite history? How do you get baseline data?

23-04-18 13

The ideal experimentation

Possible causes

Studied groups

Results of the observation

(effects)

All other incluences

The project or the policy

Control group

Experimental group

Results for the control

group Results for

the experimental

group

Net results (attributable to the project or

the policy)

The ideal experimentation

Possible causes

Studied groups

Results of the observation

(effects)

All other incluences

The project or the policy

Control group

Experimental group

Results for the control

group Results for

the experimental

group

Net results (attributable to the project or

the policy)

The Only Solution Beyond The Only Solution Beyond DoubtDoubt: : The Ideal The Ideal ExperimentationExperimentation

23-04-18 14

In theory, a single observation is not enough

Effect or impact = 13

With equivalent control group.

Extreme care must be taken when selecting the control group to ensure comparability.

Incom

e level

TimeProgram

30

10

17

The Ideal ExperimentationThe Ideal Experimentation

BeneficiariesEquivalent control group

23-04-18 15

With equivalent control group

The Ideal ExperimentationThe Ideal Experimentation

• Ethical problem (to condemn a group to not being beneficiaries)

• Difficulties associated with finding an equivalent group outside the project

• Costs

In practice, it is extremely difficult to assemble an exactly comparable control group:

Therefore, this solution is hardly ever used

23-04-18 16

Effect or impact = 13

Establishment of a baseline study

Time

Program

30

10

17

Incom

e level

14

21

Comparison with a Comparison with a Non-Equivalent Control GroupNon-Equivalent Control Group

BeneficiariesEquivalent control group

Data on before and after

situations is needed.

23-04-18 17

Findings on the impact lack

precision and soundness.

The time series make it possible to

reach better conclusions.

Effect or impact?

Incom

e level

Time

Program

30

10

1714

21Base Line

Study

Time series

?

Broad descriptive survey

Beneficiaries

Evaluation without a Evaluation without a Comparison Group, Using a Comparison Group, Using a Before/After ComparisonBefore/After Comparison

23-04-18 18

It is impossible to reach a conclusion

regarding the impact; While it is possible to

say whether or not the objective has been reached (the effect was achieved), the

result cannot be attributed to the

program.

Effect or impact ?

Time

Program

30

Reven

ue level

Evaluation Using a Simple Evaluation Using a Simple Post-Implementation Post-Implementation ObservationObservation

Beneficiaries

23-04-18 19

-Quality

+

++

+++

Four Four Broad Categories of Broad Categories of Evaluation MethodsEvaluation Methods

Equivalent control group (true experimentation)

Non-equivalent control group

Evaluation without a control group

Evaluation by comparison with a control group

Post-implementation observation

Observation before and after implementation

23-04-18 20

1- Evaluation Using a Simple1- Evaluation Using a Simple Post-Implementation Post-Implementation Observation ObservationPossibilitiesVery simple to do. Suitable for the evaluation of means and implementation. Allows evaluators to: (i) ascertain how a policy was implemented; (ii) measure its immediate results (outputs); (iii) gain a better understanding of the behaviours of the groups involved and of the tools or mechanisms.

LimitationsDifficult to isolate the effects of the policy from the other evolution factors. Huge risk of being subjective. Can hardly be used to evaluate the impact or identify a pattern explaining the phenomena that were observed. Causal relations can be suggested but the findings regarding these links are very fragile and cannot easily be expanded to other situations.

Techniques usedFile review, direct observation, expert opinions, case study, statistical surveys, data analyses, calculation of ratios, comparisons with standards, etc.

-

23-04-18 21

PossibilitiesVery common. Corresponds to the natural evaluation process: trying to check that the period over which the policy was implemented coincides with a change in some of the indicators. Same possibilities as `previous method. In addition, makes it possible to have a more refined and quantified description of the effects.

LimitationsSame as for the previous method. But allows evaluators to use more precise indicators and to frequently cross-check results to test their soundness

Techniques usedRequires the use of a good description of the baseline situation for all the project results indicators. File review, direct observation, expert opinions, case study, statistical surveys, data analyses, time series analysis, calculation of ratios, comparison with standards, etc.

+2- Evaluation Using a2- Evaluation Using a Before/After Comparison Before/After Comparison

23-04-18 22

Possibilities Comparison of the policy’s target group with a control group with slightly different characteristics. Allows evaluators to (i) better define the impact or the external results of the policy (without affirming there is a causal relation) and (ii) reveal the mechanisms and behaviours that exist** regarding incentive policies.

Limitations Limited relevance for identifying causal relations without any ambiguity. By increasing the number of control groups, it is possible to strengthen the findings. When a group is specifically consulted for the evaluation without being ‘equivalent’,complex statistical techniques sometimes make it possible to isolate the biases associated with the non-equivalence (‘quasi experimentation’) .

Techniques usedCase study, statistical survey, data analysis, time series analysis, multivariate analysis, modeling.

++

3- Evaluation Using a 3- Evaluation Using a ComparisonComparison with a Non-Equivalent with a Non-Equivalent ControlControl Group Group

23-04-18 23

Possibilities The only totally rigorous evaluation procedure, used among other fields, for therapeutic issues (evaluation of the effects of a medical treatment). It makes it possible to identify without any doubt causal relations and, as a result, the specific effects of a policy or a project.

LimitationsNumerous feasibility problems are encountered in the field of socio-economical policies when trying to assemble an equivalent control group. This group must be constituted before the project or program is launched following extremely strict rules. Can be unethical.

Techniques usedStatistical survey, data analysis, time series analysis, multivariate analysis, modeling.

+++

4- Evaluation Using a 4- Evaluation Using a ComparisonComparison with an Equivalent Control with an Equivalent Control Group (true Group (true experimentation)experimentation)

23-04-18 24

Practically all the techniques used in economics and political sciences, especially in statistics, can be used for evaluation.

Techniques, Tools, Techniques, Tools, Instruments...Instruments...

• Interview• Discussion group• Literature search• Archive file review• Questionnaire survey• Case study• Aptitude or knowledge test• Opinion poll• Content analysis• …

23-04-18 25

One of the necessary qualities for an evaluation tool is that it must be able to identify/imagine the technical processes that will allow it to reach convincing evaluation conclusions for each one of the questions.

Techniques, Tools, Techniques, Tools, Instruments ...Instruments ...

23-04-18 26

Techniques, Tools, Techniques, Tools, Instruments ...Instruments ...

It is not possible, in the time available for this workshop, to discuss in detail the techniques, tools and instruments that can be used. It should be noted that evaluators do not necessarily master all these techniques and they often have to call upon specialists.

Some of the most commonly used techniques will be presented as part of the exercises.

23-04-18 27

For those of you who want to know more on the techniques, tools and instruments available, there are numerous manuals, guides, etc. on the following topics:

Techniques, Tools, Techniques, Tools, Instruments …Instruments …

• Data collection• Interviews, discussion groups• Direct observations, case studies• Data analysis• ...

23-04-18 28

Summary of MethodologiesSummary of Methodologies

TYPE STRENGTH WEAKNESS Questionnaire Efficient

Large # respondents

Extensive planning Low response

Interview In-depth info High response

Extensive planning Time consuming Analysis difficult

Focus Group Group synergy Diverse perspectives

Extensive planning Analysis difficult Logistics

TYPE STRENGTH WEAKNESS Questionnaire Efficient

Large # respondents

Extensive planning Low response

Interview In-depth info High response

Extensive planning Time consuming Analysis difficult

Focus Group Group synergy Diverse perspectives

Extensive planning Analysis difficult Logistics