Strengthening Impact

Assessment in the CGIAR

ISPC 11, CIFOR, Bogor

31 March 2015

CGIAR Standing Panel on Impact Assessment

Doug Gollin

Chair

ISPC 11, CIFOR

http://impact.cgiar.org

Karen Macours

Activity leader

JV Meenakshi

Member

Bob Herdt

Member

John Ilukor

Research Associate

Frederic Kosmowski

Research Associate

James, Lakshmi,

Ira, Tim

SPIA Secretariat

Erwin Bulte

Activity leader

Expansion of the available set of impact studies, providing useful

and credible information to underpin and guide future

investments in the CGIAR

CRPs and Centers have institutionalized impact assessment such

that ex post impact assessment is regarded as an essential part

of prudent research management for accountability purposes

and as an input to ex ante strategic planning

SIAC: Vision of success

In the long run, judged by two key outcomes:

ISPC 11, CIFOR

http://impact.cgiar.org

Strengthening Impact Assessment in the

CGIAR (SIAC)

Timeline: 2013 – 2016 (phase 1)

Mid-term review : 20 participants, Rome, Feb 2015 – document available

Objective 1:

Experiment with new methods for estimating adoption

Objective 2:

Institutionalize the collection of adoption data

Objective 3:

Impact assessment of under-evaluated areas of CGIAR research

Objective 4:

Build community of practice on impact assessment

ISPC 11, CIFOR

http://impact.cgiar.org

Major developments since ISPC 10

ISPC 11, CIFOR

http://impact.cgiar.org

Review of irrigation and water management impact assessment Doug Merrey

Call on long-term large-scale impact assessments: 5 short-listed

Call on experimental approaches: 3 funded + launch workshop at MIT, Feb

Swarna-Sub1, Bangladesh - UC Berkeley, Tufts, IRRI

Effects on yields, welfare, water markets and environmental spillovers

Maize-vegetable inter-cropping, Nepal - Yale, ICIMOD

Effects on yields, welfare (nutrition). Study heterogeneity of impacts of

different farmers, as reached by different dissemination mechanisms

Personalized input recommendations and information on conservation

agriculture, Mexico - ITAM, J-PAL, UC Berkeley

Testing the returns to customization to specific soil conditions

The “learning” agenda in impact

assessment

A selection from across the SIAC portfolio

1 - On-farm trials

Technology: Integrated Soil Fertility Management, Kenya

Method: RCT (Paris School of Economics + IITA)

Questions:

• Do randomly selected farmers have different results from farmers chosen

by IITA?

• Do participating farmers adopt and what are the longer-term impacts?

Normal IITA practice: 5 HHs selected for on-farm trials by community

gatherings (after IITA had explained research)

Comparison: 5 HHs selected from list of all HHs in the village

ISPC 11, CIFOR

http://impact.cgiar.org

Indicators Random farmer Selected farmer Difference

years of edu of hh head 5.908 7.102 1.194***

literacy main farmer 0.710 0.828 0.118***

years of edu of main

farmer

5.413 6.717 1.304***

main farmer is female 0.627 0.508 -0.119***

no cattle ownership 0.420 0.287 -0.133***

no of cattle owned 2.342 3.222 0.880***

log (land owned) 0.255 0.2921 0.0371

number of plots 2.793 2.977 0.184***

hh uses manure 0.628 0.7137 0.0857***

hh uses some fertilizer 0.709 0.7911 0.0821***

uses intercropping 0.882 0.88727 0.00527

uses crop rotation 0.733 0.7703 0.0373

uses hybrid seed 0.351 0.48 0.129***

uses fallow 0.414 0.4437 0.0297

uses soil conservation

practices

0.255 0.3152 0.0602**

2 - Understanding “hidden” welfare gains

Technology: NERICA rice

Method: RCT (Innovations for Poverty Action, MIT, Sierra Leone

Agricultural Research Institute)

Design:

Randomly allocate (subsidized at: 0%; 50%, 100%) NERICA seed to farmers,

with and without training on use

Questions:

• What are the impacts of adoption on meals skipped, weight for height,

aggregate HH-level consumption...?

• How important is training in determining effective use of the technology?

ISPC 11, CIFOR

http://impact.cgiar.org

ISPC 11, CIFOR

http://impact.cgiar.org

But… NERICA shortens the hungry season

In remote rural Sierra Leone, there is still a pronounced hungry season .

NERICA is mature and ready for harvest 20 – 50 days earlier than local

upland varieties

Preliminary results from the RCT suggest substantial improvements in short-

term nutritional outcomes for children in the treatment group.

Earlier harvest will not translate into nutritional gains if:

• Farmers take advantage of high prices during September to sell rice

and purchase non-food items

• Rice production is simply shifted earlier, changing the timing of the

hungry season

• NERICA growing farmers substitute out of other crops with high

nutritional value.

ISPC 11, CIFOR

http://impact.cgiar.org

3 - Identifying CGIAR technologies in field

surveys

Two questions:

• How reliable are our tools for identifying specific technologies?

• Once we have reliable tools, how can we integrate them into large-scale

high-quality surveys?

ISPC 11, CIFOR

http://impact.cgiar.org

DNA fingerprinting and varietal

identification

ISPC 11, CIFOR

http://impact.cgiar.org

Estimate of IVs Correctly ID as improved variety

Farmer self-report name of variety 1.9 0.8

Farmer self-report – improved vs local 6.4 2.2

Showing farmers photos 0.2 0.2

Trained enumerators visit fields 5.4 1.8

Photos taken back to experts later 18 3.5

DNA fingerprinting 30 30

Cassava in Ghana – MSU / IITA

DNA fingerprinting and varietal

identification

ISPC 11, CIFOR

http://impact.cgiar.org

•Similar work underway on maize in Uganda

•Large-scale representative sample

•Aim is to understand when/whether/how we can use other (less

expensive) methods to measure adoption.

•What are the best methods for different contexts?

Partnership with LSMS-ISA

ISPC 11 - CIFOR

http://impact.cgiar.org

• World Bank Living Standards Measurement Study

– Integrated Surveys of Agriculture (LSMS-ISA)

• 8 countries in SSA – all important to CGIAR

• Average of 5,000 HHs / country, nationally

representative

• Panel – visited every 2 years

SPIA role:

• Surveys lack modules / questions on agricultural

technologies (varieties, NRM practices)

• SPIA’s comparative advantage to work to

improve this for benefit of CGIAR as a whole

Partnership with LSMS-ISA

ISPC 11 - CIFOR

http://impact.cgiar.org

Frederic Kosmowski

ILRI, Addis Ababa

• Scoping Ethiopian priorities

across CGIAR centers

• Later, Niger / Burkina / Mali

surveys

John Ilukor

IITA, Malawi

• Cassava DNA fingerprinting

in Malawi

• Maize DNA fingerprinting in

Uganda

Two SPIA Research Associates hosted by CGIAR and working with LSMS-ISA

Randomization design

• Scientists worked with all 960 farmers & in T1 and T2 during the season to

implement an experimental trial on their land

• Study compared random control plot to 5 randomly selected treatment plots

• 10 farmers in each T village

– 3 soya trials

– 3 soya-maize intercrop trials

– 4 maize trials

2014 long rains 2014 short rains 2014 long rains

Control 46 villages 23 villages 23 villages

T1 23 villages 23 villages 23 villages

T2 23 villages 23 villages