Photo: S. Kilungu (CCAFS)

SIAC Program Report 22nd July 2016

Page 1 of 41

SIAC Program Report: Summary of Progress To-Date

Prepared by SPIA for the SIAC External Evaluation (DRAFT)

SIAC, Strengthening Impact Assessment in the CGIAR, is a four-year program of work led by SPIA that aims to broaden the coverage and widen the range of impact measures of CGIAR research. The program started in 2013 and was motivated by a strong and growing demand across the development community for high-quality ex post impact assessment to support evidence base decision making in the CGIAR. Expanding the evidence base, including data collection across the full range of CGIAR research types, was deemed essential in underpinning and sustaining investments in the CGIAR in the post-reform period. Expanding the inventory of impact assessment of CGIAR research is one component of that effort; the other relates to enhancing the credibility and rigor of these studies through greater independence, transparency and external review. Ex post impact assessment is important in the reformed CGIAR in terms of assessing the strength of the linkages between agricultural research and the System-Level Objectives (SLOs) of poverty reduction, food security, improvements in nutrition and health, and sustainable natural resource management. SIAC activities aimed at developing new methodological tools and collecting the needed data for impact assessment will allow the CGIAR to extend its work across a broader range of research activities and impact pathways.

The key objectives of the SIAC program are as follows:

Objective 1 (Methods): Develop, pilot and verify innovate methods for collection and assembly of diffusion data;

Objective 2 (Outcomes): Institutionalize the collection of diffusion data needed to conduct critical CGIAR impact evaluations;

Objective 3 (Impacts): Assess the full range of impacts from CGIAR research;

Objective 4 (Building a community of practice): Support the development of communities of practice for ex post impact assessment within the CGIAR and between the CGIAR and the development community more broadly.

A major feature of the work being carried out in this program, and most particularly for the activities under Objective 3, is the sponsorship through a competitive grant-making process of studies of economic, social, and environmental impacts. All else being equal, employing external and independent researchers ensures a higher level of objectivity and credibility, compared with relying on CGIAR centers to carry out these studies. Nevertheless, CGIAR centers and researchers often have key knowledge and, in the majority of cases, are involved to a varying extent (depending on capacity) in many of the studies.

The program is now in its fourth year but is likely to extend another six months to allow time to bring to completion a number of studies (see Annex 1 for list of funded studies and timetable). The four-year budget for this program of work is approximately US$ 12 million1. This report summarizes the progress to-date and is organized around the Objectives spelled out in the SIAC program of work.

1 Funding for SIAC for the period 2013 – 2016 comes from three donors: - USD 5,238,799 from the Bill and Melinda Gates Foundation (BMGF); - USD 4,453,057 from CGIAR Window 1 and - USD 1,923,568 from the CGIAR ISPC.

Page 2 of 41

Underpinning this objective is the development of a robust set of methods for routinely tracking adoption of CGIAR-related technologies in a cost-effective manner. Such information is a prerequisite for achieving the highest quality assessment of outcomes and impacts. A set of activities are designed to test innovative ways of assessing the adoption of improved varieties of crops, livestock and fish technologies, and agronomic and natural resource management interventions, with the goal of eventually embedding protocols derived on these tests into large-scale surveys carried out by other institutions outside the CGIAR, such as the World Bank’s Living Standards Measurement Survey – Integrated Surveys of Agriculture (LSMS-ISA). There are four main areas of activity here which are managed by Michigan State University.

Activity 1.1. Advance methodologies for tracking the uptake and adoption of improved varieties

The objective of this Activity is to pilot test and validate alternate approaches to collect variety-specific adoption data against the gold standard benchmark (DNA fingerprinting) to determine which method/approach is the most cost-effective (i.e., which method provides a given level of accuracy at the least cost). The idea is to come up with ‘lessons learned’ and recommendations on methods / approaches that can be used in scaling up the collection and assembly of diffusion data on improved varieties. The following crop-by-country combinations were targeted:

1. Cassava in Ghana; 2. Maize in Uganda 3. Beans in Zambia

Two further crop-by-country combinations were added to the SIAC portfolio by SPIA at the start of 2015, as part of the collaboration with LSMS-ISA:

4. Cassava in Malawi 5. Sweet potato in Ethiopia

1. Cassava in Ghana:

This study tests the effectiveness of the following four household-based methods of tracking varietal adoption for cassava against the benchmark of DNA analysis of cassava leaf samples.

A. Elicitation from farmers by asking him/her the: 1) names of varieties planted and some basic questions for each variety planted; and 2) type of variety (improved vs. local)

B. Farmer elicitation on varietal characteristics by showing a series of photographs (or actual plants). This information will be later used by the analyst to identify varieties based on morphological characteristic data.

C. A trained enumerator recording observations on varietal characteristics by visiting the field and sharing their opinion on what the variety is by: 1) name and 2) by type (based on observations). The information collected will be also used by the analyst to identify varieties based on morphological characteristic data.

D. Enumerator taking photos of the plant in the field for latter identification by experts (i.e., breeders)

The field work for this study is jointly supported by SIAC and the RTB CRP and conducted in partnership with IITA, Crops Research Institute (CRI)-Ghana, and Agriculture Innovation Consulting (AIC) Ghana. Field work was completed in late fall 2013. All the samples collected from the farmers’ fields and the 40 genotypes included in the reference library were sent to IITA by the Ghanaian partners in January 2014. DNA extraction work for almost 1000 samples was completed by IITA and all the samples were shipped to Cornell for Genotyping by Sequencing (GBS). Data from the GBS analysis were submitted by Cornell to IITA in July 2014. But due to some issues on limited library materials to classify all the farmer samples, more analysis was required that included cassava genotypes from IITA’s collection. The data from this updated analysis were made available to IITA in December 2014.

A presentation summarizing the main results of this case study along with the results of the bean study (case

OBJECTIVE 1: Develop, pilot and verify innovative methods for collection and assembly of diffusion data (METHODS)

Page 3 of 41

study 3, below) were presented at ICRISAT in June 2015 (upon their invitation during a visit by M. Maredia), at the AAEA meetings in July 2015, and at the International Conference of Agricultural Economists (ICAE) in August 2015. A research paper summarizing the DNA fingerprinting methodology and results of this study was published in BMC Genetics (v. 16:115 DOI 10.1186/s12863-015-0273-1).

The proportion of farmer collected samples classified as released/improved varieties using different methods ranged from 1% for method A1, to 15% for method D. The implication of this is that there could be a large variation in the estimates of adoption of improved varieties based on which method is used to derive the estimate. Additionally, several interesting findings about the materials included in the reference library have emerged from this case study. The most interesting results were that: 1) some released varieties included in the reference library were genetically identical (e.g., IFAD and UCC); 2) many released varieties were hybrids or admixtures; and 3) Library accessions representing both ‘released varieties’ and ‘landraces’ fall under the same varietal cluster groups. This last finding especially posed a challenge for varietal identification. The problem it created is: How to classify farmer samples that fall in these three variety cluster groups? Should they be classified as ‘improved/released’ varieties or local/landrace varieties?

To address this dilemma, the analysis of effectiveness of methods (A to D) against the benchmark of DNA fingerprinting is done under two scenarios / assumptions: Liberal scenario: which assumes that all the farmer samples that fall in a variety cluster in which there is at least one released variety are essentially improved varieties. Under this assumption, 31% of farmer samples are classified as improved materials; and Conservative scenario which assumes the opposite (i.e., farmers samples that match the DNA results of a variety group in which there are both released varieties and landraces, the variety group is considered not-improved). Under this scenario only 4% of farmer samples are classified as improved materials.

In both the liberal and conservative scenario, when the results of methods A - D are compared with DNA analyses, the results indicate that: 1) a large number of farmers are mistakenly identifying varieties as improved varieties when it was not or identifying a variety as traditional when it was in fact improved; and b) the methods of varietal identification that relied on ‘experts’ were better than the farmers’ elicitation, but still way off from the truth established by the DNA fingerprinting method. At least at the variety level, the results of this study has clearly demonstrated the unreliability of both farmer and expert elicitation based methods of varietal identification.

2. Maize in Uganda:

As part of the planned DTMA (Drought Tolerant Maize in Africa) adoption survey by CIMMYT in three districts in Eastern Province of Uganda, MSU had designed and implemented modules and protocols to test the effectiveness of the household-based methods of tracking varietal adoption for maize similar to those used for the cassava in Ghana experiment. Field data were collected in June 2014 and leaf tissues from 416 maize fields across 34 villages were collected for DNA analysis. The National Crops Research and Resource Institute (NaCRRI) of NARO served as the ‘technical’ partner for DNA analysis through their ongoing project with the University of Ghana. Due to delays in transferring the leaf tissues from the field to the lab and to the large amount of compacted leaf material in the tubes, virtually all the samples were lost due to mold development. Due to the delays and difficulties experienced during this project, LGC Genomics offered to repeat the work for this project for free of charge before June 2015 (for 34 sample plates x 146 assays). An alternative was found with SPIA to piggy back on a planned LSMS experiment on maize in Uganda in 2015, and management of the study was transferred over from MSU to SPIA in March 2015 (see below). Regarding the survey data received from CIMMYT, the survey results indicate that, in general, farmers do not have a clear or accurate idea about what types of maize varieties they are growing and very few (6%) were able to show the bags in which the seed planted was obtained (most share or purchase seed with neighboring farmers).

Since March 2015, under SPIA, the context for the study has now shifted to a large methods experiment run by the World Bank LSMS-ISA team and UBOS on estimating maize productivity – the Methodological Experiment on Measuring Maize Productivity, Varieties, and Soil Fertility (MAPS). The following three methods for varietal identification were embedded in the design of the experiment:

Page 4 of 41

A. Asking the farmer to identify the variety. B. Asking the farmer to answer questions related to 15 phenotypic characteristics (using a visual aid),

checked against sets of reference responses for each variety using alternative decision rules. C. Focus group meeting with a number of experts.

These will be benchmarked against two DNA genotyping methods:

D. DNA fingerprinting using SNP markers on samples from maize leaf tissue (using the credit with LGC Genomics under their contract with MSU/NaCCRI)

E. DNA fingerprinting using DArT method of genotypingon samples from maize grain.

Expert opinion elicitation for these two districts (Iganga and Mayuge) was also carried out prior to any field work taking place.

Field work for the whole survey took place over three visits to a sample of 900 households (post-planting; crop-cutting; post-harvest) over the period April 2015 – August 2015 in 5 districts in Uganda. For budget reasons, DNA fingerprinting was possible only on a subset of 550 farms in two districts – Iganga and Mayuge. Enumerators from UBOS were recruited and trained intensively for one month, and survey data collection was facilitated by the use of networked tablets for real-time data management and processing. Leaf samples were collected at the post-planting visit in April and May 2015, from within the quadrant laid down by enumerators for subsequent crop-cutting, using leaf collection kits from LGC Genomics. Grain samples were subsequently collected from these quadrants in the follow-up crop-cut visit in June and July 2015. SNP-based genotyping data was received from LGC in September 2015 following analysis of the leaf samples. Grain samples were processed (dried, ground to flour, labelled) by NACCRI in August and September 2015, and shipped to Diversity Arrays in Australia at the end of October 2015.

Both genotyping methods (D and E) have been successfully applied and the results are currently being compared by colleagues at Diversity Arrays and NaCCRI. Early results show that the SNP-based genotyping used on the leaf samples was insufficiently discriminating among improved varieties – some of the varieties in the reference library appear genetically identical when screen against a low number of SNPs (approx. 140), whereas when a highly quantitative assay used in DArt is applied (examining more than 10,000 alleles), then genetic distance is observed.

Before any consistency with the correct genotype (correct identification) can be established, we have to consider whether unique identification is possible for a given method. This is possible only for 47% of farmer responses, and 13% of responses to questions in the morphological protocol. Farmers submit “don’t know” responses to the open question of what the variety is that they are growing in 53% of cases. In theory, a set of 11 morphological questions is sufficient to uniquely discriminate among all varieties in the reference library. However, in practice, farmers are clearly unable to respond to the morphological questions with sufficient accuracy to allow for unique identification. Only 60 (13%) of the response sets from farmers interviewed correspond completely to a valid set of responses (as determined by characterization of the reference library). How many of these 60 response sets (from method B) are consistent with the genotyped identification (as opposed to occurring by chance from the set of multiple choice questions) is currently being investigated, as is the consistency with the expert opinion elicitation process.

Only 2% of the sample of 477 farmers are able to correctly identify the variety when checked against the genotype (method E). Method E results show that every farmer sampled (all 477) is growing an improved variety of maize – an astonishing finding. Furthermore, there is deep penetration of the commercial varieties from Western Seed company (WE varieties) and YARA seeds, whereas farmer’s expectations are largely confined to the LONGE series released by the government NARO.

While this suggests that improved materials are reaching farmers’ fields some way or another, there is also a deeper, less clear picture that emerges when one considers the heterogeneity of the samples. There is considerable genetic heterogeneity within the samples used for the reference library. A good cut-off for acceptable heterogeneity in the reference material is 15%, whereas our mean reference library heterogeneity level is 33%, suggesting that genetic lines have not been well separated in the breeding process, or the seed sampled for the reference library was not pure breeders’ seed. Second, there is a very low average purity level of the field samples. Only a minority of field samples are at 80% purity or above, dropping sharply and bringing

Page 5 of 41

the average purity to 63%. This means that for an average plot, 37% of the genetic material sown is not from the primary improved variety planted in the plot, but has been introduced to the plot either deliberately (through counterfeiting, or from a farmer choosing to combine varieties together in the same plot) or through handling / labelling errors in the seed supply chain.

The MAPS experiment also has first-rate data on agricultural productivity, soil quality, varietal identification, and household characteristics. We can estimate some simple models for the determinants of productivity: once using the data that are typically collected in surveys, based on farmer testimony; and once using objective methods for varietal identification (DNA fingerprinting), yields (crop-cuts), soil quality (soil samples taken and analysed in laboratory). These results will help us understand more about the importance of data quality in context – does improved data quality substantially impact on our understanding of some fundamental issues in impact assessment? A second round of MAPS is currently in the field between June and September 2016. SPIA is considering have the second round of crop cuts genotyped, as a follow-on from the first round.

Four academic articles are in various stages of preparation with SPIA involvement (John Ilukor and/or James Stevenson), drawing on various elements of the MAPS experiments.

3. Beans in Zambia:

This study tests the effectiveness of the following four household-based methods of tracking varietal adoption for common beans.

A. Elicitation from farmers by asking him/her some basic questions for each variety planted. B. Showing the farmer seed samples representing different varieties and asking him/her to identify the

sample that matches each of the variety grown on their farms. C. Collecting seed samples representing each variety planted by farmers for latter identification by

experts (i.e., breeders). D. Enumerator taking photos of the seeds during the survey for latter identification by experts (i.e.,

breeders).

The accuracy of adoption estimates derived from the above four methods was evaluated against the varietal identification established through DNA fingerprinting of seed samples collected from the farmers.

The context for this study was a bean adoption study conducted by PABRA (Pan-African Bean Research Alliance), in collaboration with CIAT and the Zambian Agricultural Research Institute (ZARI). Seed samples and data corresponding to four methods, similar but not identical to the cassava in Ghana study, have been collected from 402 households that were surveyed under a PABRA study, thus allowing to leverage survey costs.

During a visit to Zambia, the MSU team visited the local market (in Kasama) and collected some bean seed samples from local vendors. These seeds were added to the pool of seeds for DNA analysis to check if the seeds of varieties sold in the market as named by the ‘vendors’ match the actual variety as named and identified by farmers. The total seed sample (both collected from the farmers and from the vendors) was about 900.

As a next step the seeds collected from the farmers’ fields and from the market were germinated by the ZARI breeder in June 2014. A technician from CIAT-Uganda traveled in July to Zambia to help with the DNA extraction and samples were shipped to LGC Genomics. To establish the library, ZARI included all the released varieties plus 15 other local materials in the samples shipped to LGC.

Overall, the adoption (or more appropriately, the frequency of use by farmers) of improved variety as a group based on farmer elicitation methods varied substantially from 15% in method A (asking farmers what type of variety they planted—local or improved) and 72% in method B (showing farmers seed samples). The estimate of overall adoption of improved varieties based on the two expert elicitation methods--method C (showing breeders seed samples) and method D (showing breeders’ seed photos) were similar. The estimated adoption (or frequency of use) was 36% based on method D and 37% based on method C.

Comparing the two methods based on farmer elicitation, results indicate that there was only 25% agreement on the name of the variety planted between methods A and B. In the case of the two methods based on experts’ opinion (i.e., methods C and D), there was close to 80% agreement on identifying the varieties either

Page 6 of 41

by name or by type. The raw data from the SNP analysis were received from LGC Genomics Lab in May 2015. These data were shared with CIAT researchers for interpretation and a report of the data analysis was submitted by CIAT soon after. Based on these data and report from CIAT, MSU has completed the data analysis to test the effectiveness of different methods of varietal identification.

Preliminary results show DNA fingerprinting benchmark of 16% adoption, with farmer elicitation method A returning estimates of 4% adoption of improved varieties when asked for by name, and 13% when asked for as the aggregate class (local vs improved). Method B of showing farmer seed samples resulted in an estimate of 71% improved variety adoption, whereas showing seeds or photos to breeders resulted in adoption estimates rather close in aggregate (though with many possible mis-matches at level of individual varieties, at 18% for showing photos and 15% for showing seeds.

A report summarizing the results of the two pilot studies in Ghana (with cassava) and Zambia (with beans) is under preparation and will be soon published as a MSU Staff Paper. In addition, a manuscript targeted to a peer reviewed journal is being developed and will be ready for submission by the end of August.

4. Cassava in Malawi

The context for this study is a methods experiment led by Talip Kilic of the World Bank LSMS-ISA based around alternative approaches to estimating cassava production from households using diaries and different lengths of recall data. SPIA, through Research Associate John Ilukor, have embedded the following varietal identification approaches into the design of the experiment:

A. Asking the farmer to identify the variety. B. Asking the farmer to answer questions related to phenotypic characteristics (using a visual aid),

checked against sets of reference responses for each variety using alternative decision rules. C. Focus group meeting with a number of experts.

These are benchmarked against:

D. DNA fingerprinting using GBS on samples from cassava leaf.

Cassava leaf collection was integrated in the survey, along with a phenotypic protocol of traits, which was implemented starting June 2015 and continued through to end of June 2016. Leaf collection for fingerprinting, and phenotypic data for subjective identification, are complete. The reference library of varieties, and their corresponding phenotypic attributes, was compiled by the Malawian NARS. DNA was extracted in-country by the Chitedze laboratory for samples from 1002 farms and shipped to Diversity Arrays in Australia for sequencing.

In terms of unique identification, virtually all (1001 or 99.9%) of the farmers submitted a response to the question as to which variety was being grown (method A), whereas unique identification was only possible for 23% of the sample using method B, and indeed only 73% of the sample using genotyping (method D). The low level of unique and valid responses for method B is likely a combination of error on the part of farmers but also the fact that the number of local varieties is very long and as many as 130 local varieties were not included in the reference library. This large number of distinct local varieties without samples held by the NARS for inclusion in the reference library also explains the relatively low (73%) rate at which the genotyping exercise was able to find a match between samples and a valid reference.

In terms of correct identification, farmer elicitation (method A) gives an adoption rate for improved varieties of 19%, method B suggests a 70% adoption rate, whereas the genotyping (method D) actually shows only 10% adoption of improved varieties.

Page 7 of 41

5. Sweet potato varietal identification in Ethiopia

This experiment was the initiative of SPIA Research Associate Frederic Kosmowski, working in Ethiopia with locally recruited enumerators and contacts through the NARS system in Ethiopia. The objective was to assess the accuracy of three household-based methods for identifying sweet potato varieties using DNA fingerprinting as the benchmark. The methods used were:

A. Elicitation from farmers with basic questions for the most widely planted variety; B. Farmer elicitation on five sweet potato phenotypic attributes by showing a visual-aid protocol C. Enumerator recording observations on five sweet potato phenotypic attributes using a visual-

aid protocol and visiting the field D. DNA fingerprinting using GBS on samples from sweet potato leaves (as the benchmark).

Data were collected in early 2015 from 259 plots in Ethiopia. Leaf samples were taken, DNA was extracted by ILRI in Addis, and plates for sequencing shipped to Diversity Arrays in November 2015. The reference library was collected and has been complemented by sequencing accessions from the CIP genebank. Initial analysis shows the following results:

Method D (DNA analysis) finds that 63% of the farmers are cultivating improved varieties and 37% are using Local varieties;

Method A produces almost identical results in the aggregate: across all responses 64% of the cultivars are self-identified as improved, 35% as local and 1% unknown.

However, 30% of the actual (DNA-based) improved varieties were identified by farmers as Local and, of the 85 actual Local varieties in the sample, slightly more than half were wrongly identified as improved.

Variety names (for both improved and Local) given by farmers delivered inconsistent and uncertain varietal identities and only 4% of the farmers were able to correctly identify the registered names of specific improved varieties.

Visual-aid protocols employed in methods B and C were more accurate than method A, but still far below the adoption estimates given by the DNA fingerprinting method.

Results here suggest that estimating the adoption of improved varieties of sweet potato in this area of Ethiopia with methods based on farmer self-reports is not reliable and indicate a need for wider use of DNA fingerprinting, which is likely to become the gold standard for crop varietal identification across many crops and countries, subject to further testing.

Page 8 of 41

Table 1 - Comparison of methods for estimating % adoption of improved varieties in SIAC program (DATA COLLECTED – SUMMARY BY SEPT 2016)

Focus (Study team – SIAC activity)

Sample size Corresponding expert opinion estimate

Farmer: Asking for variety name

Farmer: Asking “Improved or local”?

Farmer: Showing range of reference seeds

Farmer: Series of questions on phenotypic attributes

Experts: Visit field and identify by specific variety name

Experts: Visit field and identify “Improved or local”

Experts: Enumerator takes photo and experts identify

Experts: Seeds collected from farms and shown to experts

DNA Reference

Cassava, Ghana (MSU / IITA – Activity 1.1)

914 (all methods)

36% (DIIVA, 2009)

1% 6% 2% 5% 15% 4% – 31%

Beans, Zambia (MSU / CIAT – Activity 1.1)

Between 736 and 855 (varies)

9.5% (DIIVA, 2009)

4% 13% 71% 18% 15% 16%

Maize, Uganda – Leaf (SPIA / Diversity Arrays – Activity 1.1)

550 How to handle don’t knows?

46% Didn’t uniquely identify..

Pending from Andrzej

Maize, Uganda – Grain (SPIA / LGC – Activity 1.1)

550 Ditto 46% Ditto 100%

Sweet potato, Ethiopia (SPIA / Diversity Arrays – Activity 1.1)

259 4% 64% Ditto 63%

Cassava, Malawi (SPIA / Diversity Arrays – Activity 1.1)

1,200 61% (DIIVA, 2009)

How to handle don’t knows?

19% 70% 10%

Wheat, Bihar (MSU / CIMMYT / ICRISAT – Activity 2.1)

3,400

Lentil, Bihar (MSU / CIMMYT / ICRISAT – Activity 2.1)

3,400

Cassava, Vietnam (MSU / CIAT – Activity 2.1)

1,000

Rice, Indonesia (MSU / IRRI – Activity 2.1)

810

Cassava, Nigeria (IITA – Activity 3.1)

GIFT Tilapia, Philippines (World Fish – Activity 3.1)

Page 9 of 41

Activity 1.2. Develop protocols for tracking diffusion of natural resource management technologies

A call for pilot projects under this activity was issued by MSU in July 2013 and two studies were commissioned:

1. Innovative use of mobile phone based applications in tracking adoption of Natural Resource Management Technologies in India (CIMMYT), and

2. Hyperspectral signature analysis: a proof of concept for tracking adoption of crop management practices in Gazipur, Bangladesh (IRRI)

The final technical reports for the two competitively selected pilot studies funded under this Activity were received by MSU in April 2015 (from CIMMYT) and June 2015 (from IRRI). The reports were reviewed by SPIA and MSU, and comments of this review were shared with the authors. SPIA is publishing an impact brief on the CIMMYT study to highlight the results and lessons learned on the application of an Integrated Voice Response System to track the adoption of resource management technologies and farming practices.

Overall, the pilot study on the mobile phone based Interactive Voice Response System (IVRS) method was inconclusive about the validity of results collected through the IVRS model compared to paper based survey results. But this study has helped identify several key lessons that can guide future applications of such methods. Some of these insights/lessons gained from this pilot are:

Monitoring the adoption trends is a dynamic and time consuming process. The use of IVRS can help reduce some of the time and money costs.

The IVRS technology used to collect data on adoption resolves the issue of scalability as it can be inclusive of all types of locations with mobile penetration, and does not have bias towards the type of phone handsets and service providers thus increasing the reachability especially in rural environments.

The application of the IVRS based survey is neutral to any of the agricultural or natural resource management technologies and practices. The set of survey questions and the question sequence can be easily adapted on any technology and its adoption and can be customized in the system.

Farmers do appreciate the limited time that they have to spend with IVRS as compared to the long paper surveys done conventionally. However, the use of this method requires basic literacy and understanding of the use of mobile phones.

The technology is simple, customizable, and scalable. But it is suitable only for short surveys (i.e., focused on only one technology at a time).

The project had envisioned the risk on validity of the data collected through IVRS if the farmers are not able to understand the purpose of this survey and disconnect the phone. Thus it is important to run pilots and awareness creation on how to respond to the IVRS based survey.

For the IRRI study, the research team was unable to obtain hyperspectral imagery for the test site (as initially proposed) and resorted to using alternates (Landsat 8 and MODIS). SPIA has a number of concerns regarding the study, in particular whether this is the right type of remote sensing imagery that should be used. The lessons from these two pilots have significantly informed our strategy for Activity 2.2 implementation, which was the intention when the SIAC program was designed. Indeed, a contract to Nong Lam University and UC Santa Cruz has been awarded under that activity to apply alternative remote sensing approaches to assessing AWD adoption in Vietnam. Regarding cell-phone surveys, we have understood more about the biases from phone surveys and constraints regarding assembling a sample frame, and regarding remote sensing, through having this work externally reviewed, we have understood more about the heterogeneity of remote sensing approaches and the strengths and limitations of different methods.

Page 10 of 41

3. Measuring crop residue cover in Ethiopia

This pilot study that was added in 2015, but is directly managed under SPIA. The experiment was the initiative of SPIA Research Associate Frederic Kosmowski, working in Ethiopia with locally recruited enumerators and contacts through the NARS system in Ethiopia. The study is testing six alternative methods of crop residue coverage measurement among a sample of rural households in Ethiopia. These methods are compared against a benchmark, the line-transect method. The alternative methods compared against the line-transect include:

A. Interviewee (respondent) estimation; B. Enumerator estimation visiting the field; C. Interviewee with visual-aid without visiting the field; D. Enumerator with visual-aid visiting the field; E. Field picture collected with a drone and analyzed with image-processing methods F. Satellite picture of the field analyzed with remote sensing methods.

The survey experiment was implemented in five enumeration areas located in the sub-humid areas of East and West Shewa zones. In each enumeration area, 12 panel households from the Ethiopian Socio-Economic Survey (ESS) were interviewed. In addition, 28 households were randomly selected to participate in the experiment. Data collection took place in late 2015 and early 2016. Low-cost drones (Phantom 2+) were used to capture aerial pictures of the surveyed fields. Close supervision ensured collection of high quality data and resulted in a total sample of 197 households and 314 fields observed. After survey completion, two archived full scenes of Landsat 8 Thematic Mapper satellite imagery were acquired from the United States Geological Survey’s Earth Explorer imagery search and delivery website. These images match the dates (late 2015 and early 2016) associated with each field location. A Normalized Difference Tillage Index was calculated for each field. The experiment delivered the following results:

Survey-based methods (A, B, C, D) tend to underestimate field residue cover.

In continuous analysis, suing boxplot and scatterplots, correlations with the line-transect benchmark ranged from -0.25 (for Method E) to 0.76 (for Method D). The best, i.e., closest to the benchmark, estimates came from the two visual-aid protocols.

In categorical analysis (>30% cover or not), visual-aid protocols (C and D) and the remote sensing method (F) perform equally well with greater than 80% accuracy.

Among survey-based methods, the strongest correlates of measurement errors are total farm size, field size, distance and slope.

Results deliver a ranking of measurement options and suggest a wider use of visual-aid protocols among survey practitioners and researchers for accuracy, cost and ease of implementation. Remote sensing provides a good measure only in terms of categorical analysis but is much harder to implement for statistical institutes.

Activity 1.3. New institutional approaches to collecting technology diffusion data

Most diffusion surveys in the past have depended on CGIAR research teams, either working on their own or working in collaboration with national programs and statistical services to generate the data. In many countries, there are private market research firms as well as private survey firms engaged in carrying out household surveys for academic purposes. A call for proposals was issued by MSU with a focus on doing a case study in India. The call was issued in February 2015, and applied to either for-profit or non-profit entities with the relevant capacity.

A total of six proposals were received and after review carried by MSU, proposals received from two private sector firms based in New Delhi (Synergy Technofin and Creative Agri-Solutions Private Limited-CASPL) and one firm based in Chennai (Nathan Economic Consulting India Private Limited) were recommended to SPIA for funding. After receiving an approval from SPIA, MSU established Letters of Agreement with the three firms to undertake the pilot studies to test the innovative approaches. The scope of these pilots is outlined below:

1. Led by Synergy; Haryana (Karnal) and Bihar (Vaishali); Technologies: Zero till, direct seeded rice, LLL; Wheat-rice based farming systems

Page 11 of 41

2. Led by CASPL; Haryana (Karnal) and Punjab (Ludhiana); Technologies: Zero till, direct seeded rice, LLL; Wheat-rice based farming systems

3. Led by Nathan; AP (Anantapur and Kurnool); Technologies: 15 soil conservation measures promoted by ICRISAT; Groundnut farming system

To validate the estimates of technology adoption to be obtained from the three pilot studies, MSU is conducting representative surveys in the 5 study districts by using a more ‘traditional’ approach. This approach consists of working with a survey firm in India (identified from a process of issuing expressions of interest) to help with the logistics of doing data collection. The questionnaire and sampling design was developed by MSU with little involvement of the contract survey firm. But the survey firm provides enumerators (hired specifically for this survey), organizes training for the enumerators (with one MSU PhD student actively participating in the training of enumerators and making sure all the field activities are planned as per the survey design), takes charge of programming the survey questionnaire as a CAPI survey, provide logistical support to the field staff, and receives data, does data quality checks, data verification, and submits the clean data to MSU. All data collection for the validation surveys has been completed and currently the data are being reviewed, cleaned and organized for analysis. All the three contracted firms have completed the field work and submitted their deliverables. MSU will review the outputs and summarize the findings of this pilot project in a report format by end of April 2016.

Activity 1.4. Develop and disseminate best practices for collecting diffusion data

The idea with this activity is to take stock of activities, results and lessons learned from activities 1.1, 1.2 and 1.3, in order to generate guidance for the CGIAR system more broadly. This activity will be organized in the form of a workshop and earlier discussions with SPIA and PIM (F. Place) has resulted in the decision to organize this workshop in Boston post-AAEA meetings (August 3-4, 2016). This workshop will bring together results/learning from SIAC Objectives 1 and 2, and other private and public sector partners involved in finding innovative ways to collect technology adoption in developing countries. The main objectives of this workshop are to:

1. Take a stock of current and innovative methods for measuring adoption of agricultural technologies 2. Share and discuss results and insights from pilot studies and experiments conducted to establish proof

of concepts to harness the potential of new methods for tracking adoption of agricultural practices and other types of technologies

3. Further the discussion on scaling up proven methods for measuring technology adoption We are expecting more than 40 participants to attend this workshop. Information about the workshop, including the draft agenda and the participant list, is available from the SPIA website. A document summarizing the main outcomes of the discussion of this workshop will be developed as a deliverable of this Activity.

Page 12 of 41

The objective here is to compile and make available the best information on outcomes that are at least plausibly attributable to CGIAR research outputs, and on a large-scale. This is where a key bench-marking function for the CRPs is most obviously fulfilled by this program. Large gaps in existing adoption databases for genetic improvement technologies (activity 2.1), natural resource management technologies (activity 2.2) and policy-oriented research (activity 2.3) will be filled for priority regions. In addition, under activity 2.4, the World Bank Living Standards Measurement Study-Integrated Surveys of Agriculture (LSMS-ISA) team and SPIA and Centers are working together with NARS partners and statistical agencies to see how some of these processes can best be integrated into existing surveys to reduce cost and increase frequency of data collection. MSU is exploring similar objectives in Zambia and Mozambique and in dialogue with Indian counterparts for a similar objective.

Activity 2.1. Organize the collection of crop germplasm improvement research related direct outcomes

Under the SIAC project objective 2, this Activity (2.1) has expanded on the DIIVA and TRIVSA projects that have come to a closure, and focus on the collection of varietal diffusion data in South and Southeast Asia.

MSU is leading a process for which varietal release and varietal adoption data are collected for 62 crop x country combinations (CCCs) (which increases to 130 if we count individual states within India and regions within China as equivalent to countries – they all have their own data collection efforts) using expert opinion elicitation methods. Towards the planning of Activity 2.1, a two day inception meeting with Center and NARS partners was held in Bangkok on January 15-16, 2014 for a total of 35 participants. Based on the discussion and input from resource persons and participants, a guideline document on the methodology for collecting varietal release and varietal adoption data using expert elicitation methodology was finalized by MSU and shared with all the Centers and NARS partners. Subsequent to the inception meeting, each participating Center prepared a budget and workplan, upon which MSU established sub-contracts with the centers to collect varietal release and adoption data (using expert elicitation method) for 130 CCCs (see Table 2).

For 3 CCCs (all legume crops), MSU will work directly with NARS to collect the information. These include chickpea in Pakistan, and Lentil in Bangladesh and Nepal. For the former two CCCs, MSU has identified and contracted local NARS partners (NARC in Pakistan and BARI in Bangladesh) to collect the information and develop the two datasets by mid-2015. The NARS partners in Nepal were also contacted for their assistance in completing this Activity for lentil crop. But they have not been able to give their commitment to complete this task. MSU is working with the ICARDA researcher based in India to find an appropriate partner to collect this data for Nepal.

For the work contracted to CGIAR Centers, activities have progressed as per the plan. Towards the implementation of this Activity, CIMMYT organized a training workshop in August 2015 in Nepal for the NARS coordinators. Sushil Pandey and M. Maredia participated as trainer and resource person at this workshop. A similar training workshop was planned by IRRI in September 2014 in Laos, by ICRISAT in October 2014 in India, and by CIP in China in February 2015. Sushil participated in all these training workshops as a trainer and resource person. M. Maredia and T. Kelley participated in the India workshop in October 2014. CIAT has identified a regional economist to lead this Activity working closely with the NARS coordinators in each CCC.

Since the last report, Centers have made significant progress in completing the data collection for their targeted numbers of CCCs. Table 3 provides a summary progress report on work accomplished and still pending towards completing data collection for the two databases (varietal release and varietal adoption). Overall, data collection to compile the two databases has been completed for all but 3 CCCs. This represents an overall achievement of 98% of the targeted numbers of CCCs. Three centers have completed the data collection for 100% of their CCCs, and have also submitted the technical reports and the two databases. The LOA for ICRISAT, CIP and IRRI has been extended till end of July or August and they plan to submit the datasets for the remaining CCCs over the next month. According to the last progress report received from CIP, they will have completed all EE workshops by August 9.

OBJECTIVE 2: Institutionalize the collection of the diffusion data (OUTCOMES)

Page 13 of 41

Table 2: Final list of CCCs as per the workplan submitted by Centers (July 2014) – amended in September 2014

Country Rice Maize Wheat Barley Sorghum Ground-nut

Chick-pea

Pigeon pea

Lentil Cassava Potato Sweet potato

ALL

Afghanistan 1 1

Bangladesh 1 1 1 1 1 5

Bhutan 0

Cambodia 1 1 1 3

China (provinces listed below)

8 8 6 2

1 12 9 46

India (states listed below)

4 8 6 4

4 2 6 3 37

Indonesia 1 1 1 1 1 1 6

Iran ** 0 0 0 0 0 0

Laos 1 1 2

Malaysia 1 1

Myanmar 1 1 1 1 1 5

Mongolia 0

Nepal 1 1 1 1 1 5

Pakistan 1 1 1 1 1 5

Papua New Guinea 1 1

Philippines 1 1 1 1 4

Thailand 1 1 1 3

Vietnam 1 1 1 1 1 1 6

Total 21 25 17 5 0 5 3 1 7 10 23 18 130

Lead center IRRI CIMMYT ICRISAT CIAT CIP/RTB

Commitment from lead center

21 39 15 10 41 127

Gap * 3 0 3

*MSU will work directly with national programs or consultants to get information for the two data base for these 3 CCCs. ** Due to US Government’s restrictions on ‘working’ with Iran, five CCCs have been removed from MSU’s workplan and LOA.

Page 14 of 41

Table 3. Data collection status by Centers, as of the end of July 2016.

Total mandated CCCs

Data collection in mandated CCCs

Center Completed To be completed

Percentage completed

Report submitted

Databases submitted

CIMMYT 40 40 0 100% Yes 40

CIAT 10 10 0 100% Yes 10

IRRI\a 21 21 1 100% Yes 21

ICRISAT \b 15 15 0 100% No 11

CIP 41 39 2 95% No 18

MSU\c 3 2 0 67% Yes 2

Total 130 127 3 98% 102 \a For Indonesia, the NARS collaborator plans to implement the EE method to collect adoption data for Upland rice. This will be in addition to the adoption estimates obtained for Lowland rice in Indonesia using the seed sales data. \b ICRISAT has indicated they will share the outputs of 2 additional CCCs (groundnuts in China) for which they are collecting the data using similar methodology. Similarly, CIP may submit data from Bhutan for potato, in addition to the 41 CCCs in their workplan \c Adoption data for one CCC -- Lentils in Nepal will not be completed due to non-response from the local NARS partner identified by ICARDA last year.

MSU, in consultation with SPIA had identified the four CCCs for doing validations of adoption estimates to be derived using expert elicitation method or secondary data sources. Two methods are being used for validation—estimating adoption using representative farmer surveys and DNA fingerprinting on all or a sub-set of seed samples. The four CCCs identified for validation of Activity 2.1 are:

Wheat in Bihar (state level)

Lentil in Bihar (state level)

Cassava in Vietnam (country level)

Rice in Lampung Province, Indonesia (province level)

Field work (data and sample collection) for all these CCCs has been completed. Samples have been finally received by the labs (ICRISAT lab for wheat and lentils; IRRI lab for rice; and CIAT lab for cassava) and genotyping analysis is currently undergoing. No DNA fingerprinting results have yet been received for these validation CCCs.

As part of the audit requirement, and to collect opinion and assessment of experts on the elicitation methodology used in Activity 2.1, MSU has designed a survey (see Annex 2), which is being sent to all the EE workshop participants. This email and online survey is being facilitated by each Center focal point with the help of their national coordinating partners. The results of this survey will help assess the approach used to collect crop varietal adoption data across different CCCs, and to get some additional information on the status of the seed system for specific crops. Hopefully, the feedback from this survey will help us improve this methodology of collecting adoption data in future studies.

Activity 2.2. Organize the collection of natural resource management (NRM) research outcomes

This was initially part of the Michigan State University sub-grant but it was agreed in Jan 2014 that SPIA would manage this part of the program. Following a delayed start after this work was transferred back to SPIA, a call for Expressions of Interest was finally issued in October 2015, for case-studies focused on the following priorities NRM practice – country combinations:

Page 15 of 41

Table 4 – Priority NRM practice-country combinations for call for EoIs issued October 2015

PRIORITY NRM PRACTICES PRIORITY COUNTRIES

AGROFORESTRY (PARTICULARLY “FERTILIZER TREES”, LEGUMINOUS FODDER SHRUBS)

Kenya, Zambia, Zimbabwe, Rwanda

ALTERNATE WETTING AND DRYING (AWD) IN RICE PRODUCTION SYSTEMS

China, Vietnam, Philippines, Indonesia, Myanmar, Bangladesh

CONSERVATION AGRICULTURE IN MAIZE-BASED SYSTEMS

Zambia, Zimbabwe, Mozambique, India, Pakistan, Nepal, Bangladesh, Kyrgyzstan, Uzbekistan, Tajikistan, Turkmenistan, Kazakhstan, Iraq, Mexico

COCOA INTEGRATED CROP AND PEST MANAGEMENT (ICPM)

Cameroon, Cote d’Ivoire, Ghana, Liberia, Nigeria

MICRO-DOSING OF FERTILIZER IN MAIZE-BASED SYSTEMS

Kenya, Zimbabwe, Mozambique

INTEGRATED SOIL FERTILITY MANAGEMENT

Kenya, Rwanda, Burundi, DRC

From this call, 62 expressions of interest were received, and these were scored and review by SPIA in November 2015. Proponents from 18 expressions of interest as well as a number of resource people and SPIA secretariat members were invited to participate in a workshop in Rome in December 2015 comprising: discussions of the nature of the priority practices; the existing data infrastructure in place in the relevant countries that can serve as the basis for generating adoption estimates; prospects for remote sensing; and group work clustered around the six practices. The overall objective of the workshop was to try and broker collaborations across interested parties to ensure we got a strong set of full proposals.

Following the workshop, SPIA issued an invitation to the workshop participants specifying a set of 9 work packages that full proposals should be targeted towards. Proponents were invited to outline “core” and “upgraded” budget options for their proposals, with sets of activities to match. In February 2016, the 12 full proposals received (together covering a total of 25 of our practice-country combinations) were externally reviewed by a five-member expert panel, and a recommendation for funding proposals was put to the PSC for discussion and decision on 17th March 2016. Work by the proposal teams takes place throughout the remainder of 2016 and run to mid-2017. Hence, this is one of the activities that has made the no-cost extension to mid-2017 necessary. The following contracts are now in place for work being carried out between April 2016 and June 2017.

Table 5 – Funded NRM practice-country combinations with institutions and methodological approaches

TEAM NRM PRACTICE COUNTRIES METHOD(S)

MUTENJE ET AL, CIMMYT / ICRISAT

Conservation agriculture Mozambique, Zambia

Panel methods

HOLDEN ET AL, NMBU Conservation agriculture Malawi Panel methods, lead farmers and followers

ARSLAN ET AL, FAO (EPIC TEAM)

Conservation agriculture and Agroforestry

Zambia, Malawi Coordination across all other CA projects, analysis of secondary data

MAVZIMAVI ET AL, ICRISAT / UIUC

Conservation agriculture and micro-dosing

Niger, Zimbabwe

New surveys

Page 16 of 41

BUTLER ET AL, IFMR / U MICH

Conservation agriculture India - Bihar, UP, Haryana, Punjab

New sur vey + remote sensing

SONDER ET AL, CIMMYT Conservation agriculture Mexico Remote sensing

LOVELL ET AL, NONG LAM UNIVERSITY / UC SANTA CRUZ

Alternate Wetting and Drying Vietnam Remote sensing

VAGEN ET AL, ICRAF Fertilizer trees and fodder shrubs Zambia Remote sensing + HH survey

NKONYA ET AL, IFPRI / GEOPOLL

Integrated soil fertility management

Zambia, Kenya, Rwanda

Panel (Zamiba, Kenya) + SMS survey (Rwanda)

Paul Vlek has been appointed as a senior consultant to help guide this set of studies over the period of implementation, working with the SPIA team - James Stevenson and Nuri Niyazi in particular. A results workshop will be organized in May / June 2017.

Related to the documentation of NRM outcomes, James Stevenson represented SPIA in a workshop held in Cairns, Australia in June 2015, on assessing the effectiveness of landscape level interventions. The consensus in the group was that there is too little attention paid to demonstrating whether, and under what circumstances, a landscape scale approach is beneficial and will bring about impact. A paper reflecting these ideas, led by Jeff Sayer, is under review at the journal Ecology and Society.

Activity 2.3. Organize the collection of policy-oriented research outcomes

This activity focuses on another under-assessed area of CGIAR research – policy oriented research, in particular, identifying intermediate outcomes of CGIAR research that bear on macro level policies and practices plausibly linked to Center outputs. Work under this Activity attempts to document several categories of policy research related to:

Agricultural and relevant macro, trade and nutrition/health policies, all of which can have a large impact on economic incentives in agriculture, as well as modulating the poverty and nutrition impacts of some new technologies Management practices/protocols/agreements adopted at national or international levels

Levels and types of investments in agricultural research, roads, markets and other infrastructure

Expansion of training and institutional capacity (e.g., through farmer field schools)

Major international conferences / workshops around a highly relevant theme, e.g., IFPRI’s 2020 Vision conferences

Activity 2.3, therefore, focuses on outcomes of CGIAR policy-oriented research (POR) that have influenced significant policy changes related to agriculture, food and nutrition at the regional, national or global level. The aim is to compile and make available to CGIAR stakeholders the best available information on outcomes that are, at least plausibly, attributable to CGIAR policy research outputs. Ultimately, the objective is to build an inventory of CGIAR policy-oriented research outcome claims that have been externally vetted and passed minimum plausibility test, as a basis for selecting more in depth case studies of influence and impact could (through Objective 3 type activity).

In the first phase completed in 2014, consultant Mitch Renkow drew on earlier CGIAR PMS data files from 2006 through 2010 to compile a list of 93 outcome statements that credibly describe significant achievements of ‘deriving from Center POR outputs’. For each POR outcome, information is provided on the constraint or problem that was addressed, the key research outputs underpinning the outcome, a description of the specific POR outcome itself, what supporting evidence exists, and the region or country in which the outcome took place. Sixty-one of these were assessed as Category I “strong” cases – ones that satisfied certain specific criteria. In addition to the 61 strong outcomes, there were 32 other outcome statements that were deemed to have significant potential but required further documentation to be considered plausible cases of influence.

Page 17 of 41

Of the latter 32 outcomes, 17 were judged to require additional evidence linking the outcome to specific Center outputs, e.g., the existing outcome statement provided insufficient information to make a compelling case that the policy outcome could be reasonably attributed to the Center. Fifteen additional statements described outcomes that look promising, but either were at an early stage, e.g., described early outcomes emanating from pilot projects, or were simply not described well enough to make a strong case for being a POR outcome – but, again, appear to have good potential to generate meaningful policy outcomes. The categorization draws on: original Science Council commissioned external reviewers’ evaluations and the consultant’s own judgement about the strength of evidence/logic.

Phase 2, also led by Renkow, focused on updating the 2006-2010 database, primarily by searching the websites, annual reports and other relevant documents published by Centers and CRPs between 2011 and 2014, and applying a similar set of criteria to potential cases of POR influence/impact. This resulted in an updated (2011-2014) inventory of plausible case study outcomes. Typically, though with a few exceptions like IFPRI, there were much fewer cases to report over this latter period and with much less information to substantiate the stories, presumably due to the lack of any strong incentive to produce evidence of outcomes – compared to earlier years. A follow-up activity, but not as yet undertaken in SIAC Phase 1, entails offering Centers the opportunity to verify earlier submitted information or provide updated information to substantiate or modify earlier claims in the phase 1 & 2 inventories. That activity will take place in the Fall 2016, with the possibility of initiating an external validation process of POR outcome claims assembled under Phases 1 and 2. The latter may also feature in a SIAC Phase 2.

Two other Activity 2.3 related outputs are worth noting here:

IFPRI, the PIM CRP and SPIA recently co-sponsored a Workshop on Best Practice Methods for Assessing the Impact of Policy Oriented Research at IFPRI HQ in Washington DC. The workshop brought together more than 40 people, including evaluation experts from within CGIAR, the academic community, donors, and developing country policymakers. The workshop format was designed to foster the expression of different perspectives on the current state and prospects of impact assessment of POR. One of the workshop’s objectives was to seek agreement on realistic expectations for what can and cannot be achieved in evaluating the impact of different types of policy research, and how best to undertake the work. Key findings of the workshop can be found on the IFPRI website: https://www.ifpri.org/publication/workshop-best-practice-methods-assessing-impact-policy-oriented-research-summary-and

Renkow authored and presented a paper on ‘assessing the impact of policy-oriented research in the CGIAR: methodological challenges and reasonable expectations’ at the International Conference on Impacts of Agricultural Research – Towards an Approach of Societal Values (French National Institute for Agricultural Research INRA, Paris, November 3-4, 2015). The paper offers a critical assessment of efforts by the CGIAR and kindred national agricultural research institutions to evaluate the welfare impacts of policy-oriented research conducted under their auspices.

Activity 2.4. Long-term institutionalization of collection of adoption data

SPIA’s long-term vision in achieving this objective is to involve a broader and more diverse set of national institutional partners in the collection of adoption data so as to systematize the collection of nationally representative data (on a regular basis) in the most cost-effective way possible. MSU is working in India, Mozambique and Zambia to explore the integration of technology adoption data into existing surveys. On a parallel track, SPIA is working with the World Bank Living Standards Measurement Study – Integrated Surveys of Agriculture (LSMS-ISA) team through two researchers – Frederic Kosmowski and John Ilukor.

1. India (MSU):

The initial efforts (meetings and discussions) focused on ICAR to leverage existing data or future data collection efforts (cost of cultivation data) for the purpose of tracking and monitoring the adoption of improved varietal technologies (and any other technologies, if data are available) by farmers on a regular basis. While there was some initial interest, subsequent interactions suggested that ICAR did not have institutionalized data collection mechanism in place to integrate this data, and a better target for such efforts might be the Ministry of

Page 18 of 41

Agriculture or National Sample Survey Organization (NSSO) or to try and work at the state level (in 1-2 states) and see if the Department of Agriculture in a given state is open to this idea of institutionalizing the collection of technology adoption data at least on a pilot stage.

Since the SIAC update in February 2015, Mywish Maredia traveled to Odisha, India, in May 2015 for a day, and visited the Department of Economics and Statistics for the State of Odisha to find out more about the types of agricultural data being collected at a state level. From this visit and the desk review of questionnaires used to collect different types of data through surveys that are routinely conducted (such as the crop cut experimental data, input surveys, agriculture census surveys and NSSO surveys), the emerging conclusion is that India is a data rich country. There is an impressive amount of data being routinely collected (many at representative scale), and all these efforts are already institutionalized within the government system. However, despite these efforts, the fact remains that it is not easy to get an overall representative picture and trend of the adoption of different types of agricultural technologies that are generated by the Indian research system (and the collaborating CGIAR centers) due to a number of reasons, including government confidentiality laws.

Due to the characteristics of the way data are collected, processed and reported in India, there is limited utility of these data for tracking technology adoption at a representative scale. There is certainly room for improvements in this data system, but a local institution or a research center needs to champion this cause. The goal would be to make some changes in the institutionalized data collection system so that the data collected using public resources can serve the research and monitoring needs of the agricultural research communities. MSU has initiated a conversation along these lines with the National Institute of Agricultural Economics and Policy Research (NIAP/ICAR), and will continue to pursue these efforts: NIAP/ICAP Director has written to the Secretary of Agriculture to make household unit level data available to researchers, and intends to approach the Chairman of the Statistical Commission. However, to date, we have not been able to make any meaningful progress towards our objective with this work. The reason being that the Director of NIAP with whom MSU had initiated the discussions has left NIAP and according to the new Director the chances of influencing any change in the current data collection efforts is highly unlikely in the short time frame of the SIAC project.

2. Mozambique (MSU):

MSU has liaised with the Directorate of Economics and Statistics (DEST) within the Ministry of Agriculture and Food Security (MINAG) that is responsible for producing official agricultural statistics. The Integrated Agricultural Survey (IAI) is a routine data collection effort – representative at the provincial level – and done every 1-3 years. Last year, MSU reviewed the IAI survey instruments and provided feedback on integrating some technology specific questions in different sections of the survey. However, DEST was unable to incorporate all the suggestions as it was planning to conduct only a “light” round of IAI last year. They have also expressed interest in testing new methods of tracking adoption of varietal technology, especially using DNA fingerprinting, but no concrete plans emerged on implementing this method due to resource constraints. No other activities or plans for institutionalizing data collection were discussed or planned in Mozambique.

3. Zambia (MSU):

MSU reviewed the Crop Forecast Surveys (CFS) that is conducted annually by the Ministry of Agriculture & Livestock and Central Statistical Office. This survey is representative of small and medium scale holdings at the country level. Suggestions for modifications and addition of a one page section on the adoption of conservation technology were made to the CFS coordinator – this was pilot tested in February 2015, but was not implemented in the March-April round of CFS due to time constraints (increased survey length and time). However, the team has agreed to integrate a page of questions in the second follow-up round (post-harvest season in September-October 2015). During a recent visit to Zambia (on another project), MSU (M. Maredia) visited the Ministry of Agriculture and Livestock to get an update on this activity. All data collection has been completed and currently undergoing data entry and cleaning. Once the data are cleared by the Central Statistical Office, it will be shared with MSU and broader research community to assess the adoption of conservation technologies at the national level. This is an example of a successful outcome from this process of engagement with country statistics agencies.

Page 19 of 41

4. Ethiopia (SPIA and World Bank LSMS-ISA):

The third wave (2015/16) of the Ethiopia Socioeconomic Survey (ESS) presents an opportunity for integrating a number of questions related to the adoption of CGIAR-related agricultural technologies. The ESS is a nationally representative survey of 4,000 households, and is managed by Central Statistics Agency (CSA) via a network of some 300 resident enumerators.

SPIA were able to incorporate additional adoption-related questions into the ESS for the following technologies: Orange-fleshed sweet potato; Awassa variety sweet potato; Crop rotation in previous three years; Treadle pump; Motorised pump; Desi / Kabuli type of chickpea; Weather index insurance; Broad-bed maker; Improved livestock feed module. Data collection is complete, but we can expect to have access to the data in September 2016 – ahead of the formal release in 2017.

5. Uganda (SPIA and World Bank LSMS-ISA):

The Annual Agricultural Survey (AAS) is a new survey funded by the Ugandan government and implemented by the Ugandan Bureau of Statistics (UBoS). The survey instruments were pre-tested in the second season of 2015 and the main survey will start in September / October 2016. SPIA were able to incorporate questions into the AAS for the following technologies: bean varieties; cassava varieties; maize varieties; sweet potato varieties; sorghum varieties; agroforestry; livestock; conservation agriculture. John Ilukor has played an important role in testing the questionnaires, and SIAC funds paid for 20 tablet computers for UBoS to use in the survey.

In Uganda, the fourth wave of the Integrated Household Survey (the true LSMS-ISA panel survey) has been delayed, but a second round of the maize experiment (MAPS, described in Activity 1,1) has is currently in the field.

6. Malawi (SPIA and World Bank LSMS-ISA):

In Malawi, the Integrated Household Survey 4 (LSMS-ISA panel survey) is taking place in 2016. Training began in February 2016, and fieldwork started in late March 2016. John Ilukor and James Stevenson, with input from the FAO EPIC team, have introduced questions on a number of NRM practices into the survey instrument, relating to inter-cropping, crop residue management, agroforestry, crop rotation. John Ilukor helped in training enumerators with the Malawian National Statistics Office.

Page 20 of 41

While work under Objectives 1 and 2 paves the way for future ex post impact assessment studies, Objective 3 activities are focused on carrying out a number of impact assessments of CGIAR research and development initiatives along the entire chain of causation - from research investments to the System-Level Outcomes. Since this causal chain is long and complex, SPIA is approaching it from a number of different perspectives: case studies that focus on measuring the impact of CGIAR research on health and nutrition (activity 3.0); long-term large-scale studies of impact for major areas of CGIAR investment (activity 3.1); sets of micro-scale impact studies using experimental methods (activity 3.2) to provide evidence on the impact of CGIAR research-derived technologies to adopting households; studies of a number of under-evaluated areas of research, e.g. irrigation and water management; livestock, agroforestry and biodiversity (activity 3.3); a system-level meta-analysis of ex post IA of CGIAR research (activity 3.4).

Activity 3.0. Assessing the impacts of agricultural research on nutrition and health

Evidence of causal linkages between agricultural research and effects on health and nutrition is anecdotal at best, and yet the demand for such evidence has never been stronger. This activity is motivated by the need to broaden and deepen the evidence base regarding the potential for agriculture research and development to leverage health and nutrition benefits, and to improve our understanding of the multiple pathways linking those two variables. The intention is to complement, not to duplicate, on-going work in the A4NH and other CGIAR Research Programs, and giving priority to areas that until now are relatively “under-evaluated.” This prominently includes activities related to measuring the impact of research-derived interventions that plausibly impact on nutrition and health.

A competitive call for case studies was issued in July 2013. Led by Erwin Bulte at Wageningen University, an external review team identified an interesting portfolio of studies with different methods and focal technologies. An inception workshop for the five funded studies was held in July 2014 and since late 2014, we have had the five studies running as follows:

1. Adoption of high iron bean varieties in Rwanda (CIAT, Harvest Plus, Virginia Tech, Rwanda Agric Board)

The study is assessing the adoption and nutrition impacts of High Iron Bean (HIB) delivery efforts. This involves verifying the adoption of HIB varieties in Rwanda and then comparing bean consumption and iron intakes of adopters to those of non-adopters of HIB varieties. Two cross-sectional surveys of bean growers in Rwanda are planned in order to collect adoption and in depth socioeconomic and nutrition data from a sample of randomly selected 91 communities and 1104 bean growing farmers in Rwanda. The impact of HIB delivery interventions on nutritional outcomes, i.e. bean consumption and iron intake, will be assessed using two methods: propensity score matching and an instrumental variable approach. Cost-effectiveness of HIB delivery interventions will be calculated by comparing costs of delivery of HIB to the health benefits of the intervention (measured in terms of DALYs saved).

Progress: A progress report received from the team in December 2015 demonstrated that the project has overcome some logistical difficulties and is progressing well. Erwin Bulte has been providing ongoing support to the team to try and ensure they identify a good instrumental variable for their analysis, and that the follow-up survey rounds in 2016 include dietary diversity and food security modules. Household and community surveys were completed in 2015, and DNA fingerprinting will take place during 2016, with sampling from 120 communities taking place in January 2016. Survey preparation and implementation has taken a long time, including a long delay for a permit from the Rwandan government to allow blood sampling.

FINAL REPORT EXPECTED IN DECEMBER 2016

2. Shortening the hungry season through NERICA in Sierra Leone (IPA, MIT, Sierra Leone Agr Res Inst)

This study investigates the impact of early maturing NERICA rice on consumption and nutrition outcomes of farming communities in Sierra Leone. Most agricultural communities in Africa experience large seasonal variations in the price of crops. High prices and low stocks of staple crops prior to the new harvest create a

OBJECTIVE 3: Assessing the full range of impacts from CGIAR research (IMPACTS)

Page 21 of 41

“hungry season” when households reduce food intake with potentially important health and productivity impacts. All else equal, the worse the base level of nutrition, the more damaging a prolonged reduction in food intake is likely to be. In the previous phase of this project, high yielding rice (NERICA-3 and ROK16), of which NERICA rice is also early maturing, were allocated to four treatment arms with varying subsidies. Endline results from 2013 showed that NERICA treatment households harvested up to 5 weeks earlier and purchased less imported rice. A survey will be administered to a subset of this sample to estimate the impact of early NERICA-3 rice harvest on consumption and health at different points in the year.

Progress: This project has been granted a one-year no-cost extension owing to disruption caused by the Ebola outbreak in the country in 2014. The final report is now expected at end of December 2016. Early results show that children in households in that received NERICA seed (either for free or at 50% or 100% of market price) and agronomic training on how to grow it, see positive effects using anthropometric measures that persist up to the beginning of the next hungry season. The coefficients for the same measures for the group that did not receive training but did have access to NERICA at the same fractions of market price are positive, but not statistically significant and much smaller than those on the treated and trained group at the end of the hungry season. Previous studies by the same authors have shown that NERICA is susceptible to crop failure when not grown under correct agronomic conditions, and these findings would suggest that farmer training may be a necessary condition for achieving certain development outcomes with NERICA.

FINAL REPORT EXPECTED IN DECEMBER 2016

3. Crop diversification for food and nutrition security in Malawi and Ethiopia (CIMMYT, Lilongwe University, Georg-August-University of Gottingen, Ethiopian Institute for Agricultural Research)

Crop diversification (CD) is advocated as an essential component for making agricultural systems more sustainable and remunerative. However, the role of CD on food and nutrition security for smallholders in SSA has not been rigorously examined. In principle, CD involves cultivation of more than one crop and/or variety belonging to the same or different species in space and time; to achieve higher spatial and temporal biodiversity on the farm. The objective of this project is to assess the ex-post impact of CD options in combination with improved maize varieties on food and nutrition under different social and market conditions. It aims to estimate the impacts of various types of CD (e.g., legume-maize rotations & legume-maize intercropping) on nutrition indicators such as calorie and protein consumption, food diversity, micronutrient consumption (especially iron, zinc, and vitamin A), and childhood anthropometrics. These objectives will be achieved through the analysis of panel data collected in 2010 and 2014 from 4,000 farm households in Ethiopia (2,300) and Malawi (1,700) from 39 and 16 districts, respectively.

Progress: The progress report received in September 2015 included a draft of a paper for Ethiopia that has evidence that the joint adoption of crop diversification and modern varieties has higher impacts on calorie, protein and iron consumption and diet diversity than adopting each practice in isolation. This result was not particularly unexpected but there was previously little empirical evidence to support claims that inter-cropping could have this range of impacts. The results also suggest that adoption of combination of CD and modern seeds has higher impacts than adopting each practice in isolation. Results from Malawi are expected in mid-2016.

FINAL REPORTS ARE EXPECTED END OF JULY 2016

4. Looking beyond income: impact of dairy hubs on nutrition in Tanzania (ILRI, Emory U., Tanzania NARS)

This study aims to assess the relationship between farmers’ participation in dairy business hubs and human nutrition in Tanzania in the context of the dairy value chain More Milk in Tanzania (MoreMilkiT) project, linked to the CRP Livestock and Fish. ILRI and its partners are working in two regions on designing and pilot testing dairy business hubs as a mechanism to increase smallholders’ dairy productivity and income. While the existing project will monitor annually changes in farmers’ uptake of technologies, milk productivity and dairy income, a plan to monitor change in household livelihood indicators like human nutrition is lacking. Still, prior studies have shown that an increase in milk production and dairy income at the farm level need not translate into increased consumption of milk and overall better nutrition for dairy farmers. The proposed study aims therefore to complement the MoreMilkiT project and to assess change in nutrition linked with changes in productivity and dairy income brought about by farmers’ use of the dairy business hubs

Page 22 of 41

Progress: The progress report received in January 2016 showed that this study has moved in a different direction than expected. The team are attempting to explain variation in nutritional status of household members for a sample of 373 households. SPIA sent detailed comments on the progress report to the research team led by Isabelle Baltenweck, to which they responded. The explanatory variable of interest is participation in a dairy hub, of which the researchers are hoping to identify the causal effect using a combination of endogenous switching regression (SPIA has some concerns that there could be identification from functional form) and an instrumental variable (which has yet to be determined from a number of potential candidates). The outcomes measures for the comparison are dietary diversity for women and children, and total household food expenditure. Econometric analysis is currently in progress, with a draft of the final report expected in later in 2016.

FINAL REPORT EXPECTED IN SEPTEMBER 2016

5. Nutritional impacts of irrigated horticulture in Senegal (Columbia., George Washington U., MDG Center)

Evidence is beginning to emerge about the pathways through which intensification of irrigated horticultural production can affect nutritional outcomes, but most of this evidence is focused on small-scale, home garden type interventions, rather than larger-scale, commercial and technologically advanced production. This study will provide new, experimental evidence on the nutrition and food security impacts of an ambitious irrigation expansion initiative in the Western Sahel, and on the pathways through which these impacts occur. It leverages a large, funded, randomized controlled trial that evaluates the impact of the PAPSEN-TIPA project in Senegal which works with groups of smallholder farmers, mostly women. PAPSEN-TIPA disseminates improved horticultural technologies and equipment based on past and recent agricultural research, including adaptations of drip irrigation technologies co-developed by ICRISAT and complementary vegetable seeds and cultivation practices. Previous agricultural studies predict the interventions will generate large impacts on horticultural and cereal production, women’s income, labor and time use (associated with water delivery to plots). These are all important potential pathways by which nutrition outcomes can be affected, and our survey data will shed new light on their relative importance. The broad research question addressed by this study is: when do agricultural productivity interventions also result in nutritional improvements, particularly for children? Specifically for this case study, the key questions are “Does the intervention improve diets? On its own, or only when coupled with nutritional communication (via mobile phones)?” “What are the effective mechanisms at work, i.e., income, diverse production, time use, etc.?”

Progress: This project got off to a slow start and got underway only in early 2015, but the December 2015 progress report showed that the project was back on track. In June-July 2015 the survey instrument was piloted in four villages in Senegal. The pilot data indicated that overall, dietary diversity was low among infants and young children as well as their mothers. All the study villages were visited in September-October 2015 to ascertain a list of all the households in order to allow the study team to randomly select households that had a woman with a child between 6-23 months at baseline. The pre-baseline survey was thus essential in order to allow efficient random sampling of the target population. During the pilot data collection, six focus groups were conducted to help inform the development of the nutrition education intervention. The topics discussed in the focus groups included food production, food procurement, infant and young child feeding practices, seasonal variation and sources of nutrition information. The focus groups have helped identify some potential barriers and enablers for the nutrition education intervention and will be combined with findings from the baseline data collection to identify the key infant and young child feeding messages that need to be targeted in the nutrition education intervention. Baseline survey results related to horticulture cultivated areas, income, food consumption, food security, anemia, women empowerment, women time use, and diet diversity for both control and treated are now available.

UPDATE EXPECTED AT THE END OF JULY 2016

Activity 3.1. Long-term / large-scale impact assessment studies

The basic idea behind this work is to generate studies that credibly document the impacts of successful CGIAR research adopted at scale and over the long term using best available methods. Estimating the direct and indirect impacts from widely adopted technologies and policies is of special relevance to CGIAR donors and

Page 23 of 41

other stakeholders, particularly in a climate of high accountability and expectation of linkages between agricultural research investments and socially desirable outcomes.

While experimental and quasi-experimental approaches potentially have much to offer in terms of rigorous estimation of causal effects during early stages of adoption and at limited scales within producer populations, other methods, often less quantitative and seemingly less rigorous but more comprehensive, are needed to estimate impact over longer time periods and larger spatial scales. In addition to measuring the effects on crop yields and total farm income (or nutritional improvements) of adopters, estimating the impact of widespread technological change requires consideration of effects on other groups. Widespread technological change often generates significant partial and general equilibrium effects on farm product prices and farm production resources, especially labor, but potentially land and other inputs that in turn have significant impacts on poverty, nutrition and other welfare measures affecting adopting farmers as well as other populations. Indeed, in many cases, it is believed these widespread indirect effects dwarf direct impacts in the adopting regions. The usual impact studies, which estimate producer and consumer surplus, take the first step of including effects on consumers of the product whose production efficiency has improved, and such studies undoubtedly have shortcomings that should be addressed. But in addition, they often do not in any way consider the indirect effects on farm input markets or on markets of production complements or substitutes. To what extent it is possible to demonstrate direct and indirect causal linkages from CGIAR-related technologies in these fairly complex pathways remains to be seen, but this is the goal of this activity.

In early September 2014, SPIA issued a call for expressions of interest to fund studies that seek to measure the impacts of widely-adopted CGIAR research related innovations. Seven studies were funded out of the 12 full proposals received (8 impact + 4 adoption studies) in January 2015. An inception workshop for the set of studies selected was held in July 2015 at IFPRI, DC. The workshop had two objectives: (1) to provide specific feedback on technical and operational aspects of the funded studies, and (2) to provide an opportunity for participants to exchange views on the operational and data-related aspects of long-term, large-scale studies of CGIAR research impact studies – for instance, lessons from DNA fingerprinting work to estimate varietal diffusion; reflections on using micro data for macro analysis; and challenges in sampling and extrapolation for such studies.

The seven funded studies are as follows:

1. Adoption and diffusion of C88 potato variety in China: Spatial variability of productivity gains and cost savings and value chain development (CIP, Virginia Tech, and Yunnan Normal Univ)

Potato variety Cooperation 88 (C88) is among CIP’s biggest single varietal successes to date. In 2010, C88 was estimated to be grown on about 400,000 ha, with economic benefits estimated at US $350 million annually. But there are questions about the validity of adoption estimates for the variety that were based on non-structured expert elicitation methods. The objectives of the study are to:

A. Obtain rigorous estimates of the adoption of potato varieties in Yunnan province B. Analyze the impact of C88 by comparing yields and costs relative to the varieties it replaced, and, C. Estimate market-level impacts of C88 by examining benefits along the potato value chain.

Progress: Household and community surveys, DNA fingerprinting exercises and interviews with various actors along the potato value chain (mainly potato chip producers) to understand qualitatively the value chain, have all been completed. Economic analysis is on-going. A surprising result (thus far) is the observation of some dis-adoption and degeneration of C88, which means that the variety’s impact has likely plateaued in Yunnan. The project is on track and over the next four months the study team will be finalizing estimates of the economic impacts of adoption of C88 in Yunnan province and completing their qualitative assessment of the value chain and writing up results.

FINAL REPORT EXPECTED IN NOVEMBER 2016

2. Estimating improved Tilapia adoption using DNA fingerprinting: Philippines and Bangladesh (WorldFish)

The study is being undertaken to update (and improve on) estimates of adoption in the Philippines and Bangladesh of an improved tilapia strain (GIFT) developed by WorldFish by using innovative tracking of fingerling diffusion. Specifically, it aims to characterize nucleus populations of key GIFT and non-GIFT tilapia

Page 24 of 41

strains using genome-wide SNP genotyping approaches, and to validate (or otherwise) recently completed field-based adoption estimates.

Progress: Successful collection of tissue samples from nucleus populations of all the major tilapia strains identified during the inception meetings has been achieved. The strains sampled are: GIFT-Malaysia, GIFT-Philippines, FaST, GET-ExCEL, Chitralada and BEST, which account for the majority of tilapia production in Philippines and Bangladesh. Genomic marker development has identified over 13,000 SNP markers, which will be used to characterize the nucleus populations and for testing the genetic origin of hatchery samples. Currently, hatchery tissue collections, and the compilation of government production records are underway and expected to be completed soon. The project is on track.

FINAL REPORT EXPECTED IN DECEMBER 2016

3. Adoption of improved lentil varieties in Bangladesh: Comparison between expert estimates, nationally representative farm household survey and DNA fingerprinting (ICARDA and Virginia Tech)

The overall objective is to document current lentil variety adoption levels and identify the determinants of adoption and trait preferences of lentil farmers in western Bangladesh. In particular, the study compares lentil varietal adoption estimates obtained by expert opinion with those obtained by household surveys and compares those with more reliable estimates generated from DNA fingerprinting.

Progress: Data collection, digitization and cleaning for 1000 households across 10 districts in western Bangladesh completed. Village level data for 52 sample villages has been collected and is being digitized, and samples have been taken and analysis is underway for DNA fingerprinting exercise. The project is on track. An early result shows the total area under lentil in these 10 districts alone is estimated to be well over 250,000 ha of which about 80% is under improved varieties (subject to DNA validation), whereas official statistics had the total lentil area (in 2010/11) at 70,000 ha and was expected to have grown by 10% each year to reach 100,000 ha for the same period.

FINAL REPORT EXPECTED IN DECEMBER 2016

4. A systematic and global assessment of the impact of CG technologies on poverty (IFPRI and World Bank)

The study will provide a systematic and global assessment of the overall impact of CGIAR research on growth, poverty, food security and environmental indicators by combining large-scale global multi-sectoral dynamic computable general equilibrium and households modelling using an outmatched dataset of household surveys covering 80 percent of global poor. The framework is supported by researchers used to using such tools and data to capture the full payoff of CGIAR research both in terms of macroeconomic and microeconomic, direct and indirect effects.

Progress: The primary focus to-date has been on assessment of productivity implications. Three complementary but highly different approaches have been/are being used: (i) a review of the evidence available in the existing literature; (ii) back-casting approach to assess productivity growth in key commodities, and; (iii) a Delphi approach to elicit opinions from experts—informed by evidence from 1 and 2. Progress is underway on all three approaches. The study leaders have adapted an enhanced MIRAGRODEP model to be able to perform the back-casting exercise at the macroeconomic and sectoral level. MIRAGRODEP is a recursive dynamic multi-region, multi-sector model CGE model. International economic linkages are captured through international trade in goods and foreign direct investment. The project is on track.

FINAL REPORT EXPECTED IN JANUARY 2017

5. Using global agricultural, health and demographic datasets to identify the impacts of CGIAR’s modern seed varieties since 1960s (UC San Diego and George Washington University)

The objective of this study is to undertake a comprehensive global assessment of the economic, demographic and health impacts of MV releases by integrating hundreds of spatially precise Demographic and Health Survey (DHS) samples from around the world with detailed information on the timing of MV releases and high-resolution geospatial data on crop distribution. The study proposes to estimate the impacts on the following outcomes:

Page 25 of 41

Agriculture: Yields, area planted, and use of agronomic inputs (fertilizers, irrigation)

Demography: Birth rates, sex ratios, infant and child mortality

Nutrition and Food Security: Anthropometrics (birth weight, weight-for-height and height-for-age); proxies of nutritional status (blood haemoglobin, night blindness as a proxy of Vitamin A availability); direct indicators of food intake (available across a range of food groups)

Economic Indicators: Wealth proxies (including asset indices), school enrolment and attainment, and use of health care. Night light data (post 1992) used as spatial indicator of economic productivity.

Progress: After harmonization and integration of various data sets, initial econometric analysis of MV diffusion effects on agricultural (yields) and demographic and health outcomes in 18,000 rural villages across 37 developing countries was completed. A report is being prepared summarizing all of the analysis using Evenson-Gollin MV diffusion data and DHS outcomes, and a research paper analysing the impact of MV diffusion on infant mortality (the most robust empirical result) is also being prepared. The use of GAEZ data to exploit biophysical conditions for improved empirical identification of MV diffusion effects is being explored. A country-level case study of impact of rice MVs in Cambodia is also planned.

FINAL REPORT EXPECTED IN DECEMBER 2016

6. Measuring the impact of IFPRI’s research on Strengthening Food Policy through Intra-Household Analysis on the behavior of international NGOs (TANGO)

The overall objective of this study is to assess the extent to which findings of the IFPRI research program on Strengthening Food Policy through Intra-Household Analysis have been widely adopted at policy level and, in particular, evaluate the extent to which research findings and policy guidance on use of intra-household gender analysis have been operationalized in smallholder agriculture programs, projects and components funded by OECD-DAC members and implemented by INGOs.

Progress: Three main tasks identified in the proposal relate to: the critical junctures analysis; gender policy analysis of DAC members; and agricultural project documents review for 4 selected least-developed countries. At its launch workshop in October 2015, the study team agreed on a detailed work plan for implementing these tasks and decided to hold a second workshop in February 2016 to review results, approve the mid-term progress report, and plan the implementation of Phase Two. The first task included a bibliographical review, a citation search and a documentary network analysis of the women in agriculture literature since 1994, as well as semi-structured interviews with key contributors to the IFPRI research program. These are completed. The second task involved a number of discrete activities including: an Inventory of gender policy documents of DAC donors; a key word search; qualitative analysis of gender documents; structured Interviews, and identification of pathways through which research results pass from originator to donor. Most of these are completed. The third task involves assembling documentation for projects retained for in-depth study but has met with little initial success (due to limited response of gender experts for phone interviews in the DAC donor HQ to facilitate access to this documentation). This task will now be shifted to the collection and analysis of project documents in Phase 2. Phase 1 is therefore completed (or soon will be) and the team is now moving on to Phase 2, the in-country fieldwork survey and analysis.

FINAL REPORT EXPECTED IN AUGUST 2016

7. Assessing the impacts of improved cassava varieties in Nigeria (IITA)

The study aims to document the extent of adoption of improved varieties of cassava in Nigeria – as a group and for individual varieties, identify the determinants of uptake and spread of these improved varieties, and estimate the causal effect of adoption of improved cassava varieties on crop yields, incomes, food security, and poverty. It also intends to investigate heterogeneity effects focusing on gender differentials in adoption of improved cassava varieties in Nigeria.

Progress: 2500 randomly selected cassava growing households have been surveyed across the four major geopolitical regions of Nigeria based on an earlier sampling frame. Data collection, which included a wide range of information related to adoption of varieties, plot level input-output information, and household consumption and other outcome variables, started in 2015 and was completed early in 2016. The household data indicates high variability in adoption across the regions, from 79% to 30%. A total of 7428 different leaf

Page 26 of 41

samples were taken for DNA finger printing. Extraction of DNA from all samples was completed, sent to Cornell University for genotyping by sequencing and based on those results, identification of improved cassava varieties has been completed by IITA. Data coding, cleaning and analysis has been done in STATA. Analysis is now underway. Comparisons between DNA fingerprinting and self-perception based adoption estimates will be made. The impact of adoption across a range of social and economic outcomes will be assessed. Project is on track.

FINAL REPORT EXPECTED IN JUNE 2017

All but one of these studies will be completed by January 2017. A workshop reviewing progress with SIAC Activity 3.1 (long-term large scale) studies is scheduled for 30 July 2016 in Boston, immediately prior to the AAEA meetings. Information about the workshop is available from the SPIA website.

Activity 3.2. Micro-scale impact studies using experimental and quasi-experimental methods

There are two parts to this activity led by Karen Macours at the Paris School of Economics: three studies were funded in 2013; and in 2014, a two-stage competitive call was launched (details below).

The 2013 studies addressed two of the following:

convincingly show the effect of adoption of a CGIAR technology on productivity (downward shift in marginal cost of production or upward shift of production function)

help understand the linkages/mechanisms from adoption to impact (on income /nutrition / good or bad environmental outcomes)

help understand other “deeper” questions that limit the effectiveness (sustained adoptability) of current CGIAR research – understanding the technology production process

address academically interesting questions related to behavioural/welfare/development economics etc.

Three studies were identified for funding:

1. Study of the impacts of adopting rainwater harvesting techniques on yields and soil quality in Niger (Tufts, Catholic Relief Services, ICRISAT)

This pilot study aimed at understanding the barriers to adoption of rainwater harvesting technology (demi lunes), as well as the effect of adoption on soil quality, yield, costs and revenues. Adoption of demi-lunes technology is low in Niger – less than 10% of the farmers are estimated to use micro-catchments on any part of their land. This despite research station and on farm trials showing positive impact on soil erosion/degradation, and reduced risk of crop failure.

The pilot was across 30 villages in the Dosso region of Niger: villages were chosen/targeted because they are most affected by soil degradation. Within each village, 25 farmers were selected and villages were randomly assigned to one of three interventions (a) training: basic training for RWH techniques, including the necessary equipment; (b) non-conditional cash transfer (UCT): training on RWH techniques and an unconditional cash transfer (cash advance) worth US$ 40, approximately 50% of the labor costs labor and equipment costs associated with constructing these structures on one hectare of land; and (3) conditional cash transfer (CCT): training on RWH techniques and a conditional cash transfer, whereby farmers receive a cash payment for every demi-lune of acceptable quality, similar to the per-RWH value of the unconditional transfer.

Progress: Baseline data collection and training was completed in February 2015. In November 2015, the endline survey captured impact on soil quality and yields (subjective measures, self-reported by farmers). This data is being analyzed and results will be available in July/August 2016.

The pilot results on training and adoption found the following:

Training: The interest in training was high – 85% of the households attended training regardless of the group (training, UCT, CCT) they were assigned to.

Adoption: households in the CCT group were less likely to construct demi-lunes on their plots (13 percentage points, significant at 1%). However, the number of demi-lunes constructed per hectare

Page 27 of 41

was the highest in the CCT group i.e., 47 per hectare versus 30 (training group). The technical norm suggests 250-300 per hectare. So, while cash transfers help ease credit constraints (farmers can hire labour), CCT seems to discourage such investment.

FINAL REPORT EXPECTED IN SEPTEMBER 2016

2. Drought Tolerant (DT) seeds alone or bundled with innovative financial technologies? A proposal to evaluate the impact of bundling DT maize and index insurance to reduce vulnerability in smallholder African farming (UC Davis, IITA, CIMMYT, Univ. of Georgia)

SPIA funded a planning workshop to determine the way forward on an RCT to evaluate the impact of bundling DT maize with index insurance by researchers from CIMMYT, IITA, UC Davis and Univ. of Georgia. Considering that DT maize does not perform well under extreme drought conditions (~20% of the time), bundling an insurance product that covers extreme events might lower insurance costs and deepen impact of DT technologies.

Progress: The outcome of the workshop was a full proposal submitted to a number of donors including USAID and CCAFS. USAID funded the study (US$ 2.2 million) in 2015, and the project will run through 2019. Outcome variables to be measured include adoption of DT seeds, agricultural investment (area planted, investment in yield) as well farm yields and real incomes (especially in years of drought). The study will also document nutrition and human development impact indicators (dietary diversity, food security).

WORKSHOP REPORT SUBMITTED

3. Learning regarding ISFM through farmer’s own experimentation during researcher-managed on-farm trials (Paris School of Economics, IITA)

This study sheds light on the biases in agronomic trials by analyzing how different variations of inputs in trials affect the input allocation in differential ways, by analysing (a) whether farmers randomly selected to participate have different results than farmers chosen by researchers, (2) whether participating farmers themselves adopt ex-post, and (c) longer term impacts. This is important because it is assumed that profitable technologies produced by the CGIAR are not adopted because of other binding constraints, but the evidence on profitability comes from research station or researcher-managed on farm trials, and hence may not be relevant for real world conditions.

Progress: The study commenced in November 2013 with identification of farmers interested in experimenting with new commercial products, and collection of socio-economic baseline data. 96 villages were assigned to two treatment and one control group, with 10 interested farmers in each village participating in the study across three seasons, offering an opportunity to understand how learning occurs. Standard agronomic protocols were followed wherein a scientist worked with each farmer and in each season in the treatment groups to implement an experimental trial on their land in which a random control plot be compared with 5 different randomly selected treatment plots. The treatments on each of the plots varies, as does the combination of treatments to be tested by each participating farmer. Hence, in a given season, a farmer tests different combinations of seeds (soy, maize, soy-maize intercrop) and fertilizer packages. Through August-September 2014, follow-up surveys were conducted, and in December 2014, qualitative data collection took place during the second season. Throughout agricultural seasons, monitoring of the trials continued, providing data on compliance with the experimental design, issues with pests, weather damages, and other confounding factors.

Take-up during the second season was much lower than the first – a high proportion of farmers (18%) selected for treatment did not participate for various reasons that included a lack of interest in crop cultivation during a short rainy season or miscommunication on land preparation between agronomists and farmers. Baseline data confirmed that community selected farmers for experimental trials are significantly different from randomly selected farmers – they have higher skill levels on all dimensions. While there were some difference in agricultural practices, this was mixed – significant differences for some practices (use of manure, fertilizer, hybrid seeds etc.) and no significant differences for others. At the end of the first season, while community selected farmers appeared to be more productive (control subplot production was higher compared to the same for randomly selected ones), the returns to treatment was not significantly different. And while all types

Page 28 of 41

of farmers adopted some of the practices (no significant difference between community and random), community selected farmers were more likely to have updated their beliefs about returns to inputs for maize production. The analysis of the future rounds of survey will shed more light on these fundamental questions in technology adoption.

The 2014 two-stage call was launched in mid-2014 with the objective of furthering our understanding of how Randomized Control Trials (RCTs) can contribute to an understanding of specific causal pathways from technology adoption. Three studies were contracted between November 2014 and January 2015, and an inception workshop for the studies was held at MIT, Cambridge MA, in February 2015. At the inception workshop, researchers invited to submit full proposals but not funded benefitted from discussions on the funded studies, and on the second day, received detailed feedback from other researchers (Alain de Janvry, Karen Macours, Tavneet Suri, MIT PhD Econ students, SPIA) to improve their proposals.

1. A multiple intervention approach to increasing technology adoption with a view towards scaling-up: Evidence from Mexico (QFD Mexico, UC Berkeley, ITAM, World Bank)

This study of how improved recommendations (based on soil analysis) and grants in combination with other extension services affects technology adoption (fertilizer, precision drill) will contribute to a better understanding of heterogeneity-adoption links. Heterogeneity is one among the many plausible explanations for low adoption, missing markets for credit, information, etc. being another.

As a part of the experiment, soil analysis is used to make input recommendations on fertilizer dosage and timing of application with additional information on precision sowing (using a drill) and timing of herbicide use. Studies have suggested that soil quality is central to understanding fertilizer adoption levels, and in the study area, farmers have been found to overuse urea but use much less KCl and no micro-nutrients. Besides providing recommendations in a document and explaining this to farmers, the experiment also provides information on prices, quantities and total costs for each recommended input; the geo-location of agro-dealer for fertilizer; and aggregates demand and coordinates logistics for hiring precision sowing drills. In-kind grant of US$ 150 worth of inputs (half of average per-hectare cost), an inflexible grant restricted to items on the recommendation list, or a flexible grant (any inputs can be purchased) is experimentally varied.

Progress: Work on the SPIA-funded aspects of this project began in January 2015 with a baseline, and 981 maize farmers were recruited for the study in February 2015. Between February and April 2015, various interventions were delivered. Mid-term survey that collected plant density information was completed in July 2015, and a commercialization survey was done in February/March 2016. By the end of November 2016, results from various follow-up surveys will be available. The study will also conduct an endline post harvest in November 2016 with a focus on welfare impacts, and these results should be available by May/June 2017.

Preliminary results from follow-up surveys indicate that take-up rates are significantly higher (not very surprisingly, grants matter) for farmers who received individualized recommendations and inflexible grants (T1), averaged recommendations and inflexible grant (T2), and averaged recommendations and flexible grant (T3) along with extension - compared to control (C) and farmers who received averaged recommendations and agricultural extension (T4). However, the take up of fertilizer and precision drill was similar regardless of the flexibility in grant i.e., similar rates for T2 and T3. Curiously, T1 (individualized+inflexible grant) take-up is lower than T2 (averaged+inflexible grant). Farmers’ certainty about soil quality increased in all treatments that received recommendations (T1 through T3), and their expectations of yield volatility was lower. And finally, whether recommendations were individualized or averaged did not seem to matter for plant density.

PROGRESS REPORT EXPECTED IN DECEMBER 2016

2. Drought resistance and water saving in rice production in Bangladesh (UC Berkeley, Tufts, IRRI)

This study of drought-tolerant, shorter duration rice (BD56) in Bangladesh will examine the impact on productivity, water savings and the local market for water, and welfare impacts. Much like other studies in this SIAC activity, it will examine the heterogeneity of impacts (large versus average farmers), and also, how diffusion occurs, by randomizing who the demonstrator farmers are. Such a variation (in demonstrators) is motivated by an interest in improving agricultural extension, and this study is only one of the three experiments along similar lines in India and Bangladesh by the same team.

Page 29 of 41

Progress: Work on this project began in May 2015 as a pilot experiment across 35 villages, instead of a full-fledged RCT because there was a large-scale flooding that destroyed crops and made census difficult. During the pilot, 5kgs of seed was provided to 5 farmers in each treatment village. Demonstrators were of five types. That is, they were (1) either randomly chosen, (2) the largest farmers, (3) picked on the basis of self-reported willingness to pay (WTP), (4) voted by peers, or (5) closest to the average farmer (village mean variables). Pilot data analysis confirms that BD56 offers a shorter duration, and affects the crop choice for dry season. That is, farmers who grow BD54 are more likely to plant wheat and pulses – increasing the number of cropping seasons from two to three in an agricultural year, compared to control. BD56 was also shown to produce reduced yields compared to longer duration varieties, but this varied by the type of demonstrator farmer. The pilot also showed that larger farmers spread information better.

For the full experiment, a census survey to select demonstrator farmers as well as the baseline survey was completed in April and May 2016. In 192 villages, 5 demonstrators have received 5kg of BD56 (about 35% of wet season rice area), and demonstrators are either (1) chosen randomly, (2) the largest rice farmers, or (3) were selected by an agricultural officer. Hypothetical (comparable) recipients selected by the same selection mechanism in 64 control villages have received BD51 (Swarna Sub1) to enable an assessment of impact. The first follow-up survey on production and welfare outcomes will occur in February 2017, and the study will progress independent of SIAC funds beyond June 2017 to track diffusion (August 2017 and August 2018) and complete a second follow-up survey in February 2018.

FINAL REPORT EXPECTED IN APRIL/MAY 2017

3. Social networks to promote new agricultural technologies in Nepal (Yale, ICIMOD)

The study, drawing on previous work in Malawi (paper), examines how social learning can improve agricultural extension and increase the adoption of new technologies (in this case, maize intercropping in Nepal). Intercropping adoption rates are below 10% in Nepal, and pilot data indicates people are not aware of the potential to intercrop maize with tomatoes, ginger, or French beans. Hence, an information intervention could increase adoption rates. At the same time, agricultural extension services face challenges in promoting new technologies, and there may be fairly inexpensive ways to leverage social learning to improve extension. Following the Malawi experiment where it was found that peer and lead farmers did no better than pure control but incentivizing them had a positive impact on adoption (of pit planting), this study experiments with the type of incentives (flat versus performance based). Besides impact data on yields and welfare (nutrition), this study will contribute to our understanding of the heterogeneity of impacts: there will be differential adoption by farmers from varying extension/information delivery mechanisms, allowing such an analysis of data. SPIA funding is for additional survey rounds of the existing experiment which began in 2014.

Progress: This project experienced significant delays due to the Nepal earthquake in April 2015 (two of the ten districts were severely affected), and a fuel crisis between September 2015 and February 2016. Hence, the timeline is being extended by a full year. That is, while the fourth survey round and related data cleaning/analysis ends November 2016, the contract will be extended so SPIA can have an intermediate output by April 2017 with results, and a final report in July 2017 (after the program ends).

Analysis of baseline data indicates that the lead and peer farmers have been chosen appropriately: lead farmers (community leaders and early adopters) and peer farmers (well-connected and seen as leaders but similar to the average farmer) are wealthier than others in the sample; they tend to own more land and get more extension visits in a year; and, they are more aware of maize intercropping. After one year of intervention, data shows that knowledge of maize intercropping has increased across all ‘treatment’ groups. Take-up data (adoption) is still being analyzed.

PROGRESS REPORT EXPECTED IN DECEMBER 2016

A mid-term workshop for the studies was held at FERDI, Clermont-Ferrand, France in June 2016, and included a discussion on lessons from the research implementation and early results, and the implications for SIAC Phase-II. This half-a-day workshop followed a larger ‘Learning for Adoption’ (2-day) workshop, coordinated by Karen Macours (PSE and SPIA), Alain de Janvry and Elisabeth Sadoulet (UC Berkeley), and Vianney Dequiedt (CERDI), that aimed to explore research avenues and innovative designs for farmer learning for adoption.

Page 30 of 41

Activity 3.3. Under-evaluated areas of CGIAR research

Many studies over the years have sought to document the impacts of agricultural research although the vast majority of these have focused on crop germplasm improvement, i.e., adoption and impact of improved crop varieties. As such there remain serious gaps in the extent to which impact assessment of other components of the CGIAR portfolio have been conducted. To fill this gap, one of the activities of the SIAC program targets what we consider to be under-evaluated areas of CGIAR research, such as livestock management, irrigation and water management, agroforestry, policy and social sciences, biodiversity and natural research management.

There are two parts to this activity: (1) as an initial step in the SIAC work on under-evaluated areas of CGIAR research, commissioning of critical reviews of the impact assessment work-to-date on specific research areas (irrigation, livestock, agroforestry etc.); and (2) a wider call for ex-post impact assessments to document the adoption and the direct impacts of CGIAR research (e.g. reduced water usage for irrigation or improved yield), and where possible, poverty, social, and environmental impacts of these innovations.

As an initial step, SPIA, in April 2014 commissioned Doug Merrey, a consultant with considerable experience in water management research, to conduct a critical review of the impact assessment work to-date on irrigation and water management research. This desk study includes IA work done within and outside the CGIAR, and evaluates how comprehensively and effectively these assessments cover the field of irrigation and water management research since 1990. Merrey’s report concludes that while CGIAR has invested an estimated US$ 800 million (nominal, not constant dollars) in irrigation and water management research over the past 25 years, it has substantially underinvested in economic, social and environmental impact assessments of this work. Only about 14 cases (out of 25 available) qualified as IAs of uptake, outcomes, and impacts of irrigation and water management research. There is no definitive answer to the question on overall influence, outcomes, and impacts of CGIAR water management research since 1991, and there is too much missing data to make credible claims, even as there is some evidence of positive impacts by individual projects/programs on broader scientific literature, policy and research agendas, and food security/incomes of farmers. The review identifies the major constraints and limitations, e.g., methodological, data-related, resource-related, etc., of previous studies and offers guidance regarding some specific candidates for IA studies of CGIAR research on irrigation and water management that have good potential for documentation.

The review of the impact assessment work on livestock and livestock-related research in the CGIAR is the second scoping study commissioned by SPIA to assess the impact evidence across these under-evaluated areas within the CGIAR portfolio. In March 2015, two consultants, Sam Jutzi and Karl Rich were commissioned to evaluate the extent and quality of ex-post impact assessment activity on livestock related research in the CGIAR to-date. The report was internally reviewed by SPIA, and finalized at the end of November 2015 with a foreword by SPIA Chair. The final report was published in May 2016 along with a SPIA commentary. As was the case in the previous scoping study, the literature does not provide strong evidence of impact from the CGIAR’s past investments in livestock and livestock-related research. Of the twelve studies that form the basis of the report, none offers evidence that would justify, in aggregate, the investment in livestock research in the CGIAR – estimated to be about US$ 1 billion (unadjusted for inflation). That does not mean that there has been no impact, nor does it imply that investments in animal agriculture have been unjustified. But it is clear there are no credible studies showing long-term or large-scale adoption of CGIAR-derived innovations or policies from livestock research. At the same time, it is important to note the existence of a larger set of micro-level case studies that have examined the link between the adoption of livestock-related technologies /management practices and their direct farm-level impacts. Some of those may offer useful analysis of impacts at the farm level. These too, in SPIA’s view, merit serious attention in terms of their potential for adding to the evidence base, understanding constraints to adoption and impact, and feeding a learning agenda. These qualifiers are not intended to detract from the thoroughness of this review or the importance of the main findings and recommendations. On the contrary, the substantial gaps identified by this and the irrigation and water management review underline the need for building a stronger and more rigorous evidence base in examining the impact of these under-evaluated areas of CGIAR research.

Given the dearth of evidence of impact of CGIAR research outputs in major areas of CGIAR investment, in June 2015, SPIA issued a two-stage call for proposals for impact assessment studies of under-evaluated areas of CGIAR research, particularly inviting studies in areas such as irrigation and water management, livestock

Page 31 of 41

management, agroforestry, biodiversity, social science & policy and NRM. Of the 26 EOIs received and internally reviewed in August/September 2015, 10 were invited to submit full proposals at the end of October 2015. Four studies were approved for funding in December 2015, as per below. An inception workshop was held on 10 March 2016 in Washington D.C., immediately prior to the meeting on the future of SIAC and SPIA. Contracts were finalized for all four studies and work has commenced.

1. Forest co-management in Guinea: a multi-scale, multi-output ex-post impact analysis (Virginia Tech and CIFOR)

This ex-post impact assessment aims to document the durable environmental and household livelihood benefits of research on the institutional management of forests under the LAMIL project in Guinea eight years after the cessation of project activities. The durability, spread, and impact of innovations generated between 2005 and 2008 in project treated forest reserves and adjacent villages will be estimated through a unique two-level analysis of environmental and household livelihood benefit. A household survey will be conducted in 8 villages adjacent to the project forest reserves and 8 villages adjacent to control forest reserves to document the durability of project innovations within treated forest reserves and adjacent villages, as well as the spread of project innovations to control forest reserves and their adjacent villages. Livelihood impacts will be inferred through the estimation of differences in forest, agroforestry, and agricultural benefit flows associated with project innovations in adopting households and otherwise similar non-adopting households.

PROGRESS REPORT EXPECTED IN NOVEMBER 2016

2. Adoption and impact of Alternate Wetting and Drying (AWD) water management for irrigated rice in the Philippines (North Carolina State University, IRRI and NIA)

IRRI and national agricultural research and extension system (NARES) partners have worked together to develop and promote the AWD water management approach for a number of rice-producing countries in Asia. This study aims to comprehensively determine the multi-dimensional impact of AWD in the Philippines, i.e., document the economic, poverty, socio-cultural, gender, and environmental impacts of AWD, as well as providing an estimate of the rate of return on the research investments made for the development of this technology. State-of-the-art quantitative impact assessment techniques (i.e., combined DID/ randomization) as well as qualitative approaches (i.e., network/contribution analysis) will be used to achieve project objectives, and will address: (1) selection issues and confounding factors, (2) impact heterogeneity (i.e., upstream vs downstream), and (3) poverty, socio-cultural, and environmental impacts.

PROGRESS REPORT EXPECTED IN NOVEMBER 2016

3. Assessing the adoption and economic and environmental impacts of Brachiaria grass forage cultivars in Latin America focusing on the experience in Colombia (CIAT, Michigan State University, Universidad de los Andes-Colombia, and CORPOICA)

Brachiaria grasses were introduced to Latin America from Africa in the early 1950’s and has reached a large adoption in Brazil. CIAT and other partners have led research and agronomic evaluation that has expanded the use of Brachiaria cultivars to other countries in Latin America. Some project reports and statistics on seed sales suggest a large adoption of Brachiaria grasses in these countries but there are no rigorous studies that have documented this adoption process. There is also no evidence of the impacts that the adoption of Brachiaria may have generated. This proposal aims to contribute to filling this gap by rigorously documenting the adoption and impacts (on livelihood and environment) of Brachiaria cultivars in Colombia using an econometric analysis of household level data, and by estimating aggregate impacts of Brachiaria in Colombia, Peru, Nicaragua and Costa Rica using existing data.

PROGRESS REPORT EXPECTED IN NOVEMBER 2016

Page 32 of 41

4. Assessing the downstream socioeconomic and land health impacts of agroforestry in Kenya (ICRAF, Vi Agroforestry, and Univ. of Illinois)

There is a dearth of evidence on the longer‐term impacts of agroforestry practice, despite its widely purported ability. The proposed study aims to make a significant contribution to addressing this evidence gap by comparing villages in three distinct locations in western Kenya where a number of integrated agroforestry practices have been intensively promoted with comparable villages where they have not. To address program placement and self‐selection bias, propensity score matching (PSM) and spatial econometric methods will be combined with difference‐in‐differences estimation. Qualitative methods will also be used to triangulate key findings and interrogate impact pathways, unforeseen outcomes, and unexpected quantitative results. Key outcomes of interest include household wealth status (as measured by asset ownership and other locally relevant wealth indicators); Progress Out of Poverty Index; Multidimensional Poverty Index; levels of livelihood diversification; and various land health metrics constructed through the analysis of satellite imagery, including soil organic carbon, soil erosion, soil pH, root depth restriction, and fractional vegetative cover.

PROGRESS REPORT EXPECTED IN NOVEMBER 2016

All four studies are scheduled for completion by the end of June 2017. These studies will provide high-quality evidence of the impact of some of the more important CGIAR outputs/outcomes that have remained under-evaluated, and showcase alternate approaches to IAs when traditional experimental or even the more credible of the quasi-experimental approaches are not feasible.

Activity 3.4. Meta-analyses at System level: various studies

1. A comprehensive empirical assessment of the returns to agricultural R&D worldwide (University of Minnesota)

Drawing on the newly completed version 3.0 of the InSTePP Returns-to-Research Database (a global compilation of the returns to research evidence spanning the period 1958 to 2015), the research team led by Phil Pardey is preparing several themed summaries and undertaking a formal analysis of new data on returns to agricultural research to develop a series of briefing notes and technical reports which will:

Summarize the new and comprehensive returns-to-research evidence emphasizing graphical, tabular, and mapped representations of the data, with associated descriptive text and commentary.

Undertake a critical review of agricultural R&D returns, investigating on the one hand whether or not there is a tendency for the returns to investments in R&D to decline over time and on the other hand analyzing methodologies that will enable the historical internal rate of return evidence to be recalibrated into modified internal rates of return.

SPIA received a proposal from the International Science and Technology Practice and Policy (InSTePP) Center at the University of Minnesota on 1 November to support the above program of work which extends the work that InSTEPP have already been doing in estimating the economic returns to agricultural research using meta-analysis techniques. A recommendation to fund the study was put to the SIAC PSC in late November 2015 and approved. Target date for completion is November 2016.

Progress:

“Returns to Food and Agricultural R&D Investments Worldwide, 1958-2015”: A complete draft of this brief has been prepared and formatted, and is ready for posting on line.

“Returns to Food and Agricultural R&D Investments in Sub-Saharan Africa, 1975-2014”: A complete draft of this brief has been prepared, and is under review for publication by Food Policy.

“Returns to Food and Agricultural R&D Investments in Latin America and the Caribbean, 1970-2013”: Data work completed, the associated figures and tables have been prepared, and drafting has begun.

“The Returns to CGIAR Research, 1970-2015”: The data are finalized and analysis and write up of this brief will begin later this summer.

“Are Agricultural R&D Returns Declining and Development Dependent?”: An initial round of econometric analysis has been completed and a first draft of the paper has been prepared. The paper

Page 33 of 41

was accepted as a selected paper for presentation at the Agricultural & Applied Economics Association (AAEA) annual conference in Boston in early August 2016.

2. Impact of new technology on agricultural productivity – a re-examination (Bob Herdt & Sarah Mine)

There is a long and complicated pathway from adoption of technologies/practices to impacts (as in the CGIAR System Level Outcomes). SPIA, under the leadership of Bob Herdt and with the assistance of consultant Sarah Mine, has sought to examine the first and most direct link in this pathway – from adoption of technology/practice to productivity or other measurable outcomes, with these specific objectives in mind: (a) to identify the evidence, in the form of impact assessment (IA) studies, (b) to characterize the IAs, and, (c) to evaluate the quality of the evidence using pre-determined criteria. The study has progressed in two phases. The first step of this study involved designing, testing, and finalizing the approach to characterizing the IAs, and identifying and testing criteria that could be used to assess the quality of evidence. SPIA had a list of 214 studies that were identified (and passed the first screen) in the Loevinsohn et al. review (2013) submitted to DiFD. Hence, instead of building a database of relevant IA studies afresh, the characterization exercise initially focused on this set of 214 IAs. The second step involved (a) completing the in-depth quality review of a sub-set of studies from the list of 214; (b) expanding the database of IA studies that have examined the adoption-productivity (or other outcome indicator) pathway using snowball searches and references listed in reviews other than Loevinsohn et al. 2013, and (c) repeating the characterization and quality assessment exercise for the IA studies that qualified.

Using criteria slightly less restrictive than Loevinsohn et al., the authors identified 34 of the 214 studies that reported a relationship between technology and agricultural productivity: 20 of the 23 with yield data showed a positive relationship between use of technology and yield, three were inconclusive; 22 of 27 examining income showed a positive relationship between technology use and income, the others were inconclusive. They found 20 additional studies published since 2013. Nine of the ten reporting yield found a positive relationship and all 14 reporting income found a positive relationship. All of the additional studies used modern analytical procedures that attempt to recognize and account for the fact that adopting farmers may differ in many ways from non-adopters, enhancing our confidence in the robustness of the findings.

FINAL REPORT SUBMITTED - TO BE REVIEWED AND PUBLISHED

3. Trends in investments in CGIAR research (Sarah Elven)

This exercise was, in part, motivated by the lack of ready-to-access data on historical investments in the CGIAR for specific areas of research (as found during the Merrey irrigation review – activity 3.3). A consultant (Sarah Elven) reviewed annual reports and financial statements to carefully construct estimates of research investments by output/area of research/region since the CGIAR inception. Such data can feed into returns to research investment analyses as well as help identify trends in investment over a period of time.

Progress: Between August 2015 and January 2016, a database of investment (between 1972 and 2013) by five strategic areas as identified by the CGIAR, investments disaggregated by commodities (between 1972 and 2006), and investment by Center (1970 to 2015), and investment by region (between 1986 and 2014) was constructed. Report summarizing results and analyzing some of the larger trends was completed over January and February 2016. A more thorough classification by research areas – beyond increasing productivity, saving biodiversity, protecting the environment, improving policy, strengthening NARS, or commodities (cereals, legumes, roots and tubers, bananas, livestock, trees, fish, water) will involve significant effort and time (sifting through documents that may not necessarily be readily available online). Hence, SPIA will review the report and decide on a direction of this work in the coming months.

FINAL REPORT SUBMITTED (NOT PUBLISHED)

Page 34 of 41

The CGIAR will benefit from a structured attempt to support the existing capacity and some emerging collaborations on ex post impact assessment. Information-sharing and regular interaction are important in enabling the kinds of dialogue that can raise standards of impact assessment in the CGIAR, as well as ensuring that individuals have the skills that they need to be successful in their work. Activities towards this objective include a small grants program (activity 4.1); a targeted program of capacity-building using competitive calls for collaborations with advanced research institutes / universities (activity 4.2); conferences and workshops on impact assessment (activity 4.3); support for independently reviewing and publishing quality ratings of impact assessment studies carried out by CRPs and Centers (activity 4.4); maintenance and enhancement of the impact website (http://impact.cgiar.org) (activity 4.5).

Activity 4.1. Small grants

In 2013, four projects were funded through the small grants program (a total of US$30K) that has been discontinued since 2014 owing to the heavy administrative burden for small amounts of money. We are currently evaluating the utility of this mechanism by following up with small grant recipients as an input to our decision-making for the future. The four projects were:

1. IWMI, electricity and water pump policy in India: evaluation to assess impact of policy change 2. ILRI, pastoral value chains in Senegal: MSc Fellow to develop social sustainability/environmental

sustainability indicators 3. CIMMYT, agri. technology package in Malawi and Ethiopia: applying endogenous regression switching

model to a panel dataset 4. Bioversity, Home Gardens evaluation in Nepal: new approaches to measurement and evaluation of

gender impacts

Activity 4.2. Strengthening IA capacity in the CGIAR through new partnerships

In an effort to enhance the capacity within the CGIAR, at both Center and CRP levels, to conduct highly credible ex post impact assessments across a range of research areas, SPIA issued a call in August 2013 for concept notes from any organization or individual interested in undertaking an activity or a set of activities aimed at building capacity within the CGIAR for conducting ex-post impact assessment of agricultural research. Eleven concept notes were received of which five were shortlisted and invited to submit full proposals. SPIA reviewed and ranked the full proposals, and recommended to (and approved by) the PSC, one for funding initially and a second after re-considering and re-allocating the activity budgets next year. The top ranked proposal was submitted by Virginia Tech (involving CIP and CIFOR) and the second from ICRISAT involving the University of Illinois.

1. Virginia Tech with CIFOR and CIP

Virginia Tech (George W. Norton, Bradford F. Mills, Catherine Larochelle, Jeff Alwang) has been working with CIP (Guy Hareau, Willy Pradel) and CIFOR (Daniel Suryadarma; Herry Purnomo) since early 2014 to strengthen ex post IA activities at those respective Centers, focusing on these specific areas:

Classification of CIFOR and CIP research according to how IA can be done

Potential IA methods and approaches for Center research themes

CRP-specific impact pathways and theories of change, measuring outcomes and impacts

Impact-related data collection and archiving methods

Data collection protocols and management systems to meet IA needs; potential IA approaches

Conduct two pilot IAs in each Center, jointly with IA officers and scientists

Conduct learning workshops for project participants and other audiences

Progress: Workshops were held at each institution (CIP in May 2014; CIFOR in July 2014) to provide an overview of the IA strengthening project, present an overview of IA methods and their use, create broad understanding of data needs for assessment of impacts of the Center portfolio, and engage in consultations about specific

OBJECTIVE 4: Support the development of communities of practice (CAPACITY-BUILDING)

Page 35 of 41

impact assessment needs. Meetings were held with senior management and scientists to gain understanding of IA needs. Findings from these activities are being used to classify research according to IA needs and methods. Center staff were engaged in identifying the most appropriate themes for the pilot IAs.

For CIP, the two pilot IAs are: (a) an evaluation of impacts of Cooperation 88, an important and highly successful germplasm variety released and widely disseminated in China; and, (b) an assessment of the impacts of the CIP gene bank. For CIFOR, the two pilots are: (a) an assessment of the impact of the furniture value chain (a project conducted in Jepara, Indonesia to better articulate small-scale artisanal furniture producers to higher-value markets); and, (b) an assessment impacts of CIFOR research on carbon sequestration in peat bogs and mangroves. Virginia Tech has been working with CIP and CIFOR on the pilot impact assessments. While the initial plan was to hold a final synthesis workshop in November 2015 and submit a formal report by March 2016, Virginia Tech requested a 10 month no-cost extension. SPIA proposed a revised timeline for reports and payments and the contract was amended with a final report now due by November 2016.

UPDATE EXPECTED END OF JULY 2016

2. ICRISAT and University of Illinois

This project aims to enhance impact assessment capacity of CGIAR Centers through a workshop, and by implementing selected case study analyses addressing specific methodological issues using long-term panel data. More specifically, the activities proposed in this project are expected to:

raise the standards of impact assessment approaches and protocols, and promote their application by social and bio-physical scientists at ICRISAT and other CGIAR Centers.

provide opportunity to pre or post-doctoral fellows to conduct impact assessments through working on selected case studies under joint supervision from ICRISAT and collaborating partners.

establish a community of interest for agricultural scientists to engage in discussion and collaboration on impact assessment activities, and help establish operating procedures and create an environment for consistent, credible impact evaluation at ICRISAT and other CGIAR consortium members.

Progress: In September 2015, a multi-center workshop on advanced methods in impact assessment was held in Nairobi (hosted by ICRAF). The objective of the workshop was to raise the standards of impact assessment approaches and promote their application by social and bio-physical scientists at the CGIAR Centers. The agenda was based around the range of econometric tools available for impact evaluation. After the workshop, University of Illinois researchers and SPIA Secretariat met briefly to reflect on the workshop. Judging by participation rates, the workshop was a success: while University of Illinois/ICRISAT were hoping to attract 25 participants, they received 52 applications (including from NARS partners), and accepted 38 applicants to attend the workshop representing 10 CGIAR Centers. At the request of participants who strongly felt the need for an online discussion space, a board was launched in April 2016 (and is moderated by Univ. of Illinois). Despite some material posted by Univ. of Illinois and SPIA, there has not been any CGIAR activity on the board till July 2016.

Key milestones achieved to-date in this project include:

1. CGIAR workshop on Impact Assessment

Held in Nairobi Sept 21-26 2015 for 38 attendees from 10 CGIAR centers 2. Develop Common Framework for Impact Assessment

Developed exercises in workshop to foster inter-center discussion and collaboration

Established online discussion board http://impact.cgiar.org/boards/

Host symposia at multiple conferences to discuss issues of identification and impact assessment 3. Strengthen the capacity of CGIAR scientists and scholars

Awarded three mini-grants to CGIAR scientists

Made field visits to support implementation of impact evaluations 4. Complete two case studies based on ICRISAT data

One case study using VDSA/VLS data completed and currently under review

Second case study using Zimbabwe CA data is in final stages of revision before being submitted to a journal

Page 36 of 41

Activity 4.3: Biennial conference on ex post impact assessment results and methods

On July 26th 2014, SPIA organized a workshop entitled “Measuring poverty impacts of agricultural research” as a pre-conference workshop at the American Applied Economics meetings in Minneapolis. Seventy participants, approximately one third from the CGIAR and two thirds from academia, participated in full day of presentations and extended discussion on the potential and limits of the following families of studies: micro-level econometric studies; model-based approaches; randomized-control trials; meso, macro and cross-country studies. The invited speakers were of the highest caliber (e.g. Michael Carter, Tavneet Suri, Will Martin, Marc Bellemare, John Antle, and Julian Alston). The day concluded with a panel discussion on “Reducing rural poverty as a System-Level Outcome for the CGIAR”. The workshop was very successful in fostering open discussion across these specialists within the agricultural economics discipline (Poverty impacts workshop, Minneapolis July 2014 report).

Following the workshop, SPIA (Doug Gollin, with research assistance from Lilli Probst) took on the task of summarizing these findings and reviewing the related literature in a paper for a non-specialist audience. An outline of this paper (Poverty impacts paper, Gollin Sept 2014) was presented to the meeting of the CGIAR Independent Science and Partnership Council at the University of Copenhagen in September 2014.

Immediately prior to the poverty impacts workshop, on July 25 2014 in Minneapolis, SPIA hosted a meeting of the Impact Assessment Focal Points (IAFPs) from across the CGIAR Centers / CRPs. This was a full day of presentations from each of the 15 focal points, as well as from SPIA on progress with the SIAC program. This was a valuable opportunity for Center scientists to exchange information on their current impact assessment projects and to benefit from advice from a number of high-quality resource people that SPIA had arranged to attend to provide feedback (Julian Alston, Jeff Alwang, George Norton, Greg Traxler, Bob Herdt, JV Meenakshi). Proceedings and presentations from the IAFP workshop can be found on the Events page on SPIA website.

No IAFP meetings were held in 2015, but SPIA participated in IEA’s Evaluation Community of Practice (ECoP) meeting in Rome (November 2015) – there is a fair amount of overlap between ECoP members and IAFPs.

SPIA is holding an IAFP meeting in Boston on 29 July 2016, immediately prior to the annual AAEA. There will also be a mid-term workshop for SIAC Activity 3.1 which IAFPs will also be able to attend. The objective of the IAFP meeting is to give CRPs an opportunity to present their impact assessment and results based management (RBM) plans for Phase-II of CRPs, and provide them an update on SIAC Phase-I activities, including the IEA evaluation as well as plans for a potential SIAC Phase-II. The discussions at the meeting are expected to feed into the Phase-II planning. Information about the workshop is available from the SPIA website.

Finally, SPIA is scoping options for a 4-day impact assessment conference to cover all of SIAC outputs and solicited papers from external researchers, for May or June 2017.

Activity 4.4. Enhancing quality and rigor: Introducing a Star Rating System for IA studies

SPIA launched the online external review and rating system in early April 2016 as a key mechanism for ensuring high quality assessments of impacts by the CGIAR. The system differs from journal reviews (that focus on methodological approaches and innovative research) in that it also focuses on criteria such as scale and link to agricultural research outputs, responding to donor needs. It is not intended to replace journal publications, but is a systematic way of identifying and showcasing (thereby setting an example of) high-quality work to the CGIAR and donors. The idea was presented at the Impact Assessment Focal Point (IAFP) meeting in Minneapolis in July 2014 and received with enthusiasm. This document (Criteria for quality rating) outlines the vision for this system: a significant change in the management process outlined in the draft being that the SPIA Chair will function as the Chief Editor and supported in that role by SPIA members (as Associate Editors). For each manuscript submitted for rating, the Chief Editor or one of the Associate Editors will identify external reviewers and manage the review process with support of SPIA Secretariat staff.

The process started in September 2014 with a contract with the vendor (Allen Press), and a testable version of the review system online completed in December 2014. The website went live in February 2015. Discussions on approach to quality rating and criteria continued through 2015, but the process of inviting submissions was delayed. Since April 2016, SPIA has not received queries about the system or received submissions. The reasons

Page 37 of 41

for disinterest will be probed at the IAFP meeting in July 2016. SPIA is considering random audits of impact claims and the associated studies motivating them, as part of a second phase to the SIAC program.

Activity 4.5. CGIAR Impact Website

The CGIAR impact website was re-launched in May 2014. The entire front-end has been redeveloped to enhance users’ ability to find the information they need, and the visual identity has been brought up to date. New features such as an impact blog, a latest news section, and a dedicated section for the community of practitioners of IA, all add useful resources to help increase awareness of IA activity in the CGIAR.

Since then, the Impact website continues to get maintained and updated with information on new calls for proposals and blog entries. In 2016, a private discussion board was setup to facilitate interactions between impact assessment scientists at CGIAR Centers/CRPs in relation to capacity building activity 4.2.

A SIAC M & E related activity: Survey of donor demands for impact assessment

In 2014, the Secretariat conducted a ‘Donor Demand for ex post IA’ study as a follow-up to an earlier survey carried out in 2004-05. This recent survey sought to better understand:

A. How/whether use of epIAs has changed/evolved over time; current perceptions of CGIAR epIAs B. Are/how some of these changes in use/perceptions relate to actions (e.g. publication of impact briefs)

taken by SPIA on the issues identified in 2005 C. Additional actions required of SPIA, to feed into SPIA strategy. And use this opportunity to solicit

feedback from donors on quality ratings of CGIAR epIAs, methodology and parameters proposed

A brief draft report (Donor demands summary) - qualitative with data summaries, rather than quantitative given the small number of respondents – was produced. SPIA member J V Meenakshi provided guidance and assistance in interpreting results of the survey. The report classifies the survey questions into themes and identifies the key emergent messages. The initial plan was to conduct follow-up interviews via skype/telephone with individual donors to help clarify ambiguities (similar to the process used for 2004-05 study). However, after discussing with Rodney Cooke, whom SPIA contacted to lead this component, SPIA decided against a follow-up at this point given the other priority concerns in the System (SRF revision, second call for CRPs, etc.).

Discussion during the Mid-term Review and SPIA strategy session around the results led to some key messages emerging and these are highlighted below:

1. Accountability, i.e. demonstrating whether research outputs contribute to development goals, appears to be the most important purpose for doing epIA. Learning and critical self-analysis are less important objectives, with the exception of donors like Gates who rate this the highest.

2. While a few donors emphasize rigor and precision in impact estimates, and careful construction of a counterfactual, many others state that they are not looking for methodological perfection or purity. Given the long lags between completion of the research and ultimate impacts, they recognize the difficulty in constructing an appropriate counterfactual, and are of the view that ‘the perfect should not get in the way of the good’. At the same time, many donors prefer SLO level/large scale impact and evidence. As one donor stated, this is persuasive communication information. This is consistent with the donors’ view about the important accountability imperative of epIAs: at scale, such evidence enables donors to establish that these investments are worthwhile.

3. There is higher demand for epIAs, even with its associated higher costs per study, than for adoption/uptake studies or for efficacy studies. Science quality/output assessments scores far lower than the other 3 study types. The Gates allocation was noteworthy in that adoption/uptake studies received the highest points allocation (60 out of 100 points) followed by efficacy studies (30) and science quality/output assessments (10). epIAs received 0 points.

4. There is a slight preference for

epIAs of recent investments, rather than earlier ones

a suite of epIAs focusing on a small set of indicators, rather than one in-depth study with a rich, broader set of indicators

a number of cross-sectional studies (as opposed to a single panel) that may not be in-depth, but representative of full adoption domain.

Page 38 of 41

ANNEX 1 - List of SIAC funded studies and timetable (sorted in cronological order by date of the call and grouped, in colour scales, by the SIAC objective)

No. Grantee / Sub-grantee - Partner Donor SIAC

activity

Project

start

Project

end

Project

extensionTotal cost

Paid to

dateStatus

% funds

disbursed

Progress

towards

completion

Comment

1 Michigan State University BMGF OBJ.1/2 01.03.13 31.12.15 31.10.16 $2,968,897 $2,968,897 ongoing 100% 85%

2 Michigan State University W1 OBJ.1/2 01.02.15 31.05.16 31.10.16 $649,550 $519,640 ongoing 80% 85%

3 VIRG TECH / CIFOR ISPC OBJ. 4.2 01.01.14 31.12.15 30.11.16 $286,125 $228,022 ongoing 80% 80%

4 ICRISAT / Univ Illinois BMGF OBJ. 4.2 01.10.14 30.09.16 31.12.16 $237,070 $189,656 ongoing 80% 75%

5 CIMMYT ISPC OBJ.3.2 01.01.14 01.03.14 - $19,185 $19,185 completed 100% 100%

6 TUFTS ISPC OBJ.3.2 01.03.14 01.03.16 28.02.16 $49,671 $39,671 completed 80% 100% final payment not made, as per LOA amendment

7 Paris School of Economics ISPC OBJ.3.2 01.01.14 01.03.16 - $76,300 $76,300 completed 100% 100%

8Innovations for Poverty Action (IPA) / MIT-

JPAL, MIT SloanW1 OBJ.3.0 15.07.14 31.12.15 31.12.16 $99,876 $54,932 ongoing 55% 75%

9 ILRI / Emory Univ W1 OBJ.3.0 01.07.14 30.06.16 - $100,000 $30,000 ongoing 30% project on hold until satisfactory progress is made

10 CIMMYT / Univ Goettingen, Univ Geneva W1 OBJ.3.0 01.10.14 30.06.16 - $150,000 $120,000 ongoing 80% 99%

11 CIAT / Virg Tech W1 OBJ.3.0 15.11.14 14.11.16 - $200,000 $160,000 ongoing 80% 80%

12 COLUMBIA / George Washington Univ W1 OBJ.3.0 01.03.15 31.12.16 - $130,119 $104,096 ongoing 80% 75%

13Qué Funciona para el Desarrollo A.C /

Berkeley, J-PAL, WorldBankISPC OBJ.3.2 01.12.14 31.10.16 - $357,608 $286,086 ongoing 80% 85%

14 BERKELEY / Tufts, IRRI BMGF OBJ.3.2 01.05.15 31.12.16 31.05.17 $235,176 $70,553 ongoing 30% 50%

15 YALE / J-PAL, ICIMOD, Dep. of Agric. Nepal BMGF OBJ.3.2 01.05.15 31.12.16 - $268,609 $214,887 ongoing 80% 70%

16 ICARDA / Virg Tech BMGF OBJ.3.1 01.07.15 31.12.16 - $135,468 $108,374 ongoing 80% 70%

17 CIP / Virg Tech ISPC OBJ.3.1 01.06.15 30.11.16 - $200,000 $160,000 ongoing 80% 80%

18 IITA W1/ISPC OBJ.3.1 01.06.15 31.05.17 - $199,978 $159,982 ongoing 80% 55%

19 UC SAN DIEGO / Ram Fishman BMGF OBJ.3.1 01.07.15 31.12.16 - $181,963 $145,571 ongoing 80% 70%

20 IFPRI / WorldBank W1 OBJ.3.1 01.08.15 31.12.16 - $199,967 $159,974 ongoing 80% 70%

21WORLDFISH / Central Luzon Univ Philippines,

Phiippines Bureau of FisheriesBMGF OBJ.3.1 01.08.15 31.12.16 - $134,562 $107,650 ongoing 80% 70%

22 TANGO International / IFPRI, Westat BMGF OBJ.3.1 15.10.15 31.08.16 - $199,357 $159,486 ongoing 80% 95%

23 CIAT / Michigan State Univ, Univ Andes W1 /ISPC OBJ.3.3 15.02.16 30.06.17 - $200,000 $60,000 ongoing 30% 30%

24 VIRG TECH / CIFOR ISPC OBJ.3.3 01.03.16 30.06.17 - $140,053 $28,010 ongoing 20% 25%

25 IRRI / North Carolina State Univ / PhilRice W1 OBJ.3.3 15.02.16 30.06.17 - $199,991 $59,997 ongoing 30% 30%

26 ICRAF / Univ Illinois W1 OBJ.3.3 01.02.16 30.06.17 - $199,640 $59,892 ongoing 30% 30%

27 University of Minnesota ISPC OBJ.3.5 15.12.15 15.11.16 - $200,000 $160,000 ongoing 80% 80%

28Institute for Financial Management and

Research / CIMMYTBMGF OBJ.2.2 15.05.16 30.06.17 - $199,999 $60,000 ongoing 30% 10%

29Norwegian University of Life Sciences /

Monica FisherBMGF OBJ.2.2 15.05.16 30.06.17 - $111,099 $33,330 ongoing 30% 10%

30 CIMMYT / ICRISAT BMGF OBJ.2.2 15.05.16 30.05.17 - $74,913 $22,474 ongoing 30% 10%

31 Nong Lam University Vietnam / UC Santa Cruz BMGF OBJ.2.2 15.07.16 30.06.17 - $100,970 $0 to begin 0% 0%

32 IFPRI / IITA / GeoPoll BMGF OBJ.2.2 15.07.16 30.06.17 - $199,999 $0 to begin 0% 0%

33 ICRAF / Penn State, Univ Zambia W1 OBJ.2.2 01.06.16 30.06.17 - $190,457 $57,137 ongoing 30% 5%

34 CIMMYT W1 OBJ.2.2 01.06.16 30.06.17 - $74,888 $22,466 ongoing 30% 5%

35 ICRISAT / Univ Illinois / CIMMYT W1 OBJ.2.2 01.07.16 30.06.17 - $149,890 $44,967 ongoing 30% 1%

36 FAO EPIC - informal agreement W1 OBJ.2.2 15.07.16 30.06.17 - $129,400 $0 to begin 0% 0% consultant to be paid in 11 monthly instalments

37 FAO EPIC - informal agreement W1 OBJ.2.2 15.07.16 30.06.17 - $10,000 $0 to begin 0% 0% part of the ICRAF 2.2 LOA

Page 39 of 41

ANNEX 2 SIAC Objective 2.1. Organize the collection of varietal adoption data – Survey of Participants of the Expert

Elicitation Workshops, July 2016 This survey is being conducted by Michigan State University as the lead on objective 2.1 of the Strengthening Impact Assessment in CGIAR (SIAC) Project. As part of this objective several Expert Elicitation workshops were conducted jointly by the CGIAR centers (e.g., CIAT, CIMMYT, CIP, ICRISAT and IRRI) and national partners to collect varietal adoption data in several crop-country-combinations (CCCs) across South, Southeast and East Asia in 2014-2016. The purpose of this survey is to assess the approach used to collect crop varietal adoption data as part of this SIAC activity, and to get some additional information on the status of the seed system for specific crops. You are asked to respond to this survey because you participated as an expert in one of the Workshops organized in the past 2 years. You are free to voluntarily choose to participate in this survey, refuse to answer certain questions, or stop participating at any time without any loss or harm to you. If you choose to participate, your feedback will help us improved this methodology of collecting adoption data in future. Your responses will be kept completely confidential to the maximum extent allowable by law. Your responses will be summed together with those from other workshop participants across more than 100 CCCs. Only general averages from the analysis will be reported. For any questions about the study, contact Dr. Mywish Maredia from MSU (1-517-353-6602 or maredia@msu.edu). By continuing with this survey, you indicate your voluntary consent to participate in this study.

Instructions on submitting the survey: You can complete this Word file, save it on your computer, and email the file to: maredia@msu.edu. Alternatively, you can complete this survey online by following this link: https://form.jotform.com/61745683053156 Your time to complete this survey is very much appreciated. A. Information about the Expert Elicitation Workshop in which you participated For which ‘crop-country-combination’ workshop did you participate?

Write your responses in this column

A1. Crop

A2. Country

A3. If in India or China, specify the State / Province

A4. Year in which this workshop was held

A5. Name of the city where it was held

B. Your background and expertise

B1. Gender |___| 1=Male 2=Female

B2. Highest level of education completed: |____| 1=High school or less 2=Bachelors degree 3=Masters degree 4=Ph.D.

B3. Type of organization you belong (Mark an ‘X’ next

to the response) 1=Government (e.g., Ministry) 2=Public research organization 3=Private sector 4=NGO 5=Independent Farmer 6=Extension 9=Other (specify)

B4. Main area of your expertise (Mark an ‘X’ next to the response) 1=Plant breeding 2=Other plant sciences (agronomy, pathology, entomology, etc.) 3=Extension specialist 4=Seed system specialist 5=Socio-economist 6=Marketing 7=Farming 9=Other (specify)

Page 40 of 41

B5. Number of years associated with this /organization:

B6. Number of years of experience you have with the crop mentioned in A1:

B7. Have you ever participated in a similar type of ‘expert elicitation’ process for the purpose of:

a. Estimating varietal adoption? (Yes/no) |___| b. Estimating the adoption of other types of agricultural technologies? (yes/no) |___| c. If Yes, please specify the type of technology: ____________________________________

B8. Have you ever participated in a Delphi method of consensus building?

Yes____ / no_____ / don’t know what is Delphi method_____

C. Assessment of the EE workshop C1. Please share your experience of participating in the above mentioned expert elicitation workshop in terms of:

a. Strengths

b. Weaknesses

c. Any suggestions you may have to improve this methodology

C2. On a scale of 0 to 3 how satisfied were you with the process and methodology of eliciting your expert opinion during this workshop? (select one option by putting an ‘X’) ___0 =completely dissatisfied ___1=somewhat dissatisfied ___2=somewhat satisfied ___3=completely satisfied. C3. On a scale of 0 to 3 how confident were you with the adoption estimates at the end of the workshop? (select one option by putting an ‘X’) ___0 =No confidence at all ___1=somewhat confident ___2=moderately confident ___3=completely confident C4. In your opinion, how important were the following characteristics/background of participants in the overall workshop discussion and influencing the results (Scale: 0=Not important and 3=Very important) (Select your response by putting a ‘X’ for each item a to f listed below)

0=Not important

1=somewhat important

2=moderately important

3=very important

a. Age

b. Professional position

c. Type or area of expertise

d. Education

e. Religion

f. Gender

C5. Approximately, how much time did you spend on (indicate number of hours):

a. Any preparatory work prior to attending this workshop: ________ b. Time spent traveling to/from your place of residence to the workshop venue: _______ c. Time spent at the workshop: ________

C5. Would you participate in such expert elicitation workshop, if held regularly in future? (Yes/no) D. Experience with the group discussions D1. Did the Workshop include forming sub-groups and conducting discussions and estimations at the sub-group level (yes/no) _________

Page 41 of 41

D2. How did you come up with the results in the group discussions? (select one option) _______ 1=By taking averages of individual results (if this option, respond D2a, and then go the section E) 2=By ‘consensus’ (if this option, respond D2b, and then go section E)

D2a. In case the group estimates did not add up to 100%, how did you proceed? (select one option) ___

1=Adding another minor’ variety to the list 2=Allocating remaining percentages to “other varieties” or local varieties 3=Allocating remaining percentages to major variety 4=Evenly allocating across all varieties

D2b. In case you came up with group estimates by consensus, how was consensus reached? _______

1=Democratic process (more than 50% in favor of a certain estimate/variety) 2=Discussion by convincing the other(s) (by giving examples and evidence) 3=Only certain subgroup members’ opinion counted/ were considered important

E. Assessment of the seed system

E1. How active is the public, private and NGO sector in the following aspects of the seed system for the crop-country (or state/province) combination noted in A1-A3. Mark ‘X’ in appropriate column

0=Not at all active

1=Somewhat active

2=Moderately active

3=Very active

Public sector

1. Varietal development

2. Seed multiplication

3. Seed dissemination

Private sector

4. Varietal development

5. Seed multiplication

6. Seed dissemination

NGO or donor funded programs

7. Varietal development

8. Seed multiplication

9. Seed dissemination

E2. In your opinion, what percentage of farmers use harvest from the previous season as seed or planting material for the production of this crop in this CCC? _____

E3. In your opinion, what percentage of total cropped area is planted every year with new / fresh seed obtained as ‘seed’ from the market for this CCC? _______

E4. In your opinion how would you rate the overall development of the seed system for this CCC? (select one option by marking an ‘X’)

___0 =low level of development ___1=somewhat developed

___2=moderately developed ___3=Highly developed

THANK YOU FOR YOUR TIME