Challenges and Opportunities in Evaluating the Impact of Innovation

Platforms

IDIAR Course 2016, Southern Sun Mayfair Hotel, Nairobi, Kenya Karl Hughes, Head of Monitoring, Evaluation and Impact Assessment

World Agroforestry Centre (ICRAF)

IMPACT

IMPACT• What do we mean by impact in

the IP context?

• Most simple definition: The difference the IP(s) made—whether expected of unexpected, positive or negative.

IMPACT(continued)

• If IP facilitation is the intervention, then some (e.g. economists) would say that impact refers to the causal effects of this intervention—whether shorter, medium, or longer term.

IMPACT(continued)

• Others distinguish between outcomes and impacts, e.g. IDRC model

Sphere of interest

Sphere of influence

Sphere of control

Outcomes = changes in behavior/practice

Outputs

Impact = changes in conditions, e.g. health, poverty,

food security

In IP facilitation, what type of impacts do we care most about?

• A stronger, more viable value chain or better managed watershed?

• Better food security and incomes for producers and/or other actors along the agricultural value chain?

Approaches for evaluating IP IMPACT

1. Before after analysis

2. With and without (counter factual) analysis

3. Theory-based (mechanism-based) approaches

4. Towards an integrated approachIn science, causal inference is strongest with both a rigorous estimation of the counterfactual and evidence of the mechanism(s) responsible.

Time

Development Outcome (e.g., producer income)

Pre-IP (T0) Post-IP (T1)

Change(T1 – T0)

Before and after analysis• Used alone not so

useful, particularly in relation to Sphere of Interest level indicators

With and without (counterfactual) analysis

TimePre-IP (T0) Post-IP (T1)

Net impact

Change with intervention

Change that wouldhave happened without intervention

Development Outcome (e.g., producer income)

• Use similar control or comparison population to estimate what would have happened if there were no intervention, e.g. no IP

How using control/comparison group works to help estimate average impact (for ‘large n’ interventions):Variable (Baseline) Intervention Group Control/Comparison Group

Female Headed 10% 9%

HH Size 4.3 4.4

Adult with primary 67% 65%

Adult with secondary 14% 15%

Land Holding 0.89 hectare 0.91 hectare

Below Poverty Line 41% 39%

Asset Index 0.49 0.51

Cattle 3.2 3.5

Shoats 12.9 13.3

NGO exposure 85% 83%

Relevant unobservables, e.g. motivation Similar, esp. in RCTs Similar, esp. in RCTs

An ambitious example in the IP context• Done in the context of FARA’s Sub-Saharan

Africa Challenge Programme (SSACP)• Data from 24 wards evenly split across

Uganda, DRC & Rwanda (Lake Kivu Region)• Random assignment at ward, rather than

village-level, due to potential of spill overs• Sub-sample of ‘clean villages’ in IP wards

targeted, with control wards having both ‘clean’ and ‘unclean’ villages

• After two years, 17% average reduction in poverty in IP villages, but how this happened is unclear—IP villages seem more likely to have customized innovations

Some key challenges with the counterfactual approach in the IP context

1. Participation uncertainty

2. Fluidity of the IP intervention

3. Isolating the IP effect

4. External validity, i.e. will the result be similar elsewhere?

5. Practical issues with controls/ comparison sites in the single IP context

1. Participation uncertainty• Value chain targeted & core IP

facilitators/leaders identified

• How sure can we be about who will actually participate as the initiative evolves?

• Initiative likely to evolve in unforeseen ways, e.g. dropping unorganized producers or moving into new promising areas

• Unlikely to make commercial sense to turn actors away

2. Fluidity of the IP Intervention

• Interest in IPs in AR4D due to limitations of the “pipeline” model

Improve crop variety or practice

Extension or seed system Farmers Adoption &

impact

Evaluation, Learning & Feedback Loops

• The translation of research into impact is, unfortunately, rarely so simple and linear…

• The purpose of IPs is—almost by definition—to experiment (figure out) how to overcome common challenges for the benefit of all actors

• Developmental Evaluation becomes highly relevant in the IP context

“Developmental evaluation is an approach where evaluative thinking, logic, and approaches – as well as whatever data happen to be available –are used for the purposes of continuously developing a programme or specific intervention.”

• So even if we happen to evidence overall impact using the counterfactual approach, uncertainty about actually led to it

3. Isolating the IP effect• Not only may it be difficult to determine cause

of observed impact—it may have nothing to do with the nature of the IP concept

• e.g. an NGO could be involved that delivers inputs to the participating producers

• e.g. the IP does something that any other organization could have done

• Need to get at the mechanisms to really understand the IP effect

4. External validity (generalizability)• We typically invest in impact

evaluations to generate learning relevant for policy and/or practice

• However, IPs typically deal with such unique and context specific issues, so conclusions are likely not directly transferable

• Would the cost and effort in rigorous quantitative impact evaluation for mainly accountability purposes be worth it?

5. Practical issues with control/comparison sites in the single IP context

Option Issues1. Within IP catchment area

(e.g. district) randomize or purposively select areas (e.g. villages) for IP to target, while leaving others as controls

• Likely create some commercial or programmatic inefficiencies, e.g. transportation—need to bypass village right next to one you are working with

• Likely to be significant non-compliance and spill-overs, e.g. producers will got to intervention villages to sell their produce or buyers will ignore

2. Use producers and/orother value chain actors in one or more other settings (e.g. districts) for comparison purposes

• Many of the practical issues associated with Option 1 may be overcome

• Collecting baseline/endline data on both intervention & comparison actors and comparing relative changes over time not a bad strategy

• But high chance that the groups will be subjected to different external trends overtime, e.g. weather or NGO programmes, that affect outcomes of interest

Approaches for evaluating IP IMPACT

1. Before after analysis (alone)

2. With and without (counter-factual) analysis

3. Theory-based (mechanism-based) approaches

4. Towards an integrated approach

Theory or mechanism based approaches • Develop solid ToC, including

specifying desired behavior key VC actors and other stakeholders

• Great stuff—helps to support adaptive mgt., etc. to better facilitate the IP and/or deliver better programme

• Combine with baseline and endline snapshot of the value chain (or other entity of interest)

• For larger projects, consider commissioning an evaluator with expertise in contribution analysis and/or process tracing, to drill down on o the extent the project was responsible for changes in the evaluation of the agricultural value chain, etc.

• All the monitoring data collected will prove very useful

But challenges again if focus is on specific actors in VC:

• Increase in VC commodity net income not the same as overall household income

• Even if we switch to looking at HH income, other factors may have been responsible for the changes

• Possible shifts in production/ income streams that may exacerbate risk

Approaches for evaluating IP IMPACT

1. Before after analysis (alone)

2. With and without (counter factual) analysis

3. Theory-based (mechanism-based) approaches

4. Towards an integrated approach

Almost there

Towards an integrated approach

1. Baseline/Enline “snapshots” of status of VC and perhaps income earned by actors for targeted VC products

2. Develop and continuously review and adapt ToC complemented with Outcome Mapping and Developmental Evaluation approaches, perhaps together with an external evaluation

3. Bring in researcher(s), if necessary, to support the testing of innovations (Planned Comparisons) to overcome key challenges—e.g. farmer field trials, efficacy studies, etc.

4. Focus efforts on assessing the extent to which these evidenced innovations have been taken up and scaled, including numbers reached

Group work • Get into small groups

• Identify existing IP (or VC/NRM context in which an IP will be facilitated

• What key challenges is the existing or future IP working to overcome?

• Identify at least one challenge for which there is significant uncertainty on which innovation would be most appropriate (cost-effective) to help overcome it

• Identify at least one innovative—in addition to the status quo—that appears promising to address the challenge

• Use template provided to outline a planned comparison to test the effectiveness of this innovation(s)

• If time, discuss how you would promote the scaling up and out of the innovation(s) proven to be most effective and evidence the extent to which such scaling up and out has taken place