1
Using multiple methods to join the dots:
Assessing Rural Transformations
Fiona Remnant, IDS
Oct 2015
2
The ChallengeHow can we really tell what impact a project
has had on beneficiaries?
Trade-offs between credibility and cost-effectiveness in impact evaluation… particularly in complex environments.
– Large surveys – often inflexible, expensive and slow– RCTs – limited applicability; honeymoon over; – Qualitative approaches – but what is credible whilst
still cost-effective?
3
GoalsAction research to design and pilot a ‘good enough’ credible, rapid and cost-effective qualitative impact assessment protocol (QUIP) which in the context of complex rural transformations can: • confirm project theories of change, and • explore unanticipated drivers of change• complement other M&E activities by
addressing the impact attribution gap
4
Development Agencies
(X)
Diverse,
complex
risky
contexts
(Z)
Improved livelihoods &
wellbeing (Y)
QUIP:Respondents’
accounts
Development Agencies
(X)
Improved livelihoods &
wellbeing (Y)
Livelihoods & wellbeing?
The attribution problem
5
Case study projects (X), indicators (Y) and confounding factors (Z)
X Y Z
Project 1. Groundnut value chain (Central Malawi)Project 2. Climate change resilient livelihoods (Northern Malawi)Project 3. Malt barley value chain (Southern Ethiopia)Project 4. Climate change resilient livelihoods (Northern Ethiopia)
• Food production • Cash income • Food consumption• Cash spending• Quality of
relationships • Net asset
accumulation• Overall wellbeing
• Weather • Climate change• Crop pests and
diseases• Livestock mortality• Activities of other
organisations • Market conditions• Demographic
changes• Health shocks
6
Ten features of the QUIP1. Data collection by independent field researchers
without knowledge of the project (blinding).2. Purposive then random household sampling based
on quantitative project monitoring.3. Semi-structured household interviews plus focus
groups.4. Data collection instruments structured by wellbeing
domain, with alternating open then closed question sections.
5. Data entry using pre-formatted Excel sheets to facilitate coding and analysis.
7
… continued6. Systematic coding of impact evidence as explicit,
implicit or incidental to project actions.7. Additional flexible coding of drivers of change
(positive and negative). 8. Rapid semi-automated report generation to speed
analysis. 9. Easy to drill down from summary evidence to raw
data for auditing and learning purposes.10. Summary report, starting point for project level
debriefing between project staff and researchers.
8
Central Malawi improved groundnut seed: responses to QUIP2 closed questions
Code Gender Beneficiary?* 1. Food Production 2. Cash income
3. Purchasing power
4. Food consumption 5. Assets
6. Overall Wellbeing
PK3 Male ++ + + + + + + PK6 Female + + - = - = = = PK7 Female + + - = - - + - KM3 Female + + + + + + + +
KM10 Male + + + + + + + + KM11 Male + + + + + + + + KM12 Male + + - + + = + + PK8 Female + - - - - + - PK5 Female + - - - - = =
PK10 Female + - - - = - - KM1 Female + = - - - + - KM2 Male + + + + + + +
KM15 Female + - - - = = = PK1 Female / - - - - = - PK2 Female / - + = = + + PK4 Female / - = - - - + PK9 Female / - - - = - - KM4 Female / = - = = - = KM5 Female / = - = = + = KM6 Female / = - = - = - KM7 Female / - + - - + + KM8 Male / - - - - + - KM9 Male / - - - - + -
KM13 Female / - - - - + - KM14 Female / + = - = + =
KM16 Female / - - - - = -
* + Beneficiary to some degree; ++ Beneficiary to a larger degree; / Non-beneficiary
9
Positive change statements by domain
Bold figures indicate beneficiary households
1 Project explicit 3 Project implicit 5 Other
Food production KM2, KM3, KM10, KM11, KM12,
PK3 PK1, PK2, PK9, KM13,
FGKM2, FGPK1,
Cash income KM2, KM3, KM5, KM10, KM11, KM12,
PK3, KM12, PK1, PK2 PK5, KM7
FGKM2, FGPK1,
Purchasing power KM2, KM3, KM10, KM11, PK3 PK3, PK5,
FGKM2, FGPK1,
Food consumption KM2, KM3, KM10, KM11, PK3
FGKM2, FGPK1,
Intra-hh & villagerelationships
KM10, KM11, PK4,
FGKM2, FGKM2, FGPK1 FGPK1, FGPK2
Asset accumulation KM2, KM10, PK3,
FGKM2, FGPK1,
Links with external orgs
PK4, KM2, KM3, KM5, KM10, KM11, KM12, KM15,
KM16 PK3, KM7, KM13, KM14, KM15,
FGKM2, FGPK1, FGKM1 FGKM1, FGPK1
Overall wellbeing KM10, KM11, KM12 PK3, PK4 PK2, KM7, PK3
FGKM2, FGPK1
10
Drilling into positive statements Section C1: Food Production
1. Explicit project and linked causes (positive) Attribution
[KM10 C1] The respondent noted that the ability to produce their own food had changed for the better. He attributed this to use of livestock manure, compost and Bocash manure in the fields as taught by Self Help Africa. He also said that the adoption of hybrid seeds as recommended by Self Help Africa has contributed to this because the hybrids are high yielding. He also said that the ability to own livestock had increased too. He said he received two pigs from Self Help Africa some time back which at some point sold some and then bought cattle. On taking up new activity he cited the manure making and application as an activity he had just embarked on recently which has helped ensure that own produced food never runs out in his household. He said he had stopped total reliance on inorganic fertilizer. He said that he was not doing anything differently to the majority of the community members because they too have taken up manure application and use of hybrid seeds in their crop production.
SHA Training
SHA seed credit
11
Negative change statements by domain
Bold figures indicate beneficiary households
2 Project explicit
4 Project implicit 6 Other
Food production KM15, PK7 PK1, PK2, PK4, PK5, PK6, PK7, PK8, PK9, PK10, KM6, KM7, KM8, KM9, KM12, KM13, KM14, KM15,
FGKM1, FGPK1, FGPK2Cash income PK1, PK2, PK4, PK8, PK9, PK10, KM1, KM7, KM8, KM9, KM12, KM15,
FGKM1, FGPK2Purchasing power PK1, PK2, PK4, PK8, PK9, PK10, KM5, KM7, KM8, KM9, KM12, KM13,
FGKM1, FGPK1, FGPK2Food consumption
PK1, PK4, PK5, PK10, KM6, KM8, KM9, KM13,
FGKM1, FGPK2Intra-hh & villagerelationships
KM15, PK5, PK7, KM5,
FGPK2, FGKM1, FGKM2, FGPK2Asset accumulation
PK10, KM7, KM8
FGKM1, FGKM2, FGPK1, FGPK2Links with external orgs
PK6, PK7, KM7,FGKM1, FGPK2
KM5 FGKM1, FGKM2
Overall wellbeing PK7, KM15 PK1, PK5, PK8, PK9, PK10, KM1, KM5, KM6, KM8, KM9, KM13,KM16 FGKM1, FGKM2
12
Positive Drivers
Health Food production Income Purchasing
powerFood
consumption
Relationships (intra hh and
village)Assets Overall
wellbeing
SHA Training
KM10, PK3
KM2, KM10, KM11, PK3
FGKM2, FGPK1
KM2, KM10
FGKM2, FGPK1
KM10, KM11
FGKM2, FGPK1
KM2, KM10, KM11
FGKM2, FGPK1
KM10 PK3, KM10
FGKM2
KM10
FGKM2, FGPK1
SHA Seed Credit KM10, KM3, KM12
KM3, KM5, KM11, KM12
KM2, KM3 KM3 KM2 KM11, KM12, PK4
SHA Gender training KM10, KM11
FGKM2
Religion/ God PK5, KM11 FGPK2
Investment in livestock PK1, PK2
Investment in soy production PK2, PK9
Village savings and loans group KM13 PK3 PK3, KM7
Increased farming of wetland
PK3 PK3 PK3 FGPK1, FGKM2 FGPK1 PK2, PK3
Other income generation KM7, PK1,
PK2, PK5 PK5
13
Negative DriversHealth Food production Income Purchasing power Food consum-
ptionRelationships (intra
hh and village) Assets Overall wellbeing
Fertiliser cost (and subsequent lower yields)
PK1, PK5, PK7, PK8, PK10, KM6, KM7, KM9, KM13, KM15
KM7 PK1, PK2, KM7, KM13
PK1, PK5, KM13 KM7 KM5, PK1,
PK5, KM13
FGKM1, FGPK1, FGPK2 FGKM1, FGPK2 FGKM1,
FGPK2FGKM2 FGPK2, FGKM1 FGPK2 FGKM1
Illnesss/ deathPK8, KM1, KM4, KM12, KM15, KM16
PK7, PK9, PK10, KM8, KM12, KM13, KM16
PK8, PK10, KM1, KM8, KM9, KM12
PK8, PK9, PK10, KM8, KM12
KM6, KM8, PK10 PK10, KM8
PK8, PK10, KM1, KM6, KM8, KM16
Loss of livestock (disease, theft) PK1, KM12, KM14 KM9 KM9 KM9
Erratic rainfall PK2, PK6FGPK1 PK2
Unprofitable business (crop sales and other)
KM15FGPK1, FGPK2
FGPK1 KM9
Village savings group debt repayments
KM5 FGPK1 FGKM2FGKM1FGKM2 KM5, KM9
FGKM2
SHA seed credit debt (failure to germinate)
PK7, KM15 PK7
Domestic dis-agreements KM14 PK9 PK9
Absent husband PK4 PK4 KM5, PK4 PK4 KM5, PK5 Low tobacco prices (stopped growing as a crop)
PK1, PK2
14
Interventions & reported drivers of changeDATE Intervention KM1 KM2 KM3 KM10 KM11 KM12 KM15 PK3 PK5 PK6 PK7 PK8 PK10
Q4 2012
Training in crop husbandry practices ü ü ü
Groundnut seed distributed ü ü ü ü
Q1 2013 Training in value addition
Q4 2013
Training of farmers in seed multiplication principles and general crop husbandry practices
ü ü ü ü
Groundnut seed distributed (basic)
ü
Groundnut seed distributed (certified)
ü ü
Soya seed distributed (basic) ü
Q3 2014
4 groundnut shellers given to the cooperative
Q4 2014
Groundnut seed distributed (basic)
ü
15
Methodological discussionIssue Limitations Ways forward
Scope for relying on self-reported attribution.
Ok for evidence of contribution, but weak on magnitude of impact.
Combine with quantitative monitoring and micro-simulation.
Addressing confirmation bias through blinding of researchers.
Possibly a one-shot opp. for field researchers & difficult for NGOs to replicate.Analysts can’t be blinded as they must also know the theory of change.
No substitute for professional integrity of field researchers. Analysis can be transparent and replicable.
Improve generalizability of findings through opportunistic (re)sampling given that the power of each additional interview is additive (cf. “problem driven iterative adaptation”)
Lack of sufficiently systematic monitoring data to inform selection.Opportunistic sampling may suffice for contribution analysis, but not quantitative attribution.
Improve monitoring of impact indicators (walk before you can run).Need for more systematic sampling if it is to inform micro-simulation.