Moving Along the Impact Pathway: The Case of IAA in Bangladesh

John Antle & Roberto ValdiviaOregon State University

Charles Crissman & Khondker Murshed-e-JahanWorldFish Center

Presented at Assessing the impacts of international agricultural research on poverty and under-nutrition: A mid-term workshop for studies commissioned by the CGIAR Standing Panel on Impact Assessment (2011 – 2013), London International Development Centre, May 8-9 2012.

Project GoalsEstimate adoption, poverty reduction and household nutrition impacts from the promotion of integrated aquaculture-agriculture technologies in Bangladesh.

H1: DSAP recommended practices are economically feasible for more than 50 percent of the target populationH2: Incomplete adoption is explained by: (a) average productivity and/or cost of production; (b) high variability in productivity and/or cost of production.H3: Adopters have lower poverty and better nutrition than non-adopters. H4: Impacts are the same for small and large farms. H5: The TOA-MD model predicts adoption rates sufficiently well for use in ex ante and ex post impact assessment.H6: The TOA-MD model predicts aggregate impacts accurately without knowing adoption-outcome correlations. H7: Impacts within adopter and non-adopter sub-populations can be predicted accurately without knowing adoption-outcome correlations.

Development of Sustainable Aquaculture Project (DSAP)

• DSAP – objective to sustainably increase productivity through Integrated Aquaculture Agriculture (IAA) which focused on resource use efficiency through better utilization of resource flows between farm enterprises

• Project utilized a strategy of decentralized local-level long-term training, exposing farmers to a basket of 19 technologies and management practices.

• Large scale project implemented in 34 of the 64 districts in Bangladesh

DSAP Impact Assessment Data• DSAP monitoring activity focused on

farm survey and regular monitoring data from 260 participating farm households from four districts

• Baseline in 2003/2004• Training and extension during

repeated visits from 2003/2004 through 2005/2006 during which regular follow up data was collected via a series of whole farm monitoring surveys

• Control: 123 non-project farmers from the same districts were surveyed in 2003/2004

RANGAMATI

SYLHET

TANGAIL

BOGRA

BANDARBAN

PABNA

KHULNA

COMILLA

DINAJPUR

NAOGAON

MYMENSINGH

SUNAMGANJ

JESSORE

SATKHIRA

HABIGANJ

RAJSHAHI

NATORE

RANGPUR

NETRAKONA

BAGERHAT

SIRAJGANJ

DHAKA

KURIGRAM

FARIDPUR

BHOLA

NOAKHALI

FENI

KUSHTIA

MAULVIBAZAR

JAMALPUR

GAZIPUR

GAIBANDHA

KISHOREGANJ

JHENAIDAH

CHANDPUR

NAWABGANJ

NILPHAMARI

SHERPUR

NARAIL

RAJBARI

THAKURGAON

GOPALGANJ

MAGURA

MANIKGANJ

BARISAL

NARSINGDI

PANCHAGARH

MUNSHIGANJ

CHITTAGONG

KHAGRACHHARI

COX'S BAZAR

BRAHAMANBARIA

SHARIATPURMADARIPUR

LAKSHMIPUR

CHUADANGA

LALMONIRHAT

JOYPURHAT

PATUAKHALI

MEHERPUR

PIROJPUR

BARGUNA

BARISAL

NARAYANGANJ

PIROJPUR

JHALOKATI

BARGUNA

PATUAKHALI

BAGERHAT

NOAKHALI

BARGUNA

SATKHIRA

PATUAKHALI

KHULNA

BAGERHAT

BHOLA

KHULNA

JHALOKATI

SATKHIRA

CHITTAGONG

BARISAL

BARISAL

BHOLA

BARISAL

BAGERHATSATKHIRA

BARISAL

PATUAKHALI

PATUAKHALI

PATUAKHALI

PATUAKHALIPATUAKHALI

DSAP Survey Data

Baseline Follow up

2002/03 2003/04 2004/05 2005/06 2011/2012

Project 225 225 225 225 225

Secondary Adopter

225

Control 123 123 123 123 123

Baseline covered 260 farm households – 35 were large commercial rice-fish operations that are dropped for this analysis

Secondary adopter – selected from matching project village

Follow up survey will cover same households as in original project

Multi-Dimensional IA: Motivation

“Moving along the impact pathway” • how can the research community respond to

stakeholders’ desire to understand tradeoffs and synergies between economic, environmental and social dimensions of sustainability?

• Given realities of data quality, costs of data and human resources, how good is good enough?

• But…complexity does not trump principle of parsimony

Simulation-based IA: Motivation

• There is a need for a feasible, generic, transparent approach to multi-dimensional IA that can be implemented at “low cost” in terms of data, time and human resources, that is based on received economic and statistical theory.

• The simulation-based approach to IA responds to this demand, by building on and integrating established concepts in the technology adoption, statistics and econometrics literatures.– It is a way to integrate various kinds of data to simulate the

economically feasible adoption rate and various indicators based on quantifiable outcome variables

– It can be linked to econometric behavioral models and market equilibrium models for aggregation and disaggregation

Some features of the simulation approach implemented in TOA-MD:• Is a parsimonious, transparent model: results can be easily interpreted in

relation to underlying data and parameters• Can utilize all types of available data:

– survey data, experimental data, modeled data, meta data, expert data• Provides a framework to carry out sensitivity analysis

– Can use preliminary or “minimum data,” provide guidance for efficient collection of additional data when needed

• Can be used for prospective and policy-relevant IA (extrapolation; ex ante; analysis of policy impacts)– Overcomes the critical “support” assumption in the econometric approach that both

“treated” and “untreated” individuals are observed• Estimates various kinds of impacts

– Conventional average “treatment” effects– Mean and threshold impacts on adopters and non-adopters, at any degree of adoption– Policy-relevant impacts: taxes, subsidies, Payments for Ecosystem Services, etc.

l

mk

= w opportunity cost

l

k

1000 0

l

mk(2,

0)mk(0

)

2

1

mk(2)

mk(1,0)

mk(1)

r(2,0)

100

r(2)

l

Conceptual Model of Technology Adoption and Impact Assessment (Antle 2011 AJAE)

Contours of equal density of joint distributions of ω and outcome variable k, for systems 1 and 2

Adoption curve for system 2

Impact indicators for outcome k (in this case, mean outcomes)

Technology adoption, environmental econ, &

policy evaluation literatures

Step 1: System choice(ω)

0

Map of a heterogeneous region

Opportunity cost, system choice and adoption

Opportunity cost = v1 – v2 follows distribution ()Generalize to any ordering, e.g. “willingness to adopt”?

System 1: > 0(non-adopters)

System 2: < 0 (adopters)

opportunity cost

Key point: a model of a population using a system, not a farm using a technology

• Key point: a fundamental piece of information about a system is its economic feasibility: an upper bound on potential use or adoption

• = v1 – v2 is distributed in the farm population– “Every farm has its ”– = 1 - 2 – 2 = 1

2 + 22 - 21212

– Ex post: observe system 2, approximate system 1 (counterfactual)– Ex ante: observe system 1, approximate system 2 (counterfactual)– How to estimate 12?

• “Unobserved heterogeneity”: how to approximate?• Populations can be stratified by various criteria: geographic,

technological and socio-economic

Opportunity cost distribution

()

100

ω > 0

Derivation of adoption rate from spatial distribution of

opportunity cost with adoption threshold a = 0

ω < 0r (2,0)

r(2)

This model shows the relationship between the mean and variance of ω and the economically

feasible adoption rate. If the mean of ω is positive (negative), the adoption rate is less than (greater than) 50%. Similarly, changes in variance

have predictable effects on the adoption rate.

Link to models with “selection on unobservables”: ω is unobservable, but

observed or otherwise approximated distribution of returns can be used to

estimate the moments of the ω distribution

50

l

mk

w

l

k

1000 0

l

2

1

mk(1)

r(2,0)

100

r(2)

l

Step 2: Impact AssessmentAdopter, non-adopter, and population means of

unconditional and conditional outcome distributions

mk(2)

mk(0

)

mk is mean of outcome kWhen k is expected returns, the population mean is maximized at the adoption rate r(2,0)

Key concepts: unconditional outcome distributions (ellipsoids) and outcome distributions conditional on adoption (filled). The latter are truncated by adoption decisions.

mk(h) = mean of outcome k when all farms use system hmk(0) = population mean at predicted adoption r(2,0) for adoption threshold a=0

Population mean

Adopter mean

Non-adopter mean

l

mk

w

l

k

1000 0

l

mk(2,

0)mk(0

)

2

1

mk(2)

mk(1,0)

mk(1)

r(2,0)

100

r(2)

l

Counterfactual mean of adopters and the Average Treatment effect on the Treated (ATT)

Counterfactual mean of adoptersATT at r(2,0)Counterfactual mean of non- adoptersATU at r(2,0)

Similarly, the distribution of system 2 for ω> 0 can be used to construct the counterfactual for non-adopters and the average treatment effect on the untreated (ATU).

mk

1000

mk(2)

mk(1)

Means, counterfactuals, ATT, ATU, ATE, LATE, MTE(see Heckman, Urzua and Vytlacil, RE Stat 2006)

r(2,a)100

0

ATT = average treatment effect on the treated (adopters)

ATU = average treatment effect on the untreated (non-adopters)

ATE

Given the joint distributions of adoption variable ω and outcome k, we can compute all treatment effects as well as mean and threshold indicators at all adoption rates simulated by varying the adoption threshold a.

Also we can show that:ATE = mk(2) - mk(1) = r(2,a) * ATT/100 + {100 – r(2,a)}*ATU/100MTE = dmk(a)/dr(2,a) LATE/r(2,a)

Adopter mean

r(2,0)

Adopter CF

Non-adopter CF

Non-adopter mean

Population mean mk(a)

MTE

l

mk

w

l

k

1000 0

l

mk(2,

0)mk(0

)

2

1

mk(2)

mk(1,0)

mk(1)

r(2,0)

100

r(2)

l

Effects of selection on impact indicators: Sorting gain and counterfactual bias

Degree of selection bias in counterfactual depends on correlation between outcome and adoption variables. If correlation is zero, means and counterfactuals have zero slopes, and population mean is linear with slope equal to ATE.

Threshold indicator for system 1(areas e+f)

Threshold indicator for system 2 (areas b+d)

Threshold indicator for system 1(areas c+d+e+f)*

e

k

0

k(2,0)

k(1,0)

System 1

System 2

b

c d f

BEFORE ADOPTION OF SYSTEM 2

k

0

k(2,0)

k(1,0)

System 1

System 2

b

c d e f

AFTER ADOPTION OF SYSTEM 2

Threshold IndicatorsThe same concepts can be used to define and simulate threshold indicators (poverty rates, poverty gap, environmental risk, nutritional risk, etc).

* Ignoring areas outside the contour as negligible

l

mk

w

l

k

1000 0

l

2

1

r(2,0)

100

r(2)

l

Policy-Relevant Impacts and “Local” Impacts: E.g., a policy reducing constraints on adoption to increase adoption rate from rc to r(2,0)

rC

As in Heckman and Vytlacil (Econometrica 2005) we consider policies that affect adoption but do not change the unconditional distributions of k and ω.

Implications for IA

• There is a fundamental symmetry between “ex ante” and “ex post” IA: each involves parameterization of:– the unconditional joint distributions between and the outcome variables– the mechanism or process determining choice between systems

• By characterizing the joint distributions of the adoption variable and the outcome variables, we can simulate the economically feasible adoption rate of system 2, and all relevant average and threshold impact indicators and “treatment effects.”

• Data collection should focus on characterizing these joint distributions in the relevant populations.

• Characterization of counterfactuals (both ex post and ex ante) should utilize all relevant information: primary, secondary, experimental, modeled, expert, meta data.

Counterfactual system design

• In most cases, counterfactual system can be constructed as a transformation of the observed system– E.g., changing crop variety leaves most of the observed system intact, but may alter

productivity and land allocation– E.g., introduction of IAA changes management , productivity and “bio-resource flows”

but not the components of the system• Random coefficient model

– In general, v2 = v1 + , where is a convolution of v1 and v2

– Then v2 = (1+ /v1) v1 = riv1 giving r = (1+ /v1) where r = r + r, (0,1)– Using this model, with observations on one system and plausible bounds on r & r we

can approximate mean, variance and between-system correlations for the other system– data for r & r can come from various sources: observations, models, experiments, meta

data– Draft paper available on this concept

See Jahan and co-authors, Aquaculture Research (2010), Agricultural Systems (2011)

“The impact of long-term IAA training provided to small-scale farmers in Bangladesh is

assessed using panel data from 260 project and 126 control farmers who were monitored

from 2002/2003 to 2005/2006. We find that the training had a significant positive impact on farmers’

technical efficiency, total factor productivity and net incomes. These result in higher food consumption

and better nutrition for trained households compared to control farmers.”

• We interpret these results as “ATE”

Bangladesh DSAP Case Study

• Here we replicate and extend Jahan et al. using TOA-MD• System 1: Farms with no training support and low integration of

aquaculture and agriculture. Low integration is 2 or fewer managed bio-resource flows among farm enterprises in 2002-2003– control group data show no significant trend from 2002/03 to

2005/06• System 2: Farms with training support and highly integrated

aquaculture agriculture (treated group in 2005/06)• Farms stratified by small and large (small is less than 1 hectare)

Bangladesh DSAP Case Study (2)

DSAP Data

• 260 participating farms in the baseline survey:– 59 classified as System 1 – after the training and extension activities, 46 of these switched to

System 2, implying an adoption rate of about 78%• Farm size, pond size, family size, and non-farm income• Production activities defined as rice, vegetables, other crops, poultry

and livestock, and fish• Net return calculations based on per-farm revenues and costs of each

activity

Impact indicators

• Mean farm income and per/capita income - $1500/year and $356/year. Fish culture contributes about 16% of farm income and 11% of total annual income.

• Poverty rate – an estimated 45% to 50% are live below the $1.25/day poverty line – in the survey 54% are below the poverty line

• Food consumption – the national Household Income and Expenditure Survey shows that on average rural households consume 2253 kilo calories of which fish contribute 52 kilocalories.

The TOA-MD Model• Software with documentation in SAS and Excel, available to

registered users at tradeoffs.oregonstate.edu– Self-guided course and training workshops

• Represents heterogeneous populations with multiple strata (e.g., small, large farms; agro-ecozones; etc)

• Generic whole-farm structure– Crop, livestock and aquaculture sub-systems with multiple

activities within each system– Farm household size, non-ag income– Income, poverty and generic indicators (mean, threshold)

• Technology adoption/impact; ecosystem service supply; environmental change & adaptation

The TOA-MD Model• Some features

– Parameter parsimony: fundamental parameters are:• 2 means, 2 variances and 1 between-system correlation of system expected returns (5

parms)• 2 means, 2 variances, 3 correlations for each outcome variable (7 parms x N outcomes)

– Unconditional joint outcome distributions for each stratum of the population is normal (thus, aggregate distribution is non-normal)

– Predicted “adoption rate” (choice between systems) is based on expected returns over a relevant decision period

– When system 1 and 2 data are not matched, the between-system correlations cannot be estimated from observations, so plausible values are estimated and used in sensitivity analysis.

• Show the model…

Adoption curves for small and large farms

-1000

-800

-600

-400

-200

0

200

400

600

800

1000

0 10 20 30 40 50 60 70 80 90 100

Small Farms

Large Farms

Observed adoption rate was approximately 76% based on 59 observations , very close to the predicted adoption rate of 78% averaged over small and large farms.

Mean opportunity cost occurs at 50% adoption rate in this model. If it is negative then the adoption rate must exceed 50%, as in this case. The predicted adoption rate depends on the mean and the variance. Sensitivity to model parameters can be performed easily.

Mean net returns/farm for small farms

-500

-300

-100

100

300

500

700

900

0 10 20 30 40 50 60 70 80 90 100

Adopter Mean

Adopter CF

Pop Mean

Non-adopter mean

Non-adopter CF

ATE = 144TT = 205

Small farm net returns: ATT, ATU and MTE

-800

-600

-400

-200

0

200

400

600

800

0 10 20 30 40 50 60 70 80 90 100

ATT

ATU

MTE

Poverty rates for small and large farms

0

10

20

30

40

50

60

70

80

90

100

0 10 20 30 40 50 60 70 80 90 100

Large Farm Adopter

Large Farm Adopter CF

Large Farm Population

Large Farm Non-adopter

Large Farm Non-adopter CF

0

10

20

30

40

50

60

70

80

90

100

0 10 20 30 40 50 60 70 80 90 100

Small Farm Adopter

Small Farm Adopter CF

Small Farm Population

Small Farm Non-adopter

Small Farm Non-adopter CF

ATE = -8.3TT = -11.6

ATE = -4.6TT = -8.1

Mean fish consumption in small farm households (kcal/person/day)

0

10

20

30

40

50

60

70

80

90

100

0 20 40 60 80 100

Adopter Mean

Adopter CF

Population Mean

Non-adopter Mean

Non-adopter CF

ATE = 13.4TT = 16.5

Percent of small farm households exceeding the population average fish consumption of 51.6

kcal/person/day

0

10

20

30

40

50

60

70

80

90

100

0 10 20 30 40 50 60 70 80 90 100

Adopter Threshold

Adopter CF

Population

Threshold TU

Non-adopter CF

Note that more than 50% of adopters exceed average consumption, whereas non-adopters are very low and have very low incomes.

ATE = 30.3TT = 38

Large farm households exceeding the population average fish consumption of 51.6 kcal/per/day (%)

0

10

20

30

40

50

60

70

80

90

100

0 10 20 30 40 50 60 70 80 90 100

Adopter Threshold

Adopter CF

Population

Non-adopter Threshold

Non-adopter CF

ATE = 33.2, TT = 34.3 (note they are close because correlation between outcome and opp cost is near zero)

Sensitivity analysis to between-system correlation RHO12, small farms

-1000

-800

-600

-400

-200

0

200

400

600

800

1000

0 10 20 30 40 50 60 70 80 90 100

RHO12=0.85

RHO=0.95

RHO=0.6

Adoption curves

-500

-300

-100

100

300

500

700

900

0 10 20 30 40 50 60 70 80 90 100

Adopter Mean RHO12=0.85

Adopter Mean RHO=0.95

Pop Mean

Non-adopter mean RHO12=0.85

Non-adopter mean RHO12=0.95

Adopter mean RHO12=0.6

Non-adopter mean RHO12=0.6

Mean returns

0

20

40

60

80

100

120

0 10 20 30 40 50 60 70 80 90 100

Adopters RHO=0.85

Adopters RHO=0.95

Adopters RHO=0.6

Non-adopters RHO12=0.85

Non-adopters RHO12=0.95

Non-adopters RHO12=0.6

Poverty rates

0

10

20

30

40

50

60

70

80

90

100

0 10 20 30 40 50 60 70 80 90 100

Adopter Mean RHO12=0.85

Adopter mean RHO12=0.95

Adopter mean RHO12=0.6

Population Mean RHO12=0.85

Population mean RHO12=0.95

Population mean RHO12=0.6

Non-adopter Mean RHO12=0.85

Non-adopter mean RHO12=0.95

Non-adopter mean RHO12=0.6

Mean fish consumption

Conclusions• TOA-MD model predicts IAA adoption rate very close to observed rate• IAA technology has substantial positive impacts on farm income,

nutrition -- to be further verified with additional new data• Analysis shows IAA technology has substantial income and nutritional

benefits for both small and large farms.• Selection MAY be (but is not necessarily) important in predicting income

and nutrition effects sufficiently accurately to draw meaningful policy implications.

• Data quality is the greatest challenge to meaningful impact assessment– Especially for large surveys based on RECALL

• How “adoption” is defined is important – “adoption of technologies” or “choice between systems”?

Conclusions• Work to be done:

– complete new surveys– compare simulation and ex post statistical analysis

• Extensions– Further testing/validation: adoption, extrapolation– Further explore systematic methods to combine primary, experimental,

modeled, expert, and meta data• Using crop simulation models to construct counterfactuals for CC

impact & adaptation analysis• Use for technology adoption• Improve survey/field experiment design

– Develop methods to estimate standard errors• Can use bootstrap to construct SEs but implementing in publicly

available software a challenge (try to maintain simplicity, generic structure)

– Link to market models (Valdivia et al; IMPACT; other)