Two recent systematic reviews for development

www.3ieimpact.org International Initiative for Impact Evaluation

TWO RECENT SYSTEMATIC REVIEWS FOR DEVELOPMENT

Annette N. Brown, Deputy Director for AIES


Outline

• Review questions• Inclusion criteria• Theory of change• Search pipeline• Results• Fun methods finding


“Interventions in Developing Nations for Improving Primary and Secondary

School Enrollment of Children: A Systematic Review”

By Anthony Petrosino, Claire Morgan, Trevor A. Fronius, Emily E. Tanner-Smith, Robert F. Boruch, November 2012


Review questions

• What are the effects of interventions implemented in developing countries on measures of students’ enrollment, attendance, graduation, and progression?

• Within those studies that report the effects of an intervention on [the above measures], what are the ancillary effects on learning outcomes?


Inclusion criteria

Population/participants: Primary and secondary school aged children in LMICs

Interventions: Intended to affect one of the four primary outcomes

Comparison: No specific program comparison

Outcomes: Enrollment, attendance, graduation, and progression

Studies: RCTs and QEDs


Typology of education interventions

Demand

Reducing costs

CCTs, scholarships and non-fee subsidies

Vouchers

Abolishing school fees and capitation

grants

Providing information

Increasing preparedness

Early child development

Health/nutrition

School feeding

Supply

Buildings

Teachers

Materials

Management







“Farmer field schools for improving farming practices and farmer

outcomes in low- and middle-income countries: a systematic review”

By Hugh Waddington, Birte Snilstveit, Jorge Hombrados, Martina Vojtkova, Daniel Phillips, and Howard White, December 2012


Review questions• What is the impact of farmer field schools

on their objectives in terms of ‘endpoint’ outcomes such as increased yields, net revenues and farmer empowerment, and intermediate outcomes such as capacity building and adoption of improved practices?

• Under which circumstances and why: what are the facilitators and barriers to FFS effectiveness and sustainability?


Inclusion criteria• Population/participants: Farm households in low and

middle income countries• Intervention: Programs explicitly referred to as ‘farmer

field school’• Comparison: No specific program comparison• Outcomes: effectiveness across the causal chain

– Knowledge → adoption →– Impact on yields, revenues, environment, health, empowerment

• Studies: – Effects: experimental, quasi-experimental with controlled

comparison– Barriers/facilitators: qualitative (Critical Appraisal Skills

Programme checklist 2006)


T of ChangeInput 1 Training of trainers

Input 2 Field school

Adoption (FFS participants)

Capacity building (FFS participants)

Capacity building (FFS neighbours)

Adoption (FFS neighbours)

Measured impacts: Yield, input-output ratio, income, empowerment,

environmental outcomes, health


Input 1 Training of trainers

Input 2 Field school

Adoption (FFS participants)

Capacity building (FFS participants)

Capacity building (neighbours)

Adoption (neighbours)

- Facilitators adequately trained- Farmers and facilitators attend sufficient meetings- FFS synchronised with planting season

- Curriculum relevant to problems facing farmers-Farmer attitudes changed (convinced message appropriate)- Relative advantage over old techniques

- Field days/follow-up- High degree of social cohesion- Geographical proximity to other farmers (observation) or market (communication)

Measured impacts: Yield, input-output ratio, income, empowerment,

environmental outcomes. health

- New technology appropriate- Market access - Favorable prices- Environmental factors including weather, soil fertility


1,112 abstracts screened

751 excluded

312 full text sought 49 no access

183 Extension impact papers:134 FFS

49 non-FFS

257 excluded

1453 abstracts screened

27,866 titles screened

369 full text obtained126 no access

186 excluded:128 on relevance58 on design (no

comparison)

134 FFS impact papers

80 individual FFS studies

25 qualitative papers

Causal Chain Analysis

Effectiveness


30 IE and sister papers


Qualitative Synthesis

BB+ Synthesis


NOTE: Weights are from random effects analysis

.

.

FFS participantsHuan et al., 1999 (Vietnam)Endalew, 2009 (Ethiopia)Price et al., 2001 (Philippines)Rao et al., 2012 (India)Reddy & Suryamani, 2005 (India)Kelemework, 2005 (Ethiopia)Mutandwa & Mpangwa, 2004 (Zimbabwe)Dinpanah et al., 2010 (Iran)Khan et al., 2007 (Pakistan)Bunyatta et al., 2006 (Kenya)Erbaugh, 2010 (Uganda)Rebaudo & Dangles, 2011 (Ecuador)Subtotal (I-squared = 93.3%, p = 0.000)

FFS neighboursKhan et al., 2007 (Pakistan)Reddy & Suryamani, 2005 (India)Ricker-Gilbert et al, 2008 (Bangladesh)Rebaudo & Dangles, 2011 (Ecuador)Subtotal (I-squared = 0.0%, p = 0.610)

IDStudy

0.02 (-0.06, 0.10)0.27 (-0.06, 0.60)0.42 (-0.17, 1.01)0.43 (-0.02, 0.87)0.45 (-0.04, 0.94)0.54 (-0.22, 1.29)0.59 (0.25, 0.92)0.67 (0.41, 0.92)0.79 (0.29, 1.29)1.03 (0.65, 1.41)1.14 (0.93, 1.34)1.79 (1.17, 2.41)0.66 (0.33, 1.00)

-0.13 (-0.68, 0.42)0.05 (-0.45, 0.56)0.17 (-0.25, 0.59)0.38 (-0.15, 0.91)0.13 (-0.12, 0.37)

ES (95% CI)

0.02 (-0.06, 0.10)0.27 (-0.06, 0.60)0.42 (-0.17, 1.01)0.43 (-0.02, 0.87)0.45 (-0.04, 0.94)0.54 (-0.22, 1.29)0.59 (0.25, 0.92)0.67 (0.41, 0.92)0.79 (0.29, 1.29)1.03 (0.65, 1.41)1.14 (0.93, 1.34)1.79 (1.17, 2.41)0.66 (0.33, 1.00)

-0.13 (-0.68, 0.42)0.05 (-0.45, 0.56)0.17 (-0.25, 0.59)0.38 (-0.15, 0.91)0.13 (-0.12, 0.37)

ES (95% CI)

Favours intervention 0-.5 0 .5 1 3

Positive impacts on knowledge

among participants



.

.

FFS neighboursPananurak, 2010 (India)Khan et al., 2007 (Pakistan)Feder et al, 2004 (Indonesia)Labarta, 2005 (Nicaragua)Pananurak, 2010 (China)Wu Lifeng, 2010 (China)Pananurak, 2010 (Pakistan)Yamazaki & Resosudarmo, 2007 (Indonesia)Subtotal (I-squared = 49.5%, p = 0.054)

FFS participantsPananurak, 2010 (India)Van Rijn, 2010 (Peru)Naik et al., 2008 (India)Huan et al., 1999 (Vietnam)Labarta, 2005 (Nicaragua)Rejesus et al, 2010 (Vietnam)Feder et al, 2004 (Indonesia)Wu Lifeng, 2010 (China)Ali & Sharif, 2011 (Pakistan)Pananurak, 2010 (China)Gockowski et al., 2010 (Ghana)Yang et al., 2005 (China)Hiller et al., 2009 (Kenya)Khan et al., 2007 (Pakistan)Cavatassi et al., 2011 (Ecuador)Davis et al, 2012 (Tanzania)Birthal et al., 2000 (India)Pananurak, 2010 (Pakistan)Dinpanah et al., 2010 (Iran)Wandji et al., 2007 (Cameroon)Mutandwa & Mpangwa, 2004 (Zimbabwe)Palis, 1998 (Philippines)Zuger 2004 (Peru)Carlberg et al., 2012 (Ghana)Yamazaki & Resosudarmo, 2007 (Indonesia)Van den Berg et al., 2002 (Sri Lanka)Davis et al, 2012 (Kenya)Pande et al., 2009 (Nepal)Dinpanah et al., 2010 (Iran)Orozco Cirilo et al., 2008 b) (Mexico)Todo & Takahashi, 2011 (Ethiopia)Subtotal (I-squared = 93.0%, p = 0.000)

IDStudy

0.79 (0.63, 1.00)0.97 (0.74, 1.26)0.99 (0.97, 1.01)1.00 (0.99, 1.01)1.02 (0.98, 1.07)1.03 (0.99, 1.08)1.03 (0.86, 1.25)1.43 (1.05, 1.96)1.01 (0.98, 1.03)

0.80 (0.61, 1.05)0.86 (0.63, 1.18)0.89 (0.83, 0.96)0.95 (0.92, 0.98)0.97 (0.92, 1.02)0.97 (0.72, 1.31)0.98 (0.96, 1.01)1.08 (1.03, 1.14)1.09 (1.03, 1.15)1.09 (1.04, 1.14)1.14 (1.03, 1.25)1.15 (0.94, 1.41)1.17 (0.53, 2.56)1.17 (0.97, 1.42)1.22 (0.97, 1.53)1.23 (1.00, 1.51)1.24 (1.13, 1.36)1.24 (1.01, 1.54)1.32 (1.22, 1.42)1.32 (1.07, 1.63)1.36 (1.06, 1.73)1.36 (0.97, 1.92)1.44 (1.09, 1.92)1.58 (1.19, 2.10)1.67 (1.23, 2.26)1.68 (1.30, 2.18)1.81 (1.15, 2.84)2.11 (1.25, 3.56)2.52 (2.05, 3.11)2.62 (2.23, 3.08)2.71 (1.11, 6.60)1.23 (1.16, 1.32)

ES (95% CI)

0.79 (0.63, 1.00)0.97 (0.74, 1.26)0.99 (0.97, 1.01)1.00 (0.99, 1.01)1.02 (0.98, 1.07)1.03 (0.99, 1.08)1.03 (0.86, 1.25)1.43 (1.05, 1.96)1.01 (0.98, 1.03)

0.80 (0.61, 1.05)0.86 (0.63, 1.18)0.89 (0.83, 0.96)0.95 (0.92, 0.98)0.97 (0.92, 1.02)0.97 (0.72, 1.31)0.98 (0.96, 1.01)1.08 (1.03, 1.14)1.09 (1.03, 1.15)1.09 (1.04, 1.14)1.14 (1.03, 1.25)1.15 (0.94, 1.41)1.17 (0.53, 2.56)1.17 (0.97, 1.42)1.22 (0.97, 1.53)1.23 (1.00, 1.51)1.24 (1.13, 1.36)1.24 (1.01, 1.54)1.32 (1.22, 1.42)1.32 (1.07, 1.63)1.36 (1.06, 1.73)1.36 (0.97, 1.92)1.44 (1.09, 1.92)1.58 (1.19, 2.10)1.67 (1.23, 2.26)1.68 (1.30, 2.18)1.81 (1.15, 2.84)2.11 (1.25, 3.56)2.52 (2.05, 3.11)2.62 (2.23, 3.08)2.71 (1.11, 6.60)1.23 (1.16, 1.32)

ES (95% CI)

Favours intervention

1.5 1 2 3

Increased yields among

FFS-beneficiaries

not neighbours


Reduced environmental risk factors


.

.

FFS participants

Pananurak, 2010 (India)

Praneetvatakul & Waibel, 2006 (Thailand)

Pananurak, 2010 (Pakistan)

Cavatassi et al., 2011 (Ecuador)

Subtotal (I-squared = 8.0%, p = 0.353)

FFS neighbours


Pananurak, 2010 (Pakistan)


Praneetvatakul & Waibel, 2006 (Thailand)


ID

Study

0.52 (0.32, 0.85)

0.54 (0.39, 0.76)

0.55 (0.41, 0.75)

0.82 (0.55, 1.23)

0.59 (0.49, 0.71)

0.58 (0.24, 1.41)

0.64 (0.37, 1.10)

0.69 (0.43, 1.11)

1.04 (0.32, 3.40)

0.68 (0.49, 0.93)

ES (95% CI)

0.52 (0.32, 0.85)

0.54 (0.39, 0.76)

0.55 (0.41, 0.75)

0.82 (0.55, 1.23)

0.59 (0.49, 0.71)

0.58 (0.24, 1.41)

0.64 (0.37, 1.10)

0.69 (0.43, 1.11)

1.04 (0.32, 3.40)

0.68 (0.49, 0.93)

ES (95% CI)

Favours intervention

1.1 .2 .5 1 2


Summary of quantitative findings

• FFS increase knowledge and improve adoption of the FFS practices

• On average increasing yields and/or incomes

• Suggestions of farmers feeling empowered

• Limited, if any, spillovers• Neighbours do not adopt the practices

consistently



.

.

Overall (I-squared = 93.0%, p = 0.000)

Birthal et al., 2000 (India)


Gockowski et al., 2010 (Ghana)

Todo & Takahashi, 2011 (Ethiopia)

Huan et al., 1999 (Vietnam)

Yamazaki & Resosudarmo, 2007 (Indonesia)Pananurak, 2010 (Pakistan)


Dinpanah et al., 2010 (Iran)

Van den Berg et al., 2002 (Sri Lanka)

Wu Lifeng, 2010 (China)Ali & Sharif, 2011 (Pakistan)

Rejesus et al, 2010 (Vietnam)

ID


Carlberg et al., 2012 (Ghana)

Pananurak, 2010 (China)

Hiller et al., 2009 (Kenya)

High risk of biasNaik et al., 2008 (India)

Medium risk of bias

Davis et al, 2012 (Kenya)

Zuger 2004 (Peru)

Davis et al, 2012 (Tanzania)

Dinpanah et al., 2010 (Iran)Pande et al., 2009 (Nepal)

Labarta, 2005 (Nicaragua)

Feder et al, 2004 (Indonesia)

Yang et al., 2005 (China)

Khan et al., 2007 (Pakistan)


Palis, 1998 (Philippines)

Orozco Cirilo et al., 2008 b) (Mexico)

Wandji et al., 2007 (Cameroon)Mutandwa & Mpangwa, 2004 (Zimbabwe)

Van Rijn, 2010 (Peru)

Study

1.23 (1.16, 1.32)

1.24 (1.13, 1.36)

1.35 (1.19, 1.52)

1.14 (1.03, 1.25)

2.71 (1.11, 6.60)

0.95 (0.92, 0.98)

1.67 (1.23, 2.26)1.24 (1.01, 1.54)

1.22 (0.97, 1.53)

1.32 (1.22, 1.42)

1.68 (1.30, 2.18)

1.08 (1.03, 1.14)1.09 (1.03, 1.15)

0.97 (0.72, 1.31)

ES (95% CI)

0.80 (0.61, 1.05)

1.58 (1.19, 2.10)

1.09 (1.04, 1.14)

1.17 (0.53, 2.56)

0.89 (0.83, 0.96)

1.81 (1.15, 2.84)

1.44 (1.09, 1.92)

1.23 (1.00, 1.51)

2.52 (2.05, 3.11)2.11 (1.25, 3.56)

0.97 (0.92, 1.02)

0.98 (0.96, 1.01)

1.15 (0.94, 1.41)

1.17 (0.97, 1.42)

1.10 (1.03, 1.17)

1.36 (0.97, 1.92)

2.62 (2.23, 3.08)

1.32 (1.07, 1.63)1.36 (1.06, 1.73)

0.86 (0.63, 1.18)

1.23 (1.16, 1.32)

1.24 (1.13, 1.36)

1.35 (1.19, 1.52)

1.14 (1.03, 1.25)

2.71 (1.11, 6.60)

0.95 (0.92, 0.98)

1.67 (1.23, 2.26)1.24 (1.01, 1.54)

1.22 (0.97, 1.53)

1.32 (1.22, 1.42)

1.68 (1.30, 2.18)

1.08 (1.03, 1.14)1.09 (1.03, 1.15)

0.97 (0.72, 1.31)

ES (95% CI)

0.80 (0.61, 1.05)

1.58 (1.19, 2.10)

1.09 (1.04, 1.14)

1.17 (0.53, 2.56)

0.89 (0.83, 0.96)

1.81 (1.15, 2.84)

1.44 (1.09, 1.92)

1.23 (1.00, 1.51)

2.52 (2.05, 3.11)2.11 (1.25, 3.56)

0.97 (0.92, 1.02)

0.98 (0.96, 1.01)

1.15 (0.94, 1.41)

1.17 (0.97, 1.42)

1.10 (1.03, 1.17)

1.36 (0.97, 1.92)

2.62 (2.23, 3.08)

1.32 (1.07, 1.63)1.36 (1.06, 1.73)

0.86 (0.63, 1.18)

Favours intervention 1.5 1 2 3

Sensitivity analysis:Yields by

risk of bias status

High risk of bias studies

over-estimate impacts


THANK YOU!www.3ieimpact.org

Date post:	16-Feb-2016
Category:	Documents
Upload:	elysia
View:	31 times
Download:	0 times

Two recent systematic reviews for development

Documents