Impact Evaluation, Replication and Ethics

Richard Palmer-Jones School of International Development

University of East Anglia Presented at

Impact, Learning and Innovation: Towards a Research and Practice Agenda for the Future

Institute of Development Studies, Brighton (UK), Convening Space March 26-27, 2013

Modern impact evaluation • Addresses the attribution issue by:

– Sophisticated econometric analyses of observational data

– Randomised control trials

– (and mixed methods, theory based, process tracing, and evaluation, agent based modelling)

– phronesis

• Methods which assert their status as science by – Mathematisation, quantification, neutrality and objectivity,

and so on

• But lack a crucial component of science – Replication

• Repeatability, checking, internal, external, construct .. validity

Identification • Problem of unobserved confounding variables

– Are benefits of microfinance due to loans from the MFI? • Placement and selection biases

– More favourable areas – People who are more likely to benefit

• Identification by: – Randomisation

• Internal and external validity

– In the absence of randomisation, or with compromised • multiple regression • Natural or quasi experiments

– Single and double difference estimation

• Instrumental variables estimation • Propensity Score Matching • Regression discontinuity • Panel data estimation

How robust are these methods? • Methods require assumptions

– Many assumptions cannot be tested

• Randomisation – Common threats to validity

• Imperfect selection of subjects • Imperfect randomisation – subject agency • Imperfect adherence to treatment • Lack of blinding

– Hawthorn & John Henry effects

• Lack of external validity

• Econometric results – Data mining, result polishing, researcher, sponsor, and

editor allegiance & reluctance, and HARKing

Replication in randomised studies • Replication is the sine qua non of science

• Many RCTs do not yield the same results – Many of the problems listed above

• Gave rise to systematic review and meta-analysis – Systematic review seldom resolves issues for all concerned

even when large number of good quality trials

– Meta-analysis can make it appear that lots of weak results combine to produce a convincing (statistically and substantively meaningful one) • But missing studies

– Publication bias – “Bad Pharma”

– Researcher and institutional allegiance and reluctance

» to publish results that are not from the right hymn sheet

• Register of all studies in advance ….

Replication in observational studies

• Pure replication – Checking the (original) data and code produce the

reported results • Data and coding errors

• Statistical replication – Is the study robust to plausible changes in data

cleaning, variable construction, alternative equivalent data

• Scientific replication – Is the study robust to alternative accounts

Experiences from Replication World …In practice ….

• Little replication in practice – More replication than realised?

• Show me! [some key cases – but more generally?]

– Low incentives for replication • Difficulties

– Difficult access to data & code – Repeating the analysis is very taxing – detective work in the face of

incomplete documentation – [Publication bias & file drawer problem]

– Deterrence • Imputation of Adverse motives

– adversarial intentions » Political » Career advancement » Lack of originality

• Belligerent refutation by original authors

Examples

• Feldstein / Leimer & Lesnoy – Admit and contest with new results

• Levitt / Lott – Law suite - dismissed

• Hoxby and Rothstein – Beligerent contestation

• Accusation of political motives and misreporting

– Delayed publication (2004 –> 2007)

• Acemoglu et al. and Albouy – Beligerent contestation – Severely delayed publication (2006 -> 2012)

Experiences from Replication World …in development practice ….

• Randomisation studies (briefly) – Karlan and Zinman – randomly relax credit constraints

• Observational studies – Boyce and Ravallion, 1991 -Declining real wages of agricultural

labourers in Bangladesh – Basu, Narayan, and Ravallion, 2002 - Benefits to illiterates of

being proximate to female literates – Pitt and Khandker, 1998 - The benefits of Microfinance

especially when loaned to women … – Jensen and Oster, 2009 -The power of TV on the status of

women in India – Banerjee and Iyer, 2005 - The lasting adverse effects of colonial

land revenue polices in India – [Macro-economics] Aid and Growth

• Dollar & Burnside -> Mekasah and Tarp vs Doucouliagos & Paldam, JDS, forthcoming, 2013

• Trade and aid -> . Perraton, 2011, Journal of Economic Methodology

Access to microfinance

reduces credit constraints

enables or increases fixed

& working capital, self-employment

Increases business profits

Increases borrowing, or reduces costs of borrowing

Wage employment, production,

turnover, sales

Increases income and, or consumption education and or health

expenditure, child health and nutritional status , subjective

well-being

Changes expenditure

patterns

Women empowerment

Business losses

+

Failure to keep up repayments

Borrowing from other MFIs or

informal sources

+

Reduces income and, or consumption

-

Women dis-empowerment

-

inputs

effects

impacts

failures

11

• Pitt & Khandker’s quasi-experimental design

Source: Armendáriz de Aghion and Morduch, 2005.

“Treatment” Village “Control” Village

Eligible non

participants

Not-eligible non

participants

Would be

eligible

Would not

be eligible

Eligible

participants

0.5

acre

s c

ultiv

able

land o

wned

0

0.5

Compare eligible participants with eligible controls in treatment villages

Compare eligible participants with eligible controls in treatment and control villages

non-eligible participants

But 20% of participants have more than 0.5 acres

Control placement bias with village fixed effects

Results

• WES-LIML-FE (Roodman & Morduch, 2011) – Modest impacts when borrowing by women

– Disappear when outliers removed

• Propensity Score Matching – Chemin, 2008

• More modest effects, some negative

• Does not distinguish by gender

– (Duvendack and Palmer-Jones, 2012) • Modest, zero and negative impacts when borrowing by

women

• Impacts highly vulnerable to “hidden bias” – Hidden bias highly likely

Conclusions

• Leamer, 1983

– Let’s take the con out of econometrics

• Ioannidis, 2005

– Most published research findings are false

• Manski, 2011

– Use of econometric results in policy requires “incredible belief”

• The devil is in the detail

Conclusions (cont) • Power and interests speaks to truth?

• The development industry …. – Status and power of applied econometrics

• Cognitive biases in (evaluation) research (belief) – see patterns where there are none; – see causal relations when there are none; – overvalue confirmation; – evaluate more favourably evidence that conforms with our prior beliefs – seek out confirmation;

• Professional interests and the avoidance of cognitive dissonance – The disciplinary doxa – States of denial (with apologies to Stanley Cohen) – Economists’ ethics?

Country donors, e.g. ODA, USAID

The ‘aid industry’ version 1

Multilaterals, e.g. World bank, EU

Government agencies

International NGOs, e.g Oxfam

National NGOs Local NGOs

Projects/activities

PEOPLE

(from Gardner and Lewis, 1996)

The development industry (version 2)

Multilateral aid agencies UN …, WHO, FAO, … Bretton Woods (IMF, WB)

Bilateral aid agencies DFID, SIDA, NORAD, USAID, JICA, CIDA, ..

National governments

International NGOs (INGOs) CARE, OXFAM, Christian Aid, Save the Children WWF,

Media -TV, Newspapers, Freelance journalists

International contractors, consultants, suppliers

Local governments

National NGOs managers fieldworkers

Project & programme staff

Activists & lobbyists

Development academics

People (beneficiaries) brokers, leaders, followers, patrons, clients diverse, differentiated, gendered, included/excluded,

Politicians

Conclusions (cont)

• Ethical analysis and publication

– Respect

• the interests of research subjects and researchers

• interests of employers and funders

• Peer groups and professions

• Research as a social practise

– Publish negative or null results

– Enable replication