Evaluating Humanitarian Aid:

November 2013Harry Shannon

McMaster University

Working with

• David Sanderson Director, Centre for Development and Emergency Practice (CENDEP), Oxford Brookes University

• Charles Parrack, CENDEP

Outline of talk

• What is (humanitarian) aid• Issues in evaluation of aid• Shelter after disaster• Proposed research agenda for evaluation• Discussion

Work in Progress!

Humanitarian imperative

• Helping fellow human beings in (desperate) need• “How we think about each other, our relationships,

our place in the ecosystem … are the most basic questions … We must recognise humility; trying and doing; governance … given them, don’t wait for the perfect answer.”– James Orbinski, former President of MSF

Maimonides: hierarchy of charitable giving

1. The lowest: Giving begrudgingly and making the recipient feel disgraced or embarrassed.

2. Giving cheerfully but giving too little.3. Giving cheerfully and adequately but only after being asked.4. Giving before being asked.5. Giving when you do not know who is the individual benefiting, but the

recipient knows your identity.6. Giving when you know who is the individual benefiting, but the recipient

does not know your identity.7. Giving when neither the donor nor the recipient is aware of the other's

identity.8. The Highest: Giving money, a loan, your time or whatever else it takes to

enable an individual to be self-reliant.

charitywatch.org

Definitions (though boundaries fuzzy)

• (Overseas Development) Aid: flows to developing countries and institutions whose aim is to promote development and welfare of the recipient countries.

• Humanitarian aid/action (HA): aid that follows a disaster (complex emergency), typically on a more urgent basis than regular ODA

Contrast with …

• Humanitarian intervention: the use of military force by a state to end human rights violations by a state (R2P)

Millennium Development Goals, 2000

By 2015:

1. Eradicating extreme poverty and hunger2. Achieving universal primary education3. Promoting gender equality and empowering women4. Reducing child mortality rates5. Improving maternal health6. Combating HIV/AIDS, malaria, and other diseases 7. Ensuring environmental sustainability8. Developing a global partnership for development

Millennium Development Goals

• Specific targets set, e.g., between 1990 and 2015, halve the proportion of people living on less than $1.25

• Ban Ki Moon: ‘The Millennium Development Goals (MDGs) have been the most successful global anti-poverty push in history.’

• However, progress has been uneven• Being replaced in 2015 by Sustainable Development

Goals

Development aid

• $140 billion / year• Dambisa Moyo, William Easterly - sceptics• Jeffrey Sachs – proponent• Paul Collier - ?intermediate• Esther Duflo, Abhijit Banerjee – RCTs

(Randomized Controlled Trials)

• All support humanitarian aid (HA)

Duncan Green is Senior Strategic Adviser at Oxfam GB

Duncan Green, From Poverty to Power

Sceptical view of Canada’s NGOs

Humanitarian aid ‘system’ - some numbers

• Estimated 4,400 NGOs worldwide doing some HA(Haiti sometimes called a ‘Republic of NGOs’)

• 2010: estimated 274,000 humanitarian workers worldwide

• Approx $14 billion / year

Humanitarian aid ‘system’

• UN humanitarian agencies• International Movement of the Red Cross and

Red Crescent• Five largest International NGOs

Médecins Sans Frontières (MSF) Catholic Relief Services (CRS) Oxfam International International Save the Children alliance World Vision International

• Increasing attention to ‘accountability’

Cluster approach – 2005 Inter-Agency Standing Committee (IASC)

CODE OF CONDUCT for the International Red Cross and Red Crescent Movement

and NGOs doing humanitarian aid activities

1) The humanitarian imperative comes first. 2) Aid is given regardless of the race, creed or nationality of the

recipients and without adverse distinction of any kind. Aid priorities are calculated on the basis of need alone.

3) Aid will not be used to further a particular political or religious standpoint. (Independence)

4) We shall endeavour not to act as instruments of government foreign policy. (Neutrality)

5) We shall respect culture and custom. 6) We shall attempt to build disaster response on local

capacities.

7) Ways shall be found to involve programme beneficiaries in the management of relief programmes.

8) Relief aid must strive to reduce future vulnerabilities to disaster as well as meeting basic needs.

9) We hold ourselves accountable to both those we seek to assist and those from whom we accept resources.

10) In our information, publicity and advertising activities, we shall recognise disaster victims as dignified human beings, not hopeless objects.

Emphases added

CODE OF CONDUCT for the International Red Cross and Red Crescent

Movement and NGOs doing humanitarian aid activities

What should humanitarian groups aspire to?

Some ethical dilemmas

• Do we respond in Japan, where it’s expensive, or in the Congo where we can do more for less?

• If we work with, e.g., the Syrian government, do we risk being co-opted into their strategy?

• If we bring food, do we drive down market prices and upset local agriculture?

• If we give out food at distribution points, do we put people at risk of attack when they walk home?

Ctd …

• What is the balance between paternalism and autonomy?

• Do we speak out about terrible things (only) we see, or stay quiet, do our work and save lives, and let others speak out?

• Relations with military?

These decisions are made under (severe) time pressure, and possibly armed force and persuasion/coercion.

Some ethical dilemmas (ctd)

Source: Hugo Slim

The Guardian, 28 August 2013

On 14 August, Médecins Sans Frontières (MSF) announced that, after 22 years, it was closing all its activities in Somalia as “the situation in the country has created an untenable imbalance between the risks and compromises our staff must make, and our ability to provide assistance to the Somali people.”

…

Of the 160 aid workers killed, 82 wounded and 99 kidnapped in Somalia since 1997, the majority have taken place since 2007 (106 killed, 53 wounded and 57 kidnapped), when the overall security environment deteriorated for both humanitarians and Somali civilians.

The Guardian, 28 August 2013

Report on aid worker securityOctober 2013

Some practical concerns with HA

• Little argument about the principle• Questions about how well HA is provided• Speed of response• Coverage of those in need• Duplication• Unnecessary help• Doing more harm than good (cholera in Haiti)

Critique of humanitarian aid

• Use of HA for political purposes has been going on for a long time

• British soldiers in Afghanistan – giving food in exchange for information on Taliban hiding places

• Colin Powell to NGOs: You are our force multipliers• In Darfur, aid workers have provided actionable

information for ICC• Secret services infiltrate NGOs• Embassies keep an eye on what ‘their’ NGOs are

doingSource: Antonio Donini

Other possible problems with aid

• NGOs attract talent from local governments• Effect of internationals on local prices• NGOs supplant government roles• Plethora of NGOs; potential competition• Government corruption• Qualifications to provide help• (Non) Involvement in politics• Promised money may not arrive

Port-au-Prince, October 2011

Humanitarian/development aid or conscience money?

Amira Hass in Israeli newspaper Haaretz, 6 Feb 2013

7 Golden Rules of Recovery Management1. Trust survivors and avoid paternalism2. Trust survivors and enable survivors to assess

their own needs3. Trust survivors and provide cash rather than kind4. Think locally5. Give priority attention to vulnerable groups6. Think ‘process’ not product, ‘sheltering’ not

shelters, ‘housing’ not houses7. Adopt a long-term perspective while addressing

short-term needsIan Davis + Anshu Sharma, 14 Nov 2013

Toby Ord: “Aid Works (on Average)”Talk in Oxford, 13 February 2013

• Cited Moyo: – Over 60 years, more than US$1 trillion in aid and not much good

to show for it– $1 trillion is nearly $1,000 per person on the planet today

• Ord:– Nonsense, actually 1 billion people in Africa– $1 tn / 440 mn (average number of people in Sub-Saharan Africa

over period)= $2,263 per person = $38 per person per year

Toby Ord: “Aid Works (on Average)” (ctd)

Successes include:Smallpox eradication Lives saved since eradication:

60-120 millionOral rehydration therapy for diarrheal diseases

Annual deaths down from 4.6 million to 1.6 million

Immunization Measles annual deaths down from 5 million to 1.2 million

Malaria Annual deaths down from 3.8 million to 0.7 million

Why evaluating ‘aid’ is difficult

• Reason aid is given• Reverse causation• Non-developmental aid • How/where aid is actually spent• Fungibility• Multiple donors• Remittances• Unpredictability of aid• Short- vs. long-term impact of projects

Aid or …?

The Guardian, 21 February 2013

Critique of methods in evaluating aidStuckler, McKee, Basu 2012

• Reviews are narrative not systematic• Evaluations don’t consider bias, confounding• Multiple outcome measures• Time lags• Data limitations• Power, Type II errors

They advocate:• Experimentation• Evaluation• Replication

• Process evaluation: how the intervention was done• Impact evaluation: longer-term, intended or

unintended • Indicator – measure of the effect

Evaluating aid - Definitions

Experiments

• Small-scale RCTs• Examples:– paying parents if child

attends school regularly

– giving food when children vaccinated

– microcredit

Increasing attention to accountability

• ALNAP - Active Learning Network for Accountability and Performance in humanitarian action

• Jim Kim, President of World Bank: “Science of delivery [of aid]”

• Various documents produced by different NGOs on how to evaluate

Evaluating the humanitarian system

ALNAP• Collates evaluations and produces summary

reports• State of the Humanitarian System:

‘assesses progress and performance in six areas: coverage/sufficiency, relevance/appropriateness, effectiveness, connectedness, efficiency and coherence.’

• Interactive online evaluation training materials

Difficulties in evaluating HA impact

• No baseline / control (or poor data)• Need for rapid information• Insecure / volatile environments• Surveys in settings with little information on

the target population• Impact of single intervention vs broader

impact of programme• Lack of capacity

Qualitative and quantitative methods

• “… much of the guidance for the evaluation of humanitarian action suggests that evaluations are more likely to provide robust evidence where they use ‘mixed methods’ approaches.”

• “However, in practice, humanitarian evaluation ‘uses mainly qualitative methods’ (Buchanan-Smith and Cosgrave 2012).”

• “… evaluations still tend to undervalue the experience of affected populations as a source of evidence …”

James Darcy and Paul Knox Clarke, 2013

Focus on Shelter

• Seen as human right• One of Clusters – incorporates settlement and non-

food items• Housing used for purposes other than purely shelter• Relatively little on shelter evaluation• ‘Providing adequate shelter is one of the most

intractable problems in international humanitarian response’ ‘(Humanitarian Emergency Response Review, 2011)

• ‘Building Back Better’

Housing and health

Shaw, Ann Rev Pub Health, 2004

Right to adequate housing

• Article 25 of Universal Declaration of Human Rights

• Adequacy includes– legal security of tenure– availability of services– affordability– habitability– accessibility– location– cultural adequacy

UNHCR, 1991

Section in Sphere Handbook

Proposed timing of shelter after disaster

Emergency shelter

Temporary / transitional shelter

Permanent shelter / reconstruction

0-2 weeks

2 years +

2 weeks – 2 years

Charles Parrack

Relief

Recovery

Reconstruction

Shelter damage in Port-au-Prince after 2010 Haiti earthquake

Damage to housing % 95% CI

Completely destroyed 24.4 22.1-26.9

Partly damaged 41.5 38.9-44.1

No visible damage 34.1 31.7-36.6

Kolbe, Hutson, Shannon et al., Medicine, Conflict and Survival, 2010

Shelter for displaced - refugees

Proposed agenda for evaluating shelter after disaster

• Step 1a: Systematically identify indicators• Step 1b: Improve list by consulting past and

potential beneficiaries• Step 2: Develop (short) questionnaire• Step 3: Conduct longitudinal study (RCT?)

after disaster

Housing trade-off: durability vs comfort

© David Sanderson; with permission

Step 3: Conduct longitudinal study/RCT after disaster

• Timing: – start 1-2 weeks after disaster – follow up at 6 months, 1 year and 2 years

• Allocate areas/communities/villages to different approaches to housing, i.e., cluster allocation/randomization

• E.g., ‘T-shelters’ vs. housing in camps vs. building materials and tools

• Mixed methods – quantitative and qualitative

Feasibility of RCTs?

Example of randomization at the individual level

Technical problems with RCTs include the need to deliver the treatment in a standardised way, the difficulties of maintaining stable membership in the treatment and control groups, and the expense and effort needed. These constraints led a review on evaluations of health promotion by the World Health Organization in Europe to find that the use of RCTs for this was ‘in most cases, inappropriate, misleading and unnecessarily expensive’ (WHO Regional Office for Europe, 1998, p. 5). RCTs also require that the evaluation be designed from the start of the project with random selection of control and treatment groups. For all of these reasons, RCTs are rare in humanitarian evaluation.

(emphasis added)

ALNAP EHA Pilot Guide p.113

Example of randomization at the community (cluster) level

Discussion