Impact evaluation: External and internal stakes
Impact evaluation seminar - 2 to 6 December, Phnom Penh
International context and stakes
Uncertainty on aid, Paris Declaration 2005, achievement of MDG (Millennium Development Goals) and after-MDG
Highlights the Results-Oriented Management (ROM) Focuses on results/effects/impact Insists on participative approach
Since the 90s, increasing initiatives to improving quality and impact of humanitarian action
Sphere, HAP Standards in Accountability and Quality, Code of behaviour, Don not harm, etc.
Increasing concern for understanding changes induced by humanitarian aid
Impact evaluation seminar - 2 to 6 December, Phnom Penh
• Stake of accountability: Stakeholders have a moral obligation to allocate necessary means to the purpose of their mission and to prove the changes made by their action;
• Stake of visibility: Need for demonstrating one’s added-value among an increasing number of stakeholders in the humanitarian sector and international solidarity;
• Stake of financial funding: donors prescriptions to prove impact and/or changes along with the development of new tools and approaches (Value for money, Theory of change, etc. )
International context and stakes
Impact evaluation seminar - 2 to 6 December, Phnom Penh
How we keep busy How we make a difference
Outputs Outcome ImpactInputs
Results Chain
Crossing the red line....
Définitions Outputs/Extrants/Produits: The immediate results of the organisation’s activities – the processes, goods and services that it produces. The organisation controls its outputs.→ 40 health professionals are trained on inclusive approach, reception and follow-up of people with disabilities
Outcomes/Effets: The likely or achieved short-term and medium-term effects of an intervention’s outputs. They contribute to the improvement of people’s life. The organisation only influences outcomes, besides external factors.→ Improved access to health services for persons with disabilities → Improved health conditions of persons with disabilities
Impact:Long-term, sustainable changes in the conditions of people and the state of the environment that structurally reduce poverty, improve human well-being. Your organisation contributes partially and indirectly to these enduring results in society or the environment.→ Improved quality of life of persons with disabilities
Impact evaluation seminar - 2 to 6 December, Phnom Penh
1950/60s 1980/90s1970/80s
Impact analysis ex ante to anticipate the economic effects of
interventions
Taking account of unexpected and negative effects, one-dimensional models (Cost-profit analysis, social
impact measurement, environmental impact measurement)
Evolution towards a multidimensional comprehension
of impact, development of ex-post evaluations
Linear approach of change, target populations as objects of studies and not actors of change
Rural assessment participative approach, populations as actors of
change
A short story of impact
2000 →
Elaboration of new approaches; theory of
change, outcome mapping
Two main approaches
Usual methods of impact evaluation: different types of studies and evaluation, sometimes requiring important resources;
Approaches measuring effects by being integrated within the intervention with the use of tools monitoring changes at different levels (short and mid-term effects, sometimes impact)E.g.: The planning, Monitoring and Evaluation tool of ACTIONAID, highlights learning and ownership (ALPS for Accountability, learning and planning system)
Impact evaluation seminar - 2 to 6 December, Phnom Penh
Why measuring impact?
CREDIBILITY
LEARNING
ACCOUNTABILITY
ADVOCACY IMPACT
What has changed?
For whom?
How significant was it?
Will it last?
In what ways did we contribute to these changes?
Key questions on impact measurement
Impact evaluation seminar - 2 to 6 December, Phnom Penh
Responsibility of impact
Impact evaluation seminar - 2 to 6 December, Phnom Penh
« The systematic analysis of lasting or significant change –positive or negative,
intended or not – in people’s lives brought about by an action or a series of actions »
(Roche, OXFAM)« Measuring the changes taking place as the
result of an emergency project (The Good Enough Guide Impact Measurement and
Accountability in Emergencies)
One or several definitions of impact?
Impact evaluation seminar - 2 to 6 December, Phnom Penh
Common elements of definition: • concept of change/effects,• changes are positive or negative,• changes are expected/desired or not
Elements of definition adapted to each intervention type and environment:
• the object of change,• the sustainable, significant or structuring aspects of change
Towards an open definition?
Impact evaluation seminar - 2 to 6 December, Phnom Penh
Impact measurement challenges
When initiating an impact evaluation?
Which approach? What level of investigation?
What resources?
What participation of the different stakeholders?
What responsibility to changes?
Impact evaluation seminar - 2 to 6 December, Phnom Penh
“Handicap International reaffirms the importance of taking effective action, generating positive and
measurable effects and introducing changes that bring about tangible improvements for the beneficiaries, and for people with disabilities in particular. For Handicap
International, the development of services and facilitating access to these services are essential to achieving a
positive impact on the living conditions of beneficiaries».
2011-2015 Federal Strategy Chapter 2 - An approach centred on
achieving a measurable impact
HI and partners’stakes in Impact measurement
Sharing common references and terminology
Proposing and implementing rigorous methods and tools adapted to projects and intervention contexts
Asking the good questions to each effect or impact measurement initiative: What, when, how, why? Integrating effect or impact measurement within or Monitoring-Evaluation tools and systems
Supporting a balance between the quantitative and qualitative analysis
Analysing and taking into account negative effects of projects
Séminaire Evaluation d'impact du 2 au 6 décembre, Phnom Penh
Seminar objectives Identify and analyse HI and partners’ practices on effect and impact measurement, share experiences, draw recommendations for improving tools and methods;
Identify, by technical domain and by intervention context, the areas of change expected, the different levels of change and the related indicators;
Evaluate the possibility of integrating and implementing new approaches of change or impact measurement; criteria of choice and identification of resources;
Identify the ways of integrating change follow-up and measurement into Monitoring-Evaluation tools
Séminaire Evaluation d'impact du 2 au 6 décembre, Phnom Penh
LET’S GO TO WORK!
Impact evaluation seminar - 2 to 6 December, Phnom Penh