Evaluating Impact: Lessons Learned

from MEASURE Evaluation

Jason B Smith, PhD, MPHDirector for Health Systems Strengthening

Presentation Overview

• What is Evaluation?

• What is MEASURE Evaluation?

• Lessons learned from Phase 3

• Special topics:

– Strategic Information for South Africa (MEval-SIFSA)

– GEMNET-Health and University of Pretoria

– Evaluation Resources

• Questions, Comments and Dialogue

What is Evaluation anyway?

• evaluation [e-val″u-a´shun]

– a critical appraisal or assessment;

– a judgment of the value, worth, character, or effectiveness of

something;

– measurement of progress.

From : The Free Dictionary (online)

Evaluation: USAID Policy

• “Evaluation is the systematic collection and analysis of

information about the characteristics and outcomes of

programs and projects as a basis for judgments, to

improve effectiveness, and/or inform decisions about

current and future programming”

USAID Evaluation Policy, January 2011

“The two purposes of evaluation are to provide information

for decision-making and contextual learning, and to

demonstrate accountability for resources”

Evaluation of USAID, November 2013

Why Evaluate Programs?

• Determine if an intervention was delivered as per plan

• Discover if the intervention had the desired effect(s), and

• Explore whether there are ways that the intervention

could be delivered faster or better or at lower cost

Basic Components

The basic components of evaluation are:

• identifying the parameters

• developing criteria

• data gathering

• measuring the data against the criteria, and

• employing the results

From : The Free Dictionary (online)

Types of Evaluation Research

• Impact

• Outcome

• Public Health

• Methodological Studies

• Population-based Surveys and Surveillance

What is Measure Evaluation?

• MEASURE Evaluation is the USAID Global Health

Bureau’s primary vehicle for supporting improvements in

monitoring and evaluation in population, health and

nutrition worldwide.

MEASURE Evaluation Timeline

• 1991-1997 - The Evaluation Project

• 1997-2003 - MEASURE Evaluation Phase 1

• 2003-2008 - MEASURE Evaluation Phase 2

• 2008-2014 - MEASURE Evaluation Phase 3

• 2014-2019 - MEASURE Evaluation Phase 4

The MEASURE Evaluation Team

MEASURE Evaluation Phase III

Evaluation Activities

North Atlantic Ocean

Indian Ocean

Arctic Ocean

Arctic Ocean

North Pacific Ocean

Nepal

Nigeria

Senegal

Ghana

Ethiopia

South Africa

Madagascar

Mali

Tanzania

Benin

Mozambique

Kenya

India Bangladesh

MalawiAngola Zambia

Rwanda

Uganda

Guatemala

MexicoHaiti

Ukraine

MEASURE Evaluation

Guiding Principles for Evaluations

• Planning in parallel with program implementation.

• Work with country programs as well as global partners

• Partnership with local institutions to build their capacity

• Where appropriate, cost components are included

• Dissemination and use of study results incorporated from

the outset.

MEASURE Evaluation

Criteria for prioritizing evaluation activities (1)

• Comparative advantage

• Theory driven

• Clearly adds to evidence base

• Produces actionable finding

• Reasonable relative cost

• Balance of cost with ability to achieve objectives

MEASURE Evaluation

Criteria for prioritizing evaluation activities (2)

• Incorporates capacity building, data use, and knowledge

management principles

• Follows protection of human subjects principles

• Generalizability of results

• Other pragmatic elements

Lessons Learned: Ongoing Challenges

• Centralized evaluation agendas hard to implement

• Evaluations are slow – lose momentum and visibility

• Need to improve publication, communication and data use

• Designs are compromised by field realities – “real world” evaluation

• “Evaluation” means different things to different people

Lessons Learned: Implementing Evaluation

• Begin with the end in mind (but easier said than done)

• Timing is everything

• Size matters

• Pressure for rapid results to inform programs now

• Be prepared to manage expectations

• The need for ownership

Archive of Phase 3 End of Project Event Sessions

www.measureevaluation.org/eop

A Message of Hope

Special Topics

• Strategic Information for South Africa (MEval – SIfSA)

• GEMNet-Health and University of Pretoria

• Evaluation Resources

Current Associate Awards

MEASURE Evaluation in South Africa

Strategic Information for South Africa (MEval–SIFSA)

Supports the Department of Health (DoH) at national,

provincial and district levels to improve strategic

information for evidence-based management of HIV and

related health programs.

MEval-SIFSA seeks to ensure that the health system in

South Africa produces useful and high quality information

that contributes to sustainable policy, planning, and

programmatic decision making.

SIfSA Strategic Approaches

• System-wide approach

• Leveraging technology

• Data use

• Commitment to sustainability

University of Pretoria

• MEASURE Evaluation Regional Training

Partner since 1999

• Pre-service Education in M&E at MPH level

• In-Service Workshops

• Provision of Technical Assistance to others

• Support for Capacity-Building under

MEval-SIfSA

• Founding member of GEMNet-Health

Acknowledgements

This presentation is not all of my own original thinking. A

number of my colleagues have contributed to this

presentation in person, in print or by providing presentation

materials. These colleagues include Sian Curtis, Jim,

Thomas, Heidi Reynolds, Tara Nutley, Peter Lance,

Gustavo Angeles, Derek Kunata and Hemali Kulatilaka. My

apologies to anyone I may have forgotten.

MEASURE Evaluation is funded by the U.S. Agency for

International Development (USAID) through cooperative

agreement GPO-A-00-03-00003-00 and is implemented

by the Carolina Population Center at the University of

North Carolina at Chapel Hill, in partnership with Futures

Group, ICF International, John Snow, Inc., Management

Sciences for Health, and Tulane University. The opinions

expressed in this presentation are those of the authors

and do not necessarily reflect the views of USAID or the

U.S. government.

Visit us online at http://www.cpc.unc.edu/measure

Questions, Comments and Dialogue

Visit our website at:

www.measureevaluation.org