WHAT IS AN EVALUATION? Impact Evaluation Course Cape Town, South Africa 18 January 2016 Jeremy Magruder
Outline 1. Evidence Based Policymaking : Why Evaluate? 2. Levels of Evaluation
a) Needs Assessment b) Theory of Change c) Process Evaluation d) Impact Evaluation e) Cost Effectiveness Analysis
3. Good Impact Evaluation Questions
Outline 1. Evidence Based Policymaking : Why
Evaluate? 2. Levels of Evaluation
a) Needs Assessment b) Theory of Change c) Process Evaluation d) Impact Evaluation e) Cost Effectiveness Analysis
3. Good Impact Evaluation Questions
What is Policy? • Dic$onary: “A course or principle of ac$on adopted or proposed by an organiza$on or individual.”
• Wikipedia: “A principle or protocol to guide decisions and achieve ra$onal outcomes.”
How do we know if a policy achieves ra$onal outcomes?
Quiz: How is policy decided?
A. Ideology (i.e., of decision makers, par$es, cons$tuencies)
B. Experience (i.e., of decision makers, par$es, cons$tuencies)
C. Financial resources/constraints D. Implementa$on capacity
E. Scien$fic evidence
How do we increase school participation (enrollment and attendance)?
A government wants to improve school aOendance at primary schools, what interven$ons would you recommend?
How do we increase school participation (enrollment and attendance)? A. Text Books
B. Lunch for free
C. Free school uniforms
D. Treat intes$nal worms
E. Merit scholarships
F. Improve curriculum & teaching
G. Provide beOer materials
H. Increase awareness of returns to educa$on
How do you think increased use of evidence would change policy decisions? A. Change policy priori$es
B. New programs
C. New implementa$on models
D. Halt of programs that do not work
E. Reallocate resources towards higher impact programs
F. Reallocate resources toward more cost-‐effec$ve programs
Why evaluate?
A. To know whether a program is implemented as intended
B. To know whether a policy/ program works and how
C. Donors require it D. To make our policies/
programs more performance-‐based
E. To be able to improve the program and make it more effec$ve, or abandon the program if it is not
F. To iden$fy if there are unintended consequences
G. All of the above
What is evaluation?
A. A survey to see how people are doing, how they feel about things
B. A structured way of thinking through the details of a program
C. Roughly the same thing as monitoring
D. Judgment of whether, and how much a person, organiza$on or program succeeded or failed
E. All of the above
Outline 1. Evidence Based Policymaking : Why Evaluate 2. Levels of Evaluation
a) Needs Assessment b) Theory of Change c) Process Evaluation d) Impact Evaluation e) Cost Effectiveness Analysis
3. Good Impact Evaluation Questions
Levels of Programme Evaluation
Needs Assessment
Program Theory Assessment
Process Evaluation
Impact Evaluation
Cost-Effectiveness Analysis
Needs Assessment
• Clear sense of target popula$on • Clear sense of need program will fill • Clear ar$cula$on of program benefits
• Clear sense of alterna$ves
Needs Assessment: Descriptive Data • Good descrip$ve data is cri$cal for understanding the nature of a problem.
• Assessing the quality of descrip$ve data: 1. Is the sample large and is it representa$ve—ie does it
represent different types of people
2. Are the ques$ons appropriate to the local context?
Example: Needs Assessment In 2012 in Monrovia (pre-‐Ebola): • Extremely high youth unemployment (only 20-‐30% of working age youth employed)
• Crime and an$-‐social behavior is very high
• Post conflict environment: concerns about social integra$on and economic opportunity
• 2 widespread programs for social integra$on and economic opportunity of youth • Life skills trainings coupled with sports programs
• Cash for Work
Theory of Change • How will the program address the needs put forth in your needs assessment? • What are the prerequisites to meet the needs?
• How and why are those requirements currently lacking or failing?
• How does the program intend to target or circumvent shortcomings?
• What services will be offered?
Example: Theory of Change
Teachers and
Coaches are hired to train young adults in life skills; curriculum developed
Classes are held and
sports events organized.
Youth Attend.
Youth Learn new life skills;
pro-social attitudes improved
Employment increases; anti-social behaviors decrease
Input Output Outcome Goal
Example: Log Frame Objectives Hierarchy
Indicators Sources of Verification
Assumptions / Threats
Impact (Goal/ Overall
objective)
Higher Employment Rates and lower crime
• Employment Status
• Crime/anti-social behaviors
Household survey Skill gaps and social integrations barriers to employment
Outcome (Project
Objective)
Youth learn valuable life skills and increase pro-sociality
• Survey data • Test Scores
• Household surveys
• Routine data • Cortisol
testing?
• Behavior change • Value to
prosociality /life skills indicators
Outputs Youth Trained in Life Skills and engage insports events
• Number of Youth participants
• Attendance logs, unscheduled audits
• Instruction is valuable
• Youth learn from this format of course
Inputs (Activities)
Teachers and Coaches hired and trained in curriculum
• Teachers and Coaches Hired
• Curriculum Documents
• Routine data * Teachers and Coaches will hold classes and follow curriculum
Process Evaluation • Are basic tasks being completed?
• Are the services being delivered? • Is the interven$on reaching the target popula$on? • Is the interven$on being completed well or efficiently and to the beneficiaries’ sa$sfac$on?
Process Evaluation • Process evalua$ons answer whether the program or policy was carried out as planned.
• Good quality process assessment goes beyond coun$ng inputs
1. Not just did the text books arrive but are they being used?
2. Not just were people hired, but how ofen are they absent?
3. What percentage of eligible people take up the program?
4. How many youth par$cipated in the program?
Example: Process Evaluation
Process Evaluation
Teachers and Coaches are hired to train young adults in life skills; curriculum developed
Classes are held and
sports events organized.
Youth Attend.
Youth Learn new life skills;
pro-social attitudes improved
Employment increases; anti-social behaviors
decrease
Input Outcome Goal
Impact Evaluation • Process evalua$ons determine if a program is running in the way it is supposed to run • Ex: Did teachers distribute deworming pills?
• Impact evalua$ons determines if a program creates a change in an outcome(s) • Approaches: non-‐experimental, quasi-‐experimental and experimental methods (randomized evalua$ons)
• Ex. Did financial rewards for hairdressers result in customers prac$cing safer sex ?
• Ex. Did the provision of deworming pills create an increase in school aOendance?
Example: Impact Evaluation
Process Evaluation Impact Evaluation
Teachers and
Coaches are hired to train young adults in life skills; curriculum developed
Classes are held and
sports events organized.
Youth Attend.
Youth Learn new life skills;
pro-social attitudes improved
Employment increases; anti-social behaviors decrease
Input Outcome Goal
Outline 1. Evidence Based Policymaking : Why Evaluate 2. Levels of Evaluation
a) Needs Assessment b) Theory of Change c) Process Evaluation d) Impact Evaluation e) Cost Effectiveness Analysis
3. Good Impact Evaluation Questions
What types of questions can you answer with impact evaluation?
• Evalua$ons of all kinds can be thought to answer three types of ques$ons (Imas and Rist 2009) : • Descrip(ve ques(ons • Norma(ve ques(ons
• Cause and effect ques(ons
• Impact Evalua$on can only answer cause and effect ques$ons.
Which one of these would make a good question for an impact evaluation? A. What share of kids in
Tanzania drop out of school before comple$ng primary?
B. How should a land contract be wriOen?
C. Does providing kids with deworming pills or school uniforms do a beOer job of keeping kids in school?
Which one of these would make a good question for an impact evaluation? A. Do s$ckers in a minibus
encouraging passengers to remind their drivers to drive safely reduce accidents?
B. What share of farmers in Kenya currently live on less than $2 a day?
C. Which kind of fer$lizer works best for a plot of maize?
Which one of these would make a good question for a process evaluation? A. Does a sexual educa$on
program or free school uniforms have a bigger effect on teenage pregnancy rates?
B. Do teenage girls have a right to have full informa$on regarding sexual educa$on?
C. How does a sexual educa$on program affect the pregnancy rates of teenage girls?
D. Are teachers implemen$ng a sexual educa$on program as designed?