E-101 Thursday Section 1 T HURSDAY, 1 ST NOV., 2012 11 12.30 P.
M.
Slide 2
Roadmap for today Housekeeping Where are we? What are the
goals? Application Feedback sticky notes 2
Slide 3
Housekeeping 3 Paper II is coming up Paper III Dec. 20 th we
will start looking at this more closely once Paper II is back
Slide 4
4
Slide 5
Where are we- Assessing the performance of a system Policy and
program evaluation Curriculum, Standards & Assessment 5
Slide 6
M&E - 6 One of the most impt. aspects of this course - make
sure you walk away with it Resources International Initiative for
Impact Evaluation J-PAL UNESCO IIEP UNICEF Innocenti Research
Center UNICEF Evaluation Database World Bank Evaluation Database
Other classes Haiyan Huas M & E class Tom Kahne - Program
Evaluation
Slide 7
Weiss Reading 7 What does she have to say? Summary? What are
the tensions that might arise in the evaluation process?
Slide 8
Monitoring & Evaluation 8 There is a difference between M
and E! Monitoring: The gathering of evidence to show what progress
has been made in the implementation of programs. Focuses on inputs
and outputs. Evaluation: Measuring changes in outcomes and
evaluating the impact of specific interventions on those
outcomes.
Slide 9
Monitoring & Evaluation 9
Slide 10
10
Slide 11
11
Slide 12
Reflections - 12 What are the tensions b/w an insider &
outsider role as an evaluator? If time is money, how does one
implement process? What are the ways in which impact might be
considered? How might cultural differences impact these? Creating
local capacity for communities to monitor their own programs &
hold states accountable for the delivery of basic services?
Dissemination to the beneficiaries?!
Slide 13
Implementation Issues? 13 Political Economy Policy Context
Methods
Slide 14
Political Economy 14 What is the policy purpose? Test
innovations & defend budget Allocate budget to certain programs
Pressure to demonstrate aid effectiveness and scale up Answer
electorate Reaching Targets
Slide 15
Policy Context 15 What questions need answering? Who are the
stakeholders who want answers? Do they also have the same amount of
clout? What policy questions need answers? How much of a change
would determine success? What does the government really need, and
what will it use?
Slide 16
Methods 16 Operationalizing the outcome Process/outcome
indicators Channels of impact theory of change Underlying
assumptions [log frame] Who has access to the program? Are they the
same as the intended beneficiaries? Data collection time vs. cost
Analysis Dissemination
Slide 17
Things to think about 17 What ? Type of information and data to
be consolidated How? Procedures and approaches for data collections
and analysis Why? Why are we doing this does it support the
program/policy When? Freq. of data collection, reporting Who?
Responsibilities and capacities of focal points and resource
persons
Slide 18
Sector Analysis: What is it? Sector Review: summary of the
state of the sector Sector Assessment: success of the sector in
meetings certain goals Sector Analysis: (most comprehensive)
includes review and assessment (above), and also describes sector
needs, constraints, and opportunities for improvement. (Kremmerer,
1994)
Slide 19
Sector Analysis: What is it? Tries to answer questions relating
to: External efficiency Internal efficiency Access & equity
Administration & supervision Costs & financing (Pigozzi
& Cieutat, 1998)
Slide 20
Sector Analysis: What is it? The contents of a sector analysis
may include: 1. Synthesis 2. Economic & Financial Analysis 3.
History, Structure & Management Capacity 4. Analysis by
Sub-Sectors: 5. Special Studies 6. A list of background documents
(Pigozzi & Cieutat,1988)
Slide 21
Sector Analysis: Steps I. Initiation II. Formation of the
National Steering Committee III. Determination of the scope of the
analysis IV. Formation of the subsector teams V. Formation of the
technical analysis team VI. Instrumentation and collection of data
VII. Preliminary analysis of data VIII. Discussion of preliminary
findings IX. Collection of any additional data X. Formulation of
draft recommendations XI. Discussion of draft recommendations XII.
Revision and prioritization of recommendations. (Kremmerer,
1994)
Slide 22
Opposition sectorsSupport sectorsOpposition sectors External
actors Sector positi on Anti-system Legal Opposition Ideological
support Core support Ideological support Legal Opposition Anti-
system The Government Social sectors Political parties Pressure
groups Political Mapping: An Example
Slide 23
Slide 24
Stakeholder Map: An Example
Slide 25
Program Theory Program theoryrefers to the mechanisms that
mediate between the delivery (and receipt) of the program and the
emergence of the outcome of interest. The operative mechanism of
change isnt the program activities per se but the response that the
activities generate. - Carol Weiss, Evaluation, p. 57 You will be
reading a section of this book for next week.
Slide 26
Activity: Program Theory For each program, there may be
multiple pathways to the intended outcome. Source: Weiss, C.
(1997). Theory-based evaluation: Past, present, and future. New
Directions for Evaluation, (76), 41-55, p. 42.
Slide 27
From Program Theory to Implementation to Evaluation Logical
Frameworks can help you to organize: Design Implementation
Evaluation Developed by USAID in the 1960s Vary slightly, but
always include: 4 x 4 Narrative Summary, Indicators, Means of
Verification, and Assumptions
Slide 28
Log Frames 28 In terms of relevance, practice could be the
development of common understandings gained via an examination of
assumptions and creation of a common language for evaluation -
Garaway p.p. 718 Log frame as tool to do examine Relevance: what,
where, who Effectiveness: Process & implementation Efficiency:
Making the best use of resources Impact: Support the program
purpose
Slide 29
How do you develop a logic model? Linear process from beginning
to end Move around (iterative) Backwards (identify results then
determine the resources/inputs and activities that are required)
Can be described as If ___________ then ____________. If we have X
resources and do Y activities, then we will have Z Outputs, W
Outcomes and T Impact.
Slide 30
Putting It All Together Impact Outcome Output Impact Outcome
Output
Slide 31
How to annoy your family with what you learned at HGSE Your
Planned Work Your Intended Results Resources/ Inputs
ActivitiesOutputsOutcomes Impact Logic Model of a Family Trip
Holiday flight schedules Family schedules Holiday weather Your
exam/ paper due dates Create family schedule Get holiday flight
info Get tickets Arrange ground transport Tickets for all family
members Frequent flyer miles used Money saved Ground transport
Family members enjoy vacation Continued good family relations
Slide 32
Slide 33
Logical Frameworks Project Structure Objectively Verifiable
Indicators Means of Verification Important Assumptions Goal:
Purpose: Outputs: Activities: (Inputs)
Slide 34
Logical Frameworks Project Structure Objectively Verifiable
Indicators Means of Verification Important Assumptions Goal Purpose
Outputs Activities (Inputs) IF AND THEN
Slide 35
Logical Frameworks Project Structure Objectively Verifiable
Indicators Means of Verification Important Assumptions Goal:
Purpose: To increase the number of primary school graduates.
Outputs: Activities: (Inputs)
Slide 36
Logical Frameworks Project Structure Objectively Verifiable
Indicators Means of Verification Important Assumptions Goal:
Develop human capacity in XX country. Purpose: To increase the
number of primary school graduates. Outputs: Activities:
(Inputs)
Slide 37
Logical Frameworks Project Structure Objectively Verifiable
Indicators Means of Verification Important Assumptions Goal:
Develop human capacity in Somalia. Purpose: To increase the number
of primary school graduates. Outputs: By abolishing school fees,
expect to see more children enrolling in primary Activities:
(Inputs)
Slide 38
Logical Frameworks Project Structure Objectively Verifiable
Indicators Means of Verification Important Assumptions Goal:
Develop human capacity in Somalia. Purpose: To increase the number
of primary school graduates. Outputs: By abolishing school fees,
expect to see more children enrolling in primary Activities:
Abolishing school fees.
Slide 39
Logical Frameworks Project Structure Objectively Verifiable
Indicators Means of Verification Important Assumptions Goal:
Develop human capacity in Somalia. Purpose: To increase the number
of primary school graduates. Outputs: By abolishing school fees,
expect to see more children enrolling in primary Activities:
Abolishing school fees. All primary schools no longer requires
fees.
Slide 40
Logical Frameworks Project Structure Objectively Verifiable
Indicators Means of Verification Important Assumptions Goal:
Develop human capacity in Somalia. Purpose: To increase the number
of primary school graduates. Outputs: By abolishing school fees,
expect to see more children enrolling in primary Activities:
Abolishing school fees. All primary schools no longer requires
fees. Poll families to ask what fees exist.
Slide 41
Logical Frameworks Project Structure Objectively Verifiable
Indicators Means of Verification Important Assumptions Goal:
Develop human capacity in Somalia. Purpose: To increase the number
of primary school graduates. Outputs: By abolishing school fees,
expect to see more children enrolling in primary Activities:
Abolishing school fees. All primary schools no longer requires
fees. Poll families to ask what fees exist. Primary can be funded
without fees.