Date post: | 04-Jun-2018 |
Category: |
Documents |
Upload: | laura-roper |
View: | 225 times |
Download: | 0 times |
of 19
8/13/2019 2013 Roper Making MEL Powerful
1/19
Making MEL Powerful
December 2013
Laura Roper, Ph.D.
Roper LYV Consulting
8/13/2019 2013 Roper Making MEL Powerful
2/19
Obstacles to MEL
Poor program planningaspirations far exceed capacities; sloppythinking (lack of theory of change; theory of action); MEL notresourced
Overwhelmed and under-resourcedlack of time biggest foe togood MEL
Reluctance to be held accountable or to hold others accountable MEL imposed from above or outside; MEL gets caught up in
politics;
MEL design inappropriate to programtoo heavy; measures thewrong things; culturally inappropriate; data not used [Somerville
housing example] No incentives to do MEL (not in work plan, not part of
performance review)
8/13/2019 2013 Roper Making MEL Powerful
3/19
8/13/2019 2013 Roper Making MEL Powerful
4/19
The Power of Good MEL
Closing the Loop
Building off tested instruments (e.g. survey questions into
culturally adapted focus group questions); then focus group
questions/responses informed survey design.
Surveys (frequently) and focus groups repeatedlongitudinal data
to test their hypothesis about changing needs over time. Mixed methodsFGD, survey, content analysis of text messages
Generated actionable information (e.g. around need for and
effectiveness of cholera prevention messaging; most trusted
media outlets; information needs [invalidating hypothesis aboutmen and women having different needs], etc. etc.)
8/13/2019 2013 Roper Making MEL Powerful
5/19
The Power of Good MEL
Closing the Loop
Well presented in a visually compelling wayMost Important Issue
8/13/2019 2013 Roper Making MEL Powerful
6/19
Closing the LoopLearn A Little, Do a Little
Diagnostic/Ne
edsAssessment
MediaProgram
Content
UserSatisfaction,
Knowledge
Adjusting Radio
Programming
AND
Aid Delivery
UserSatisfaction,
Knowledge &
Actions
Dialogic
At inception, 3 month, 6month, 1 year
8/13/2019 2013 Roper Making MEL Powerful
7/19
A Review of the Basics
The Basic Evaluation Questions
How do we do what we do better?
What difference does it make?
Does it make more difference than other investments?
8/13/2019 2013 Roper Making MEL Powerful
8/19
Self-Interested MEL
Proactive on what are fair terms for judging your work, dont
let the donor dictate
Use MEL reporting to tell a compelling narrative about yourwork (why it is important; how you confront challenges; that
you are strategically nimble; that you are always building up
your foundation of alliances/political capital)
Use MEL to educate your colleagues, make the more effective
stakeholders
8/13/2019 2013 Roper Making MEL Powerful
9/19
Laying Out Program Cycle
Problem Identification
and Diagnosis
Stakeholder
Analysis
Theory of ChangeActors
Motivations
How Power is
Deployed and
other assumptions
Causal If-Then to
Get to Solution
(Theory of Action)
Strategy
GoalsObjectives
Strategic Elements
- ActivitiesImplementation
(Operational Plans/Log Frame)
-Activities completed (outputs)
MEL
(contin-
uous and
iterative)
-- First Order Outcomes
-- 2nd, 3rd, etc. Order Outcomes
-- Impacts
G
O
A
L
8/13/2019 2013 Roper Making MEL Powerful
10/19
Addressing Gun ViolenceTheory of Change:
All the major factors and actors that affect gun-
related morbidity and mortality
Theory of ActionGun Legislation
(i.e. what your
organization can
address)
Theory of Action
Gun Violence as
Public Health
problem
Theory of Action
Drugs and Gang
Violence
Theory of ActionMedia and Video
Violence
LOG FRAME: passing gunlegislation
activities outputs outcomes impacts
Obj 1
Obj 2
Obj 3
Getting from Broad Theory of Change to
Organizational ActionA Three Step Process
8/13/2019 2013 Roper Making MEL Powerful
11/19
Assessment Levels of MEL and
what they can tell you
Theory of Change
Actors
Motivations
How Power is Deployed
and other assumptions
Causal If-Then to
Get to Solution
Strategy
GoalsObjectives
Strategic Elements
- Activities
Implementation
-Activities completed (outputs)
-- First Order Outcomes
-- 2ndOrder Outcomes
-- 3rdOrder Outcomes
-- Impacts
Monitoring: Did you
do what you said you
would and did you do
it well?Evaluation: Did you
get the results you
expected; why or
why not?
Whats the
problem?
Theory,
Strategy,
Execution?
Learning: What are
we going to dodifferently in this
instance and in
similar or future
cases?
8/13/2019 2013 Roper Making MEL Powerful
12/19
Need clarity on program goals/objectives (end state)
AND point of departure (baseline)
Problem Identification
and Diagnosis
Power Analysis
Theory of Change
Actors
Motivations
How Power is
Deployed and
other assumptions
Causal If-Then to
Get to Solution
Strategy
GoalsObjectives
Strategic Elements
- ActivitiesImplementation
-Activities completed (outputs)
-- First Order Outcomes
-- 2ndOrder Outcomes
- 3rd Order Outcomes
-- Impacts
Baseline
End State-Accountable
for delivering(aspirational vs
realistic; process vs
specific)
Intermediate
markers
Benchmarks
G
O
A
L
8/13/2019 2013 Roper Making MEL Powerful
13/19
Finding the Breadth/Depth BalanceBroad, but Light
Routine Monitoring,
Self-Assessment and reportingacross projects/programs
Targeted
Evaluation
Problem Solving
Best Practice
Cost-
effectiveness
Capacities built
Summative Evaluation
AccountabilityBroader
Learning;
More strategic, less
expensive if monitoring anddocumentation has been
strongLearning
Create space (time andincentives)
Create
Framework/Strategy forlearning
Create Platforms
Knowledge
Management - Tools
Formative
Mid-term or
critical juncture
8/13/2019 2013 Roper Making MEL Powerful
14/19
8/13/2019 2013 Roper Making MEL Powerful
15/19
What type of evaluator are you?
8/13/2019 2013 Roper Making MEL Powerful
16/19
8/13/2019 2013 Roper Making MEL Powerful
17/19
Regardless of what type of evaluator you are,
there are some basic principles to follow (AEA
Principles for evaluators:
A. Systematic Inquiry
B. Competence
C. Integrity/Honesty
D. Respect for People
E. Responsibilities for General Public/WelfareAEA Principles for Evaluators
8/13/2019 2013 Roper Making MEL Powerful
18/19
Ethical Principles in Data Collection in
Humanitarian Response - InterAction
Broad PrinciplesRespect, Do No Harm, Non-
Discrimination
Broad Principles Operationalized:
Risk Benefit Analysis
Informed Consent
Confidentiality
SecurityFairness
Dignity (subjects, not objects of evaluation)
8/13/2019 2013 Roper Making MEL Powerful
19/19
Mindful Practice
Know you strengths
Know your biases
Be clear what values youre bringing to the process
Seek information from multiple sources; ideally use
mixed methodsMake an extra effort to identify and reach stakeholders
who may not typically have a voice
Listen to what is said and what is going unsaid
Dont just document; seek to understandBe familiar with resources in your field; seek out expert
advice when you need it.