Post on 19-Jan-2017
transcript
Monitoring, Evaluation,
Accountability &
Learning
Key objectives of this session:■ Understand key components of the
global MEAL framework■ Share feedback on the global Voice
MEAL framework■ Work on a first draft of the global
Voice MEAL plan■ Address questions and concerns
Let’s review some concepts
Output Input Outcome
Monitoring Evaluation
The financial, human, and materialResources used for each intervention
Measurable results that help quantify the reach and scale of the Voice programme
Short, medium and longer term changes that Voice plans to achieve in the areas of capacity strengthening and linking and learning
IndicatorsA qualitative or quantitative factor or variable that provides a simple and reliable means to measure change
The continuing function that uses systematic collection of data to provide progress against objectives
The systematic and objective assessment of an on-going or completed project, program or policy, its design, implementation and results.
Program vs. Project
• A set of strategically aligned, mutually reinforcing interventions – that contributes to sustained, positive change or several target group members’ lives.
• A program = several projects which can begin and end within the lifetime of a program.
well defined target group
period for implementation
one / several grantees or
partners
Funding is allocated at this level
Contractual relationship
PROGRAM PROJECT OR GRANT
Understanding Theories of Change
How does change happen?
“Theories of change are the ideas and
hypotheses (‘theories’) people and organisations have about how change
happens. These theories can be conscious or unconscious and are based on personal
beliefs, assumptions and a necessarily limited, personal perception of reality.
ToC is not a diagram!
■ On-going reflection• Wider analysis of how
change comes about.• Challenging• Focused on what we
want• Complex
■ Academic research■ Linear■ Static■ Focused on what we plan■ Not (necessarirly) reality
What can we use it for?
■A way to clearly communicate (often complex) pathways of change we envisage
■Allows us to acknowledge and question our assumptions of how change does and should occur
■It further clarifies how we think about power and its role in the specific change we are working towards
■Provides a basis for evaluation, learning and accountability
Common mistakes we make
1. We overcomplicate it 2. We develop a theory and never go back to it3. We are restricted by the linear programme log
frame 4. We fail to acknowledge our implicit existing
theory & ideology5. We don’t look beyond CSOs when mapping
(binary world of civil society and state)6. We plan for steady state but we live responding
to opportunities7. We assume the system is static until we arrive8. We don’t keep a record of what’s happened
How does change happen?
Thinking about How Change could happen in relation to a particular issue
What needs to change? At what level?What actors and factors in the external context help or hinder change?
Who has the power to influence change either positively or negatively
What are critical events and opportunities in the way?This thinking goes beyond your own intervention and considers all aspects of addressing identified issues
Theory of Change Cycle
A Voice global Theory of Change (1)
What needs to be in place before deciding on the pathway:• Baseline studies x10• Current state of affairs• SMART objectives• power and gender analysis x10• power mapping (or stakeholder analysis)• Organizational assessments
A Voice global Theory of Change (2)
MEAL framework: Background
■ Basis: Tender document
■ Linked to the Theory of Change and main activities
■ MEAL outcome areas fall under pre-selected IATI outcome areas
■ Qualitative & Quantitative indicators used
Overall outcome goal
VOICE will strengthen the capacity for lobby and advocacy of civil society organisations and representatives of the most marginalised and discriminated groups in order to empower these groups and to have their voices heard and respected, using innovative ideas in the field of capacity strengthening and in the field of lobby and advocacy.
From Theory of Change to Outcome areas
Outcome areas Outcome indicators (quantitative)
1. (Informal) groups have capacity to work on their own empowerment through building safe spaces, creating confidence and skills
400 groups have presented their collective demands to external stakeholders
50% of targeted groups indicate to be able to have safe spaces to convene and work
80% groups indicate an increased confidence to raise their voice
2. (Informal) groups, organisations and networks have capacity to build alliances and networks with a variety of stakeholders, to amplify the voice of marginalised and discriminated groups.
76 alliances and coalitions indicating to initiate activities based on joint agenda
152 publications (media outlets, relevant policy documents etc) supportive to the position of marginalised groups and their change agenda.
3. (Informal) groups, organisations and networks have capacity to use a range of tools and instruments to influence key decision makers
groups, organisations and networks report a benefit (being able to influence) by using a diverse set of advocacy tools (400 groups, 128 organisations and 76 networks)
Targeted outcome area 1
Targeted outcome 1.1: (Informal) groups have capacity to work on their own empowerment through building safe spaces, creating confidence and skills
Output Output indicator
1.1.1 Programme staff and partner organizations have facilitated the development of the agenda of change by marginalised and discriminated groups.
400 agendas for change developed
1.1.2 Marginalised and discriminated groups, with support from partner organizations, have implemented their agenda for change
400 development agendas funded
1.1.3 Marginalised and discriminated groups incorporate new approaches in realising their agenda for change
200 groups indicating use of new approaches
Targeted outcome area 2
Targeted outcome 1.2: (Informal) groups, organisations and networks have capacity to build alliances and networks with a variety of stakeholders, to amplify the voice of marginalised and discriminated groups.
Output Output indicator1.2.1 (Informal) groups, organisations,
networks have defined a joint agenda and campaigns strategy
76 joint campaigns defined
1.2.2 (Informal) groups, organisations and networks have implemented a campaign to raise public support for the position and demands of marginalised and discriminated groups.
64 public campaigns funded by VOICE
Targeted outcome area 3
Targeted outcome 1.3: (Informal) groups, organisations and networks have capacity to use a range of tools and instruments to influence key decision makers
Output Output indicator
1.3.1 (Informal) groups, organisations and networks have defined a plan to strengthen their capacity to influence decision makers.
114 capacity strengthening plans defined
1.3.2 VOICE has allocated grants to support the implementation of influencing strategies by marginalised and discriminated groups
# influencing and sudden opportunity grants provided by VOICE (64 influencing grants and 50 sudden opportunity grants)
1.3.3 VOICE has provided access to expert support and to platforms of social innovators and influencing tools
20 experts and 6 platforms facilitated by Voice
Gender, women’s rights & indicators
■ Gender un-aware/blind
■ Gender neutral
■ Gender sensitive
■ Gender transformative
Gender unaware, neutral, sensitive or transformative?
■ 50% of primary change agents express satisfaction with government accountability mechanisms Priorities of people disabilities are included in community projects funded by the central government
■ 80% of indigenous people have access to safe provisional housing and food.■ 80% of Boys and Girls between ages of 10 and 15 have access to HIV prevention information.■ 200 Small scale farmers men and women are selling their produce in local markets ■ % of women participating in project activities■ 300 women-lead households increase their income by 40% through improved market linkages.■ % of female students living in Spring City who report an increased sense of personal
security/safety in public spaces. ■ 30% increase in women running for public office in local electoral processes. ■ At least 200 women report that home workload is equally shared with men and have more time
for themselves.■ 80% of women credit unions report being part of decision-making spaces and demonstrate
effective participation.■ Public opinion polls indicate that at least 50% of men strongly agree with women´s rights and
capacities to run for public office in local and national elections.■ At least 30% of local budgeted earmarked to support women rights and prevention of VAW
initiatives.
■
Links between Program and Project, Local and Global
■
Voice overall outcome goal
Grant
Grant
Grant
Grant
Grant
Grant
Voice focus
countryVoice focus
country
Hivos and Oxfam change goals/ objectives
Country change
goals/ objectivesCountry change
goals/ objectives
Group work1. How can we assess the quality of our targeted
outcomes? What are your suggestions for qualitative indicators (max 5) per targeted outcome?
2. What are the key assumptions and risks associated with each one of the outcome areas?
In teams of 3 – 1 team per outcome area
1. MEAL planningWhat requirements and within which timeframe?
MEAL matrix: Key elements
■ Establishing a baseline■ Define time-bound activities linked to
outputs outcomes (planned/unplanned)
■ Data collection plan■ Monitoring activities■ Learning and project reviews■ Regular checks on ToC assumptions■ Feedback loops
Data collection & analysis
1. Disaggregate variables2. Identify unit of analysis3. Select / prioritize methods4. Establish frequency5. Define responsibilities6. Plan for analysis 7. General revision
Data management: Key areas of consideration
Source: What are we collecting?• Collection: Who collects this data, from where, and how often?
Collation and Storage• How are data aggregated? And by whom?• Where is the data stored?• Who of the project team is responsible for managing all the data?
Analysis• Possible opportunities to transform the data into more meaningful information and thus for further
review. • Who would be involved in this review and transformation? • Other pieces of information available, besides your own collected data to help you with verification
Reporting• Who will do the write up of the analysis?• To whom will this information be reported?• Use• How can this information be used to make informed decisions?
Gathering responsible data responsibly!
1. Voluntary participation and informed consent
2. Do no harm:3. Confidentiality & Anonymity4. Health and Safety5. Purposeful use of data 6. Good use of funds7. Children-sensitive data?
MEAL framework in practice
Inception:Baseline
Call for proposalsBudget for MEAL
Partners 'assessment
Planning and Implementation:
Identify long term goal and outcomes
Define lines of actionIdentify assumptions
Define reach and scaleRisk analysis
BudgetPlan for exit
Review indicatorsHarvest outcomes
Closure:Exit/handover or scale
up?Capturing and sharing
learning for future programme work
Oxfam’s Common Approach (CAMSA)
Monitoring (1)Projects with a life span that is less than a year
■ Capture reflections on the learning journey
■ No prescriptive format – use your best judgment
■ Recommended areas to monitor: challenges/successes, level and type of engagement, socialization of knowledge
Place your screenshot here
Monitoring (2)Bi-annual light touch reporting (project min. 12 months)
■ Highlights on the evolution of the context, (emerging) lessons on the assumptions underlying ToC
■ Progress on outputs – data should be disaggregated by sex
■ 3 key lessons learnt (successes and challenges) to be used as input to the learning process.
Place your screenshot here
Monitoring (3)Annual reporting – projects min. 12 months
■ Evolution of the context, (emerging) lessons on the assumptions underlying ToC,■ Update of the risk analysis ■ Progress on outcome areas and outputs – data should be disaggregated by gender■ Key findings/learning from monitoring, learning or impact measurement methodology
implemented (outcome harvesting, stories of change).
Reporting levels
■ Milestone grant process reporting
■ Annual grant reporting (grants >= 1 year)
■ Light-touch learning report (grants > 1 year)
■ 6 – monthly internal reporting: # grants (type, amount, themes), activities; main achievements/challenges
■ Time-bound financial reporting: salaries; use of budget lines
■ Annual reporting on outcome indicators
Reporting formats for smaller grants: potential questions
Please summarise the activities conducted in the reporting period■ What key successes have you experienced so far? ■ What challenges has this project faced?■ What challenges are foreseen and how are you mitigating them?
Target Groups■ Explain how likely the intervention is to change the situation of the selected
target group(s)
Organisation, Evaluation and Learning■ Please share any updates to or concerns about the expected results and
timeframes described in the proposal. Particularly if there have been any major changes to outputs or activities.
■ What systems were put into place to monitor the implementation and learn from its results?
■ What support can Voice MEAL teams provide you? (Optional)■ What updates would you like to communicate regarding the key
achievements/messages of the project and how would you do it? (Optional)
Reporting timeline
January:Call for
proposalsDonor report
for ongoing/closin
g grants
July: Bi-annual (light touch) internal
reporting
September: Planning
Indicators reviewNew call
December: December:
Internal yearly reporting on
- Status- Outcome indicators- Learning
Hivos internal
reporting: M
arch/
April
Oxfam output
reporting (direct +
indirect #): March
Who are we accountable to?
Type Minimum standard
TransparencyEssential information on VOICE will be publicly made available and is accessible
through the website, reports and publications. Proactive information sessions will be held specifically for groups that are hard to reach.
Participation
The programme will enable (with funding and adaptive tools) that women and men, beneficiaries of the programme, as well as their organizations define their agendas for
change that will serve as the basis for the allocation of grants.
Local actors will be part of the communities of stakeholders, that participate in setting priorities for VOICE in each of the participating countries.
Monitoring, Evaluation, Learning
Groups and organizations will define indicators for success and learning questions and are actively engaged in measuring progress and sharing answers and insights.
Feedback
The program will have a feedback and complaints mechanism in place and will proactively encourage stakeholders to give feedback. This mechanisms will be
explained on the website. All feedback will be recorded and responded to.
Mechanisms deal with positive and negative feedback in addition to complaints including response
Some more group work!1. How do we make our MEAL framework more inclusive, ensuring that we do raise the voice of the marginalized and discriminated groups?
2. What methods can we use to feed the results back to our grantees (especially informal organizations) and target groups?
Key questions in the back of our head:■ What information do we need?
■ At which level do we need to look for it? ■ What method will we use to collect it?
■ Who will be responsible for generating the information? ■ How frequently will we collect the information?
(3 groups x 3 participants per group)
Unplanned outcomes
Planned outcomes
Longer term change
Measuring longer term (transformative) changes
Res
ults
Fr
amew
ork
Sto
ries
of
Cha
nge;
O
utco
me
Har
vest
ing
Sum
mat
ive
eval
uatio
n
Good practice! Consider developing criteria to identify stakeholders that should participate in
different stages of the evaluation. The criteria should prompt consideration of how to ensure that
participation is representative of the stakeholder group and equity e.g. for participation of women
Evaluation management: Roles and Responsibilities in Voice
What to plan for? Who will be responsible?
Evaluation plan and budget Programme manager
Scoping and mandate of the evaluation Programme manager
Planning of the evaluation Programme manager
Implementation of the evaluation (process – baseline, midterm,
endterm)
MEAL Officer global for final evaluation and contracting authority for the mid-term review
Quality assessment and management response Programme manager
Using the outcomes of the evaluation Programme manager
Are we talking about Impact?
Impact vs. continuous, transformative change
https://www.youtube.com/watch?v=5f4rNEsyEYY
Thanks!Any questions?
Catch me on skype @ lina.kagkliOr email me at: lina.kagkli@voice.global